DEEM: Diffusion models serve as the eyes of large language models for image perception
Authors: Run Luo
Yunshui Li
Longze Chen
Wanwei He
Ting-En Lin
Ziqiang Liu
Lei Zhang
Zikai Song
Hamid Alinejad-Rokny
Xiaobo Xia
Tongliang Liu
Binyuan Hui
Min Yang
Venue: ICLR 2025 Spotlight (2025)
Keywords: MLLM; Diffusion Model;
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Diffusion Model Can help MLLM see better

Abstract: The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal comprehension and creation, they often face challenges when confronted with out-of-distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due to their reliance on image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception? In this paper, we propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP-ViT, thereby enhancing the model's resilience against out-of-distribution samples and reducing visual hallucinations. Importantly, this is achieved without requiring additional training modules and with fewer training parameters. We extensively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM's  visual perception performance to a large extent (e.g., 4\% ↑ on RobustVQA, 6.5\% ↑ on MMVP and 12.8 \% ↑ on POPE ). Compared to the state-of-the-art interleaved content generation models, DEEM  exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10\%), and a smaller base model size. Extensive experiments demonstrate that DEEM enhances the performance of LMMs on various downstream tasks without inferior performance in the long term, including visual question answering, image captioning, and text-conditioned image synthesis.

Published as a conference paper at ICLR 2025
DEEM DIFFUSION MODELS SERVE AS THE EYEs OF LARGE LANGUAGE MODELS FOR IMAGE PERCEPTION
Run Luo Yunshui Li Longze Chen 1.+ Wanwei He Ting-En Lin Ziqiang Li Lei Zhang! Zikai Song* Hamid Sinejad-Rokny"
Xiaobo X7 Tongliang Liu Binyuan Hui
Min Yang"!
'Shenzhen Key Laboratory for High Performance Data Mining, SIAT, CAS ?University of Chinese Academy of Sciences *Tsinghua University
"Huazhong University of Science and Technology University of New South Wales
6 School of Computing, National University of Singapore
MoE Key Laboratory of Brain- Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China 8 The University of Sydney AAlibaba Group
ABSTRACT
The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal compre- hension and creation, they often face challenges when confronted with out-of- distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due their reliance image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes large language models for image perception? In this paper, we propose DEEM a simple but effective approach that utilizes the generative feedback of diffusion models align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP- ViT, thereby enhancing the model's resilience against disttribuitiinonnnnn samples and reducing visual hallucinations. Importantly, this is achieved without requir- ing additional training modules and with fewer training parameters. We exten- sively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM's visual perception perfor- mance to large extent (e.g., 4% on RobustVQA. 6.5% on MMVP, and 12.8 % on POPE Compared to the state-of-the- art interleaved content generation models, DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10%), and a smaller base model size. Extensive experiments demonstrate that DEEM enhances the performance of LMMs on various downstream tasks without inferior performance in the long term, including visual question answer- ing, image captioning, and text-conditioned image synthesis. The code and bench- mark are available at https: //g/ttttbb////////////////h com/ Ra BaiinBoLLoCS/DDER
INTRODUCTION
With the success of large language models (LLMs), large multimodal models (LMMss built on LLMs have garnered significant attention. Researchers (Liu et al., 2024a; Zhu et al., 2023; Dai et al., 2024; Alayrac et al., 2022; Chen et al., 2023) have attempted build bridge between large language models and image encoders through simple mapping modules, and have already made sig- nificant progress in multimodal understanding tasks such as visual question answering. Subsequent
Eqqual contribution. Min Yang and Binyuan Hui corresponding authors.

---

Published as a conference paper at ICLR 2025
LMX For Text Generation Ouly
iaaqe??
Sealion
Prosions Mothol
4G4
Q2is the mais object facing lcft right
Right
163
atf raany lags doeoths object iaage Show?
Pesception
Hallactnntion
Beelase
DEBM
LMM For Text Image Geaeration
Qu What Bhe mais obyet this inaqe??
DEEM!
Goose
341
Qais object toclag kit or dight?
Left
Hfl away don the raals dijec this bxage show?
Figure 1: Illustration of our DEEM. When encountering natural adversarial examples out-of- distribution data, DEEM uses the diffusion model to check if the semantic features of the image encoder match the input images. This approach allows DEEM to serve the "eyes" of the large language model, proactively identifying and correcting misinterpreted semantic information during training, thereby avoiding the loss of important visual details. This enhances the robustness, hal- lucination recognition, and foundational visual perception capabilities of LMMs. In contrast, other models rely heavily on erroneous inputs from the image encoder, making it difficult for them to handle challenges posed by such data.
studies (Yu et al., 2023; Sun et al., 2023b; Dong et al., 2023; Tian et al., 2024) utilize extra advanced diffusion models (DMs) (Rombach et al., 2022) for image generation and train the LMMs on inter- leaved text-image data in an end-to-end manner. This unified paradigm of multimodal understanding and creation brings various isolated multimodal tasks together, greatly boosting model capabilities and expanding application scenarios.
However, these models commonly rely encoder architectures like CLIP-VIT (Radford et al., 2021), which suffers from certain perceptual understanding limitations due to the contrastive learn- ing paradigm and the noisy image-text pairs used in training, to encode input images. Additionally, these image encoders are typically trained to encode images into features relevant to downstream tasks, thereby disregarding irrelevant details. Consequently, as shown in Fig. 1, when faced with im- ages outside the training scope, they often capture biased semantic features, resulting in erroneous visual information being perceived by subsequent language models. This accumulation of inaceu- racies renders the multimodal model unable to comprehend multimodal context effectively. For this reason, this makes it difficult for previous methods to discern subtle details, thereby hindering their ability to handle tasks related to basic visual perception, visual hallucinations, and visual robustness that are very simple for humans.
On the contrary, the goal of diffusion models (Ho et al., 2020a) is to learn diffusion process that characterizes a probability distribution for a given dataset, without direct training on the down- stream task objective. This enables it to capture finer details of images for better handling of out-of- distribution data. However, there have been few efforts to integrate the capabilities of the diffusion model into the image perception of large multimodal models.
In this paper, we propose DEEM, a simple but effective approach to leverage the generative feed- back of diffusion models for aligning the semantic distributions of image encoders an elegant self-supervised manner. Building upon this, we introduce an end-to-end interleaved image-text gen- erative modeling approach, where diffusion models serve as additional eyes of large language mod- els for image perception. This addresses the limitations of previous methods that solely relied on image encoders such as CLIP-VIT (Radford et al., 2021), enhancing the model's robustness against -ittof-distribution samples and reducing hallucination perception in multimodal scenarios, with- out the need for additional training modules and with fewer training parameters. the best of knowledge, we are the first to apply diffusion models to large multimodal models for image perception.
Specifically, DEEM takes interleaved image-text pairs as input to the model. It starts by encoding images and text using corresponding visual and text encoders, resulting in image tokens and text tokens. These tokens then organized according to their original layout and inputted into large language model to generate corresponding hidden state outputs. The model employs autoregressive modeling for the hidden state outputs of text and utilizes the output hidden states of images, along
2

---

Published as a conference paper at ICLR 2025
with the image tokens encoded by the image encoder, diffusion conditions. These conditions then fed into a diffusion model for image reconstruction. Through end-to-end training, the model not only acquires the capacity to generate text and images but also employs semantic consistency regularization on the semantic information produced by the image encoder during image reconstruc- tion. This compels the image encoder to incorporate more details into the semantic representation of the image, thereby mitigating the issue of semantic bias in image encoding.
DEEM is trained on mixture corpora of image-text pairs and interleaved image-text sequences data without extra in-house data following previous solution (Li et al., 2022; 2023a; Dong et al., 2023: Tian al., 2024). To assess the robustmess recognition capability of LMMs, we constructed a new robustness benchmark, RobustVQA, based on existing datasets containing natural adversar- ial samples and iitto--distribution data. RobustVQA is divided into three parts: (AauusttVAA,, RobustVQA QQAR, and Robust VQA-V, based on different data sources, aiming to provide better in- sights into the performance of LMMs in real-world scenarios. We conducted extensive evaluations of DEEM on RobustVQA and two widely recognized benchmarks, POPE and MMVP, for visual hallucination and perception respectively. Experimental results indicate that our method exhibits enhanced robustness, superior capacity to alleviate model hallucinations and better visual per- ception ability in comparison to the state-of-the-ar interleaved image-text modeling model MM- Interleaved (Tian al., 2024), using smaller-scale image encoder (CLIP-ConvNext-B (Liu al., 2022) vs. CIT--ViT-L (Radford et al., 2021)), smaller-scale language model (Vicuna 7B vs. Vi- cuna 13B (Zheng al., 2024)). and less pre-training data (without Laion-coco (Andreas al., 2022) & Laion-en (Schuhmann et al., 2022)). DEEM outperforms MM-Interleaved 9.4% on RobustVQA, 17.8% on POPE and 9.1% on MMVP. Morcover, with further enhancement via supervised fine- tuning, DEEM achieves competitive results on various multimodal tasks, including visual question- answering, region-level image captioning. and eet--to-mmaee generation.
Before delving into details, we summarize our contributions as follows.
Robustness Benchmark. We design new robustness benchmark RobustVQA for LMMs based on publicly available ImageNet-A (Hendryeks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a), and ImageNet- V2 (Recht et al., 2019) datasets, which can be utilized to effectively assess the visual robustness capabilities of the multimodal models.
Effective Method. We are the first to introduce the diffusion model into the image perception of large language models, to correct potential semantic bias in the image encoder and alleviate the excessive compression of visual details. This approach enhances the model's 's robustness and hallucination mitigation capabilities without the need for additional modules trainable parameters.
DEEM Model. Based on the proposed method, we train a multimodal model with end-to- end interleaved text-image modeling capabilities. After supervised fine tne-tuning, DEEM can perform various multimodal tasks in unified manner, such as visual question answering, text-to-image generation, and regien-level image captioning.
Comprehensive Experiments. We provide abundant qualitative and quantitative comprehensive experimental results demonstrate the effectiveness and efficiency of the proposed method.
2 METHOD
In this section, we first present our DEEM, starting with an introduction to the overall architecture in Section 2.1, followed by description of the pipeline in Section 2.2. Finally, provide details on the training and inference process in Section 2.3.
2.1 ARCHITECTURE
In this subsection, present the multi-modal architecture for processing interleaved image-text data. To excel in both comprehension and creation tasks of text and images, a multi-modal model consists of the following three key components.
VFMebased Image Encoder Ey which encodes each image V E #xxx3 into an image em- bedding e R XxC. where C' is the channel dimension and N is the number of visual tokens in image embedding. LLM-based Multi-modal Decoder DLTM that extracts context features from the interleaved image-text token sequences. Its input sequence F. E KxCC is a concatenation of
3

---

Published as a conference paper at ICLR 2025
ExTp
Cors
CNTP
EyTP
Image embeelling
<p Eilee thss intrestig veplabbe
Trauginn wien the regraible are gatherd togttbe
Ward embedding
Wadd
ScadE
deck dsp
iateesting
Causal Large M Multimodal Model (LM
Billakee
Prediction
Excander
Excader
can this aatrroting vegesable clack
Less
Emzgiee whet the
Coss
mote What an
New
Restoration
will bak mamiliba cumpled clock
istirnting clock!
vegetblls.
Figure 2: Overview of our DEEM framework. Interleaved documents serve input, decoded to produce outputs. Both text and images are encoded into sequential, discrete token embeddings for the LMM input. Here, we replace the <IMG> token embedding in the text with the image embedding before inputting it into the LLM. The text is predicted in an autoregressive manner and the images are synthesized by the DM-based image decoder conditioned semantics captured by LMM. Besides, the image token embeddings are fed into DM-based image decoder for consistent image restoration. The start of image token <SOI> is used determine the starting position of the image, facilitating the natural autoregressive generation of interleaved text- image layouts. Note that our core architecture is presented without the connectors between modules for simplicity.
holistic historical
embeddings (c1, e). where en is either a word embedding ex E R IxC or an image embedding ex E RNCC K is the total number of input tokens. DM-based Image Decoder DDM that generates the image conditioned on image-text sequences context feature.
To provide the conditional inputs for DDM and reduce the number of visual tokens in image em- bedding eV, two different Perceiver Resampler (Alayrac et al., 2022) are employed to map the output features from multi-modal decoder DLLM and image encoder yy fixed number of con- ditional tokens, respectively. Additionally, we utilize an extra mask-aware visual extractor &M for extracting region visual information from image embedding e" via simple mask-awarg operation Em(e", ,M' where M is the corresponding binary mask of image x V.
2.2 PIPELINE
As shown in Fig. 2, given an interleaved image-text sequence X = {x1, Øz.... } where each element Vn is either text token (denoted w.))) or whole image (denoted x, Text and images are arranged in the order in which they appear in the original content. To build an end-to- end generative model for interleaved image-text data, common practice is to first extract embed- ding for each text token and each image and then feed them into LLMs, i.e., e = &g (x()) and eY = (E (, ), MM where & denotes word embedding in LLM. yv is typically an image encoder followed by Perceiver Resampler (Alayrac et al., 2022) to map each image to a fixed shown in Fig. 3, we introduce mask-aware visual extractor EM for extracting region visual information from image embedding ey via simple mask-aware operation (M(e)) .M ), where MY is the corresponding binary mask of image x and the default value is Then, the interleaved generative modeling is trained to maximize the log-likelihood:
number of visual tokens.
log = Σ logp(enlecn) = Σ Togp(thlecn) DEII
logp("y (ecn), (ogp())leen)
(I)
neZy
prediction
image prediction
where Ir and Zy represent the index sets for text tokens and images, respectively. That < n in the subscript represents the abbreviation of (1,2..... 1). The following paragraphs provide explanations of Eq. (1).
Text Generation with Multi-modal Condition. p(pp///eeen is similar to traditional causal lan- guage modeling, except that the condition also includes previous images. Recent works (Alayrac al., 2022; Li et al., 2023a; Liu et al., 2024a) have demonstrated the effectiveness of using LLMMs
4

---

Published as a conference paper at ICLR 2025
for processing additional visual inputs.
The loss function for text generation is LNTP (erp(z///ee - p(a, (LLM))en)),
(2)
where DLTM denotes the LLM network.
Image Generation with Multi-modal Condition. Maximizing log p [een) aligns with the dif- fusion denoising process, which recently achieved widespread success in image generation. Maxi- mizing the log-likelihood is derived as minimizing the diffusion modeling loss as
(3) where DRM is the diffusion model for denoising process. That w.Y. is the noisy version of the original image at the denoising step I, and the denoising network DDa trained to predict the noise 6.
Lxw(r (<en) = Ee, ||c DoM (I%, (
Consistency Semantic Regularization. In addition to the above text and image generation loss functions, we propose a new consistency semantic constraint term. This term reuses the diffusion model to perform generative checks on the image semantic information extracted by the image encoder, ultimately correcting erroneous knowledge in the pre-trained image encoder. This sig- nificantly enhances the out-of dito---distribution generalization and reduces visual hallucinations in the multi-modal model. The log-likelihood function be written
pegp'(X)
Σ log 1og (22)) nEZL
Σ NETV
log log +
Σ
log
(4)
prediction
image prediction
T ZEZV
image restoration
Similarly, the corresponding log-likelihood function log p(o lea) can be equivalently written as the following loss function used in training:
LCsR( (z) len) Ee, ||e DDM ( (Ix e.
Note that the new end-to-end modeling framework brings significant improvements to the general- ization performance of the model without altering the original modeling flexibility or introducing additional modules.
2.3 TRAINING AND INFERENCE
We employ three-stage training process. consist- ing of image-text alignment pre-training, image- text instruction fine-tuning, and mask-text instruc- tion fine-tuning, Image-text alignment pre-training and image-text instruction fine-tuning are designed to validate the effectiveness and efficiency of se- mantic consistency regularization in enhancing the visual perception capabilities of LMMs. Mask-text instruction fine-tuning used verify whether the model trained with semantic consistency regular- ization negatively impacts the performance of fine- tuning downstream tasks in the long term. The image-text alignment pre-training objective is de- fined the of the next-text prediction loss in Eq. (2), next-image prediction loss in Eq. (3) and consistency semantic regularization loss in Eq. (5)
Mask-Aware Extractor
Innag
boby elephont
Provide short deseription for this creforegion cheef
Figure 3: Pipeline of Mask-Aware Extrac- tor. The mask-aware extractor can be used to extract region-level visual features based on the mask-aware operation. simple dot product is applied between the mask and the image embedding before being fed into the L.M.
as Ls, ΣNIP + X LNIP + X LCSR. where X is a coefficient used to determine the relative loss weight between the image and text decoding branches. In order to enable the DEEM per- form general multimodal comprehension and creative tasks following human instructions, we use Ls, = LNIP + LCSR to conduct image-text instruction fine-tuning. To further enhance the model's fine-grained region awareness, we conducted region-level mask-text instruction fine-tuning. Since there is no need to perform text-to-image tasks, we removed the next-image prediction loss and the training objective in mask-text instruction fine-tuning can be defined as Ls, LNTP. The whole framework can be optimized end-to-end during the three stages. During inference, the images and texts generated in auto-regressive manner. Text tokens sampled from the distribution pre- LLM. When the generated token <ScI>, the diffusion model is called for generating the next image.
dicted by the

---

Published as a conference paper at ICLR 2025
3 EXPERIMENT
3.1 IMPLEMENTATION DETAILS
In this subsection, we first introduce the network of DEEM and then showcase the stree-stage train- ing recipes. More details of datasets and hyper- super-parameters can be found in Table 11.
Network. Similar to previous work, We leverage Vicuna7B (Zheng et al., 2024) and Stable Diffusion v2.1 (Rombach et al., 2022) as the large language model, and image decoder, respectively. However, unlike their use of a 427M parameter CLIP-ViT-L as the image encoder, we use a smaller 122M parameter
et al., 2022). For the multi-modal LLM, two different Perceiver Resamplers (A (Alayrac et al., 2022) are used to connect diffusion model with image encoder and large language model respectively.
image-Text A Alignment Pre-training. Our model is pre-trained on a mixture of image-text pairs and interleaved image-text sequences, including MMC4-Core (Zhu et al., 2024), LAION-400M (Schuh- mann et al., 2021), SBU (Ordonez et al., 10, and CC- 1C-12M (Changpinyo et al., 2021). For LAION-400M (Schuhmann et al., 2021), SBU (Ordonez et al., 2011), and CC-12M (Changpinyo et al., 2021), instead of utilizing the original annotations, we use the version filtered by the pre- trained BLIP-2 model (Li et al., 2023a). For simplicity, we refer to it as BLIP-LCS hereafter. "LCS" abbreviates the LAION, CC. and SBU datasets. The sampling probability of MMC4 is twice that of BLIP-LCS. The images are inserted before or after the corresponding text sentence with equal probability. To optimize training efficiency and data utility, multiple image-tex. pairs or interleaved image-text sequences are concatenated into extended sequences with the maximum context length.
mexte-Text Instruction Fine-tuning. To enable DEEM to perform general multimodal compre- hension tasks following human instructions, we utilize publicly available datasets for image-text instruction fine-tuning, including LLaVA-665K (Liu et al., 2024a), COCO Caption (Chen et al., 2015), VQAr2 (Goyal et al., 2017), TextCaps (Sidorov et al., 2020), OCR-VQU (Mishra et al., 2019), GQA (Hudson & Manning, 2019), OK-VQA (Marino et al., 2019), TextVQA (Singh al., 2019), and AOK-VQA (Schwenk et al., 2022).
Mask-Text Instruction Fine-tuning, At this stage, we use a simple mask-aware visual ex- tractor to capture pixel-level region features and then align mask-based region features with language embeddings. We collect short text and pixel-level mask pairs from the publicly available object-level datasets (COCO (Chen et al., 2015), RefCOCO (Kazemzadeh et al., 2014), RefCOCO+ (Mao et al., 2016), Ref- COCOg (Mao et al., 2016)), part-level datasets (Pascal Part (Chen et al., 2014), Part Ima- genet (He et al., 2022)), and multiple region datasets(VCR (Zellers et al., 2019), Visual Genome (Kr- ishna et al., 2017)). Then we conduct mask-text instruction fine-tuning on the mixture of the above text-mask pairs data, enabling DEEM to complete region-level understanding tasks, such as region- level image captioning.
ImapeNa-R
ImugeYeted
ImageNar112
Figure 4: Examples from ImageNet-RR ImageNet-A, and ImageNet-V2. These exam- ples share similar backgrounds, rare materials, and unusual textures. They serve as natural adversarial examples and out-of-distribution data, which can be used to test the robustness of models.
3.2 EXPERIMENTAL RESULTS
In this study, we evaluate our DEEM model by comparing it with current state-of-the-an art (SOTA) models on various tasks including visual robustness, hallucination diagnosis, basic visual perception and image-level visual question answering. Please refer to Appendix C for more experimental results about mask-level visual question answering and text-to-image generation. All metrics and data splits are listed in Table 11 in Appendix E.
Visual Perception Diagnose. We explore the impact of diffusion feedback on the visual perception capabilities of LMMs from three dimensions: visual robustness, visual hallucinations, and basic vi- sual perception. To rigorously assess visual robustness of our model, we design a benchmark called Robust QA for robustness evaluation based online datasets, including ImageNet-A (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a) and ImageNet- V2 (Recht et al., 2019). As shown in Fig. 4, these challenging natural adversarial examples and out-of distr--iittinttionn samples in

---

Published as a conference paper at ICLR 2025
the original ImageNet dataset can be used to evaluate the neural network robustness of model. Similar to the POPE and MMVP dataset, we first choose the challenging sample from ImageNet- A, ImageNet-R, and ImageNet-V? V2 dataset and then convert the them into a VQA format that the multimodal model can evaluate simply and accurately. More details about the new benchmark Ro- bustVQA design can be found in Appendix 1. For a comprehensive visual robustness and halluci- nation evaluation, we evaluate our model against other open-source state-of-the-ard (SOTA) LMMs for text and image generation, including SEED (Ge et al., 2023). SEED-X (Ge et al., 2024), MM- Interleaved (Tian et al., 2024), and DreamLLM (Dong et al., 2023), on the RobustVQA, POPE (Li et al., 2023c) and MMVP (Tong et al., 2024) dataset with accuracy metric. The results, presented in Table 1, demonstrate that our DEEM model not only exhibits competitive performance compared with existing fine-tuned SOTA models on POPE and MMVP after fine-tuning, but also achieves the best results among visual robustness benchmark only after pre-training. Notably, compared to the larger-scale concurrent SOTA model for interleaved text-image modeling. MMMIIterlcavedd (Tian et al., 2024), our model achieves better results with a smaller scale. DEEM outperforms MM- Interleaved 9.4% on RobustVQA, 17.8% on POPE and 9.1% on MMVP. To ensure a fair compar- ison and prove the effectiveness of our method, we also train MM-Interleaved model with the same experimental setting as a baseline. Compared to this baseline, Our method achieves an 4% average gain on RobustVQA, 12.8% average gain on POPE and 6.5% average gain on MMVP, re- spectively. The experimental results demonstrate the effectiveness of our method for better LMMs' visual perception capability.
Image-Level Visual Question Answering and Captioning, In order to assess multimodal vision and language capabilities of DEEM, we conduct evaluation against current SOTA LMMs including LLaVA-1.5 (Liu et al., 2023), Qwen-V VL (Bai et al., 2023), DreamLLM (Dong et al., 2023) and MM- Interleaved (Tian et al., 2024) across several tasks, including image captioning on COCO (Chen et al., 2015), Image2Paragraph (Krause et al., 2017), visual question answering on VQAv2 (Goyal et al., 2017), OKVQA (Marino et al., 2019), GQA (Hudson & Manning, 2019), VizWiz (Gurari et al., 2018), and VisDial (Das et al., 2017). As demonstrated in Table 2, DEEM exhibits superior or comparable performance relative to SOTA models. In comparison with models for text genera- tion only, our approach consistently achieves competitive performance across various dataset splits. Against models for both image and text generation, DEEM demonstrates enhanced performance in nine dataset splits. Compared the current of-the-------ttt model DreamL DEEM outperforms
DreamLLM in six out of the seven shared evaluation dataset splits. It is noteworthy that DEEM is trained with a significantly smaller image encoder NLI--C (Liu et al., 2022), compris- ing only 1 122M parameters, in stark contrast to baselines such as DreamLLM (Dong et al., 2023), which utilize larger 427M ViT-L (Radford et al., 2021). These results indicate that our method can enhance the model's robustness performance without compromising the multimodal vision and language capabilities of our model.
Table 1: Zero-shot visual robustness, hallucination and perception evaluation of RobustVQA-AA RVQA-A, RobustVQA-R: RVQA-R, Robust RVQA-V, POPE-Random: POPE-R (Li et al., 2023c), POPE-Popular: POPE-P (Li et al., 2023c), POve-Adversarial:: POPE-A (Li et al., 2023c) and MMVP (Tong et al., 2024) benchmarks. RobustVQA-A RobustVQA-R, and Robust are robustness benchmarks designed by us in Appendix E. "AVG" denotes the overall average accuracy of seven benchmarks. "SFT" denotes the supervised fine-tuning. ago denotes bascline
model without diffusion feedback. The evaluation metries for each benchmark are listed in Table 12.
Mothod
SFT Architecture
RVQA-A RVQA-R RVQA-V POPE-R POPE-P POPE-A MMVP AVG
Models for Gext-Gennrotionn Only
Shikra (Chen al., 2023)
VIT-LALLaMA 7B
33.71
38.33
37.45
86.90
83.97
83.10
22.56
55.15
NeXT-Chat (Zhang et al., 2023a)
VTT-L/icuna 7B
44.82
43.67
47.30
87.70
84.57
81.93
27.41
59.62
Models for Text and Imuge Generation
MM-Interleaved (Tion et al., 2024) X
ViT-L/Nieuna 13B
50.76
52.71
50.60
64.73
65.33
65.
23.82
53.31
Emm-1 (Sun ot al., 2023b)
VIT-L/Vicuna 7B
46.40
49.12
47.36
61.
56.79
56.01
22.69
48.52
SEED (Ge et al., 2023)
ViT-G/Vicuna 7B
52,06
59.71
57.02
69.
56.83
59.63
25.62
54.39
Dream M. (Dong et 2023)
VVT-1.Vicuna
51.43
58.96
$7.60
86.36
80.07
72.63
26.37
61.84
SEED-X X (Ge et al., 2024)
V/TcG/Viccna. 13B
52.36
60.27
59.49
86.41
81.43
74,56
29.16
63.39
DEEM
X
Cony-Next-B BexttB//iccma 7B
53.24
56.06
54.72
50.55
52.00
51.93
20.30
48.40
DEEM
ConyNext B/Vicuna 7B
56.86
68.63
63,08
69.93
70.27
68.87
28.74
60.91
DEEM -VQA
Cony Beost-///iccaaac 7B
55.22
64.12
62.99
$7.40
82,80
78.49
32.89
65.56

---

Published as a conference paper at ICLR 2025
Table Multi-modal comprehension evaluation. "ED" denotes using extra in-house data. Bench-
marks include COCO (Chen et al., 2015); 12Para.: Image2Paragrape (Krause et al., 2017); VQA v
VQAv2 (Goyal al., 2017); OKVQA (Marino al., 2019); GQA (Hudson & Manning, 2019); VizWiz (Gurari et al., 2018); VisDial (Das et al., 2017); MMBench: MMB (Yu et al., 2024);
MMVet (Yu et al., 2024);, The evaluation metrics for each benchmark are listed in Table 12.
Model
LLM
VFM
ED COCO 12Para. VQA
OKVQA GQA VizWiz VisDial MMB MMVe
Modill for Text Goveration Only
IDEFICS S-B (IDEFICS, 2023)
LLaMA-65B
VITH
91.8
60.0
45.2
36.0
27.9
IDEFICS-SOBB (IDEFICS, 2023)
LLaMA-658
VIFH
117.2
37.4
26.0
KOSMOS-1 (Heang
2024)
MetaLM
VIEL
46.7
KOSMOS-2 (Peng ol. 2023)
KOSMOS-1
Flamingo-9B (Alayme et al., 2022)
Chinchilla-7B
VIEL
79.4
51.8
44.7
28.8
48.0
7.9
23.3
Hamingo-808 (Alayme al, 2022)
Chinchilla- 708
VIEH
84.3
56.3
50.6
31.6
$2.0
mPLLL'G-D<OOl
2023)
BLIP-2 20233)
Vicura-78
VIEL
38.6
25.3
BLIP-2 al., 20223)
Vicuna-133
VITL
41.0
41.0
19.6
IestructBLJIP (Dai
Vicuna-7B
VIELL
49.2
34.5
68.9
33.1
InstruciBLJIP (Dai et 2024)
Vicura-138
VIEL
49.5
33.4
(Chen all, 2023)
Vicura-1
WEEL
117.5
77.4
LLaVA-1.5 (Liu ot 2023)
Vicura-78
VIFL
78.5
62.0
50.0
53.1
32.9
2023)
I3B
Qwen- VL (Bai al., 2023)
Owen-
VIG
78.8
59.3
35.2
32.9
13.0
Qwen- VL-Char (Bai et 2023)
Qwen-70
VIGG
78.2
57.5
38.9
59.1
Models for Iuage Text
CM5Lzon (Ys et 2023)
61.6
10.5
47.6
23.8
37.6
22.6
Emu (Sun al., 20236)
Vicuna- 13B
VITL
112.4
52.0
38.2
34.2
47.4
WIEL
Emu2 (Son al,, 2025a]
LLaMA-338
WEL
33.3
26.7
40.4
Dreaml IM (Dong et al, 2023)
Vicuna-78
WEL
X
103.7
8.4
72.9
52.2
49.3
58.2
36.6
.VQA
Vicuna-78
ConyNext-B
115.4
22.4
534
50.4
60.8
3.3 ABLATION STUDY
In this study, we conduct ablation studies on several key components of the model, including consis- tency semantic regularization, training latency, scalability and the impact of different architectures. Benchmarks include RobustVQA-A.RVQAA
RobustVQA-R: RVQA-R; V; POPE-R (Li et al., 2023c); POPE-P (Li et al., 2023c); POPE-A (Li et al., 2023c); MMVP (Tong al., 2024); OK-VQA (Marino al., 2019). More additional ablation studies be found in Appendix D.
Consistency Semantic Regularization and Training Latency. To evaluate the effectiveness of the key elements of our design, we conduct the following ablation experiments. We first pre-train baseline model without using the consistency semantic regularization term under the same training setting for comparison to demonstrate the effectiveness of our architecture. As we can see from Ta- ble during the pre-training phase, using our consistency semantic regularization can significantly enhance the model's performance on both hallucination and robustness benchmarks. Moreover, we load the weights of the pre-trained model for image-text instruction fine-tuning experiments. In the second phase of image-text instruction fine-tuning experiments, we demonstrate the effectiveness of our model design. As detailed in Table we observe that after fine-tuning with image-text in- struction data, the model's visual hallucination ability improves further, but its visual perception robustness decreases. However, using our consistency semantic regularization can mitigate the ro- bustness degradation while further enhancing the model's 's visual hallucination ability. To explore the impact introducing consistency semantic regularization on the training latency in the two stages
Table Ablation study of LCSR and training latency. Using semantic consistency regularization during both the pre-training and supervised fine-tuning phases significantly enhance the model's robustness and resistance to hallucinations, while incurring only a marginal additional training cost.
SFT LeSR RVQA-A RVQA-R RVQA-V POPE-R POPE-P POPE-A
SPEED
X
x
53.2
56.1
54.7
50.6
52.0
51.9
8.11 s/step
X
/
57.8
69.0
64.8
69.9
70.3
68.9
9.25 s/step
/
x
51.3
56.5
57.4
85.4
78.8
76.2
2.14 s/step
/
/
53.5
57.6
58.1
86.0
79.2
77.1
2.22 s/step
8

---

Published as a conference paper at ICLR 2025
of training, conduct corresponding ablation experiments. We present the result in Table 3. Em- ploying consistency semantic regularization adds only a marginal increase in training latency, yet significantly enhances the model's robustness capabilities.
Table 4: Ablation study of model scalability. Gradually expanding the training data and model size can further enhance the model's capabilities, demonstrating the scalability of the approach.
Architecture
Training Data RVQA-A RVQA-R RVQA-V POPE-R POPE-P POPE-A A OK- OK-VQA
ConvNext-B/Vicung 7B
32K
51.86
54.31
52.73
48.44
50.10
50.06
20.74
ConvNext- B/Viccna
7B
96K
52.31
57.43
54.06
54.42
57.22
56.35
22.33
ConyNext- 7B
160K
52.89
58.93
55.31
60.28
60.74
59.96
23.65
ConyNext-I_/iamne 78
160K
53.23
60.47
56.88
61.12
62.87
62.09
23.87
ConvNext-B/Vicuna 13B
160K
53.92
61.27
57.02
62.60
64.26
63.19
31.11
Model Scalability. Although DEEM demonstrates better performance with smaller data count and model sizes, its scalability has yet to be validated. As well known, scalability is crucial for model performance. We conduct ablation experiments to assess the scalability concerning data count and model size. As shown in Table 4, gradually increasing the training data enables the model successfully scale while achieving improved results. Additionally, increasing the sizes of both the VFM and LLM leads to sustained performance enhancements, indicating that DEEM possesses good scalability.
Table 5: Ablation study of different architectures. Our method not only significantly enhances the capabilities of LLMs for text and image generation with marginal additional training costs, but it also improves the performance of LLMs for text generation only, validating the generalization ability of the approach.
Name LLaVA x
LeSR
MMVP RVQA-A RVQA-R RVQA-V POPE-R POPE-P POPE-A
18.6
54.8
60.0
58.7
55.5
53.3
54.6
LLaVA /
25.1
56.7
66.7
61.9
67.9
68.7
65.4
DEEM X
20.3
53.2
56.1
54.7
50.6
52.0
51.9
DEEM /
28.7
56.9
68.6
63.1
69.9
70.3
68.9
Impact Different Architectures. By cleverly reusing the diffusion model from LMMs for image and text generation, we can significantly enhance the model's foundational visual perception, visual robustness, and anti-hallucination capabilities with only marginal additional training costs. How- ever, whether DEEM possesses sufficient generalization ability to remain effective for LMMs on text generation only has yet to be explored. To validate our hypothesis, we employ the LLaVA (Liu al., 2024a) architecture and conducted ablation experiments using semantic consistency regularization loss, with results presented Table 5. We observe that utilizing diffusion feedback to improve the basic perceptual capabilities of LMMs-tthe preventing the model from overly compressing visual information and losing sensitivity to subtle details-is a general method that is architecture-agnstie and exhibits good generalization properties. This suggests that the benefits of approach could extend beyond the specific configurations tested, potentially enhancing a wide range L.Ms in various applications.
RELATED WORK
4.1 DIFFUSION MODELS FOR REPRESENTATION LEARNING
Diffusion models have made significant progress in various generative tasks (Song et al., 2020; Ho al., 2020b), such as image generation (Betker al., 2023), video generation (Ho et al., 2022), and object tracking (Luo et al., 2023). In addition to the aforementioned research, many studies focus on leveraging diffusion models for representation learning. Some works utilize the conditional control of pre-trained diffusion models to flexibly address different downstream tasks, including object clas- sification (Xiang et al., 2023), semantic segmentation (Xu et al., 2023), image caption (Wei et al., 2024), and keypoint matching (Nam et al., 2023). Other studies (Li et al., 2023b; Song et al., 2024) design specialized modules and train diffusion models from scratch to further enhance representa- tion capabilities. Although diffusion models have been widely applied in the generative tasks of

---

Published as a conference paper at ICLR 2025
large multimodal models, the use of diffusion models to optimize the visual representations of large multimodal models has yet to be explored. To our knowledge, we are the first to employ diffusion models in self-supervised paradigm to optimize the visual representations of large multimodal models, significantly enhancing their perceptual abilities and reliability at minimal cost.
4.2 LARGE MULTIMODAL MODEL
Image-to-text large multimodal models (LMMs) (Luo et al., 2025; Liu et al., 2024c; Zhang et al., 2023b; Wang et al., 2024; Zhou et al., 2024; Liu et al., 2024b) inject visual information into large language models (LLMs) through vision foundation models (VFMs), allowing the language models to perceive visual inputs and thus generate captions or answer questions based on the given mul- timodal content. Flamingo (Alayrac et al., 2022) tries to extract vision features with resampler, and transfer them into the text features with a cross-attention mechanism. Instead of using cross- attention layers, BLIP-2 (Li et al., 2023a) directly feed the visual features into the LLMs as soft prompts and significantly reduce the training cost by reducing the visual token number. LLaVA (Liu et al., 2024a) and MiniGPT-4 (Zhu et al., 2023) construct small-scale instruction tuning dataset to better align the LMM with the expected output format. Although this unidirectional image-to teetett paradigm has achieved tremendous success, it still fails to unify multimodal tasks like ext-to-image generation and image-to-text visual question answering, significantly limiting the capabilities of multimodal models.
In order to unify multimodal tasks into a unified manner, some works (Yu et al., 2023; Koh et al., 2024; Sun al., 2023b; Dong al., 2023; Tian et al., 2024; Ge et al., 2023; 2024; Luo et al., 2024) attempt to generate images and text in the interleaved context concurrently. The release of some public large-scale interleaved image-text datasets (Laurençon et al., 2024; Zhu et al., 2024) has significantly advanced the development of this field. CM3Leon (Yu et al., 2023) converts im- ages into discrete tokens, facilitating token-level auto-regressive modeling as traditional language modeling. Although CM3Leon showcases competitive image generation capabilities, it exhibits no- table weaknesses in image understanding. Emu (Sun et al., 2023b) and DreamI.MM (Dong et al., 2023) focus single-stage end-to-end modeling using raw image pixels as input for interleaved image-text generation modeling, but they feed image information at the input of LMMs, which are limited by the problem that fixed number of visual tokens cannot efficiently describe image de- tails. MMM-Intrleaaeed (Tian et al., 2024) addresses this limitation by integrating image details into LMMs via multi-scale visual features. However, when faced with ouitri-dist noisy data, the image encoders used by LMMs often produce incorrect visual information, ultimately leading to erroneous predictions. This significantly limits the application of the models in safety-critical scenarios. Building on an advanced interleaved content modeling mechanism, we propose DEEM, which cleverly reuses DMs to correct the outputs of the VFMs without increasing extra parameter count, thereby enhancing the model's ' generalization capabilities and reducing visual hallucinations in a self-supervised manner. Similar to previous work (Liu al., 2024a; Dong et al., 2023; Tian et al., 2024), after supervised fine-tuning, it achieves competitive performance on multiple down- stream multimodal tasks with the smallest scale.
5 CONCLUSION
Can diffusion models serve the eyes of large language models for image perception? In this paper, we answer the question by proposing a novel method called DEEM, which leverages a diffusion model the eyes for LLMs. This approach enhances the robustness of the multimodal model for in- terleaved image-text modeling and reduces visual hallucinations without introducing extra modules. Through comprehensive exploratory experiments, we demonstrate the effectiveness of the proposed DEEM method. In addition to its advanced robust performance and visual hallucination handling capabilities, we adopt an additional two-stage instruction fine-tuning process to broaden the applica- tion scenarios of our DEEM. This enables DEEM to handle variety of multimodal tasks, including visual question answering. image captioning. and region-level image reasoning. Besides, this work initiates the first step towards visual robustness via generative feedback in a multimodal model. In the future, we will continue to enhance the model's s ability to conduct better multimodal compre- hension and creation tasks. As an end-to-en framework, we hope it will spur further research in the multimodal robustness field, such as multimodal agents that can handle complex tasks that require safety abilities.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
Min Yang is supported by National Key Research and Development Program of China (2022YFF0902100) 100), National Natural Science Foundation of China (Grant No. 62376262), the Natural Science Foundation of Guangdong Province of China 1 supported by MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, Univer- sity of Science and Technology of China (Grant No. 2421002).
Xiaobo Xia is
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, al. Cm3: A causal masked multi- modal model of the internet. arXiv preprint arXiv:2201.075220 2022.
lean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, pp. 23716-23736, 2022.
Köpf Andreas, Vencu Richard, Coombes Theo, and Beaumont Romain. Laion coco: 600m synthetic captions from laion2b-en, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pp. 2425-2433, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 010222308..9666 12966, 2023.
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-ei pdf. 2(3):8, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pp. 3558-3568, 2021.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306.1519. 2023.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. De- tect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, pp. 1971-1978, 2014.
Xinlci Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: /504: 504.00325. 2015.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purposs vision- language models with instruction tuning. In NeurIPS, 2024.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, pp. 226-335, 2017.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering rext-to-image generation transformers. In NewrlPS, pp. 19822-19835. 2021.
Ming Ding, Wendi Zheng, Wenyi Hong and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. In NeurlPS, pp. 16890-16902, 2022.
11

---

Published as a conference paper at ICLR 2025
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.114: 14999 2023.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene- based text-to- ext-to-image generation with human priors. ECCV, pp. 89-106, 2022.
Yuying Ge, Yixiao Ge. Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint (tt::2307.08044 2023.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi multi- caalti-gaauuarityy comprehension and generation. arXiv preprint arXiv:2404. 20t::240..4390 /4396, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stapy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6904-6913, 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pp. 3608-3617, 2018.
Ju He, Shuo Yang, Shaokang Yang, Adam Kontlewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang. Qihang Yu, and Alan Yuille. Partimagenet: A large, high-quality dataset parts. In ECCV, pp. 128-145, 2022.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribttion generalization. In ICCV, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pp. 15262-15271, 2021
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, pp. 6840-6851, 2020a.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020b.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprini (1..i:::210000::: 2022.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. In NewrIPS, 2024.
Drew Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
IDEFICS. Introducing idefies: An open reproduction of state-of-the-art visual language model. httips://hugginngfacee
f fides cs, 2023.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pp. 787-798, 2014.
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. In NewrlPS, 2024.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In CVPR, pp. 317-325, 2017.
12

---

Published as a conference paper at ICLR 2025
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-73, 2017. 2017.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomms Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Ohelics: An open web-scale filtered dataset of interleaved image-text documents. In NeurlPS, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICMI., pp. 12888-18 1898--12000, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742, 2023a.
Tianhong Li, Dina Katabi, and Kaiming He. Self-conditioned image generation via generating representations. arXiv preprint arXiv:2312.03701, 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint urXiv:2305.1035: 2023c.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2024a.
Xiaohao Liu, Xiaobo Xia, Zhuo Huang, and Tat-Seng Chua. Towards modality generalization: A benchmark and prospective analysis. arXiv preprint arXiv:2412.18277 2024b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, pp. 11976-11986, 2022.
Ziqiang Liu, Feiteng Fang. Xi Feng. Xinrun Du, Chenhao Zhang, Zekun Wang. Yuelin Bai, Qixuan Zhao, Liyang Fan, Chengguang Gan, Hongquan Lin, Jiaming Li, Yuansheng Ni, Haihong Wu, Yaswanth Narsupalli, Zhigang Zheng, Chengming Li, Xiping Hu, Ruifeng Xu, Xiaojun Chen, Min Yang, Jiaheng Liu, Ruibo Liu, Wenhao Huang, Ge Zhang, and Shiwen Ni. li-bench: An image implication understanding benchmark for multimodal large language models, 2024c.
Run Luo, Zikai Song, Lintao Ma, Jinlin Wei, Wei Yang, and Min Yang. Diffusiontrack: Diffusion model for multi-object tracking. arXiv preprint 2r0iv:2308. 09905, 2023.
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, et al. Mmevol: Empowering multimodal large language models with evol-instruct. arXiv preprint SxXiv:2409.084444 2024.
Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. Openomni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis. arXiv preprint arXiv:2501.044 04561, 2025.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, pp. 11-20, 2016.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, pp. 3195-3204, 2019.
13

---

Published as a conference paper at ICLR 2025
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pp. 947-952, 2019.
Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyeonsu Kim, Hyoungwon Cho, Seyeon Kim, and Seun- gryong Kim. Diffusion model for dense matching. arXiv preprint urXiv:2305. 19094, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint wr.iv:2112.107444 2021.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. In NeurlPS, 2011.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 2023.
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connect- ing vision and language with localized narratives. In ECCV, pp. 647-664, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, pp. 8821-8831, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, pp. 5389-5400, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, pp. (p.36499--64944 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training generation image-text models. In NeurlPS, pp. 25278- 25294, 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, pp. 146-162, 2022.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, pp. 742-758, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pp. 8317-8326, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 134466, 2020.
14

---

Published as a conference paper at ICLR 2025
Zikai Song, Ying Tang, Run Luo, Lintao Ma, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Autogenic language embedding for coherent point tracking. arXiv preprint arXiv:2407.2073, 2024.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.1322 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In ICLR, 2023b.
Changyao Tian, Xizhou Zhu, Yuwen Xiong. Weiyun Wang. Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401. 10208, 2024.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms, 2024.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, pp. 15566-4575, 2015.
Zhaoqing Wang, Xiaobo Xia, Ziye Chen, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. vocabulayy segmentation with unpaired mask-text supervision. arXiv preprint arXiv:2402.08866. 2024.
Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, and Jiahui Yu. De-diffusion makes text a strong mross-modal interface. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 13492-13503, 2024.
Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280. 2022.
Weilai Xiang, Hongyu Yang. Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15802-15812. 2023.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2955-2966, 2023.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hermandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv: 043000 2019.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding. arXiv preprint arXhv:2307.0244 0299992999999.... 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregeessive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2(3), 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning. PMLR, 2024.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, pp. 77311 2019.
15

---

Published as a conference paper at ICLR 2025
Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXiv: 2311.04498, 2023a.
Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling-Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-driven selective annotations empower in-context learners in large language models. arXiv preprini arXiv:2310. 10873, 2023b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2024.
Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on vision-language models. Advances in Neural Information Processing Systems, 37: 3122-31 3156, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. In NeurIPS, 2024.
16

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
2
Method
2.1
Architecture
2.2 Pipeline
2.3 Training and Inference
3
Experiment
3.1 Implementation Details
3.2 Experimental Results
3.3
Ablation Study
8
4
Related Work
4.1
Diffusion Models for Representation Learning...
4.2
Large Multimodal Model
10
5
Conclusion
10
Acknowledgments
Limitation
18
Broader Impacts
18
Additional Experiments Results
18
Additional Ablation Study
19
D.I Ablation Study of Input Image Resolution
19
D.2 Ablation Study of Training Recipes
19
Additional Implementation Details
19
E.1 Dataset Construction
19
E.2
Image-Text Alignment Pre-training
20
E,3
Image-Texx Text Instruction Fine-tuning
20
E,4
Mask-Text Instruction Fine-tuning
E.5
Evaluation
21
Additional Visualization Examples
E1 Semantic Image Synthesis
21
F2 Text Condition Image Synthesis
23
F3 Robustness Comparison
24
17

---

Published as a conference paper at ICLR 2025
F.4 Image-Text Multimodal Dialogue
24
F.5
Mask- Text Multimodal Dialogue
24
A
LIMITATION
Although our method significantly enhances the visual robustness of interleaved image-text model- ing multimodal models after image-text alignment pre-training, it, unfortunately, cannot eliminate but only alleviate the robustness knowledge forgetting issue caused by subsequent fine-tuning, as shown in the Table Additionally, our model requires using a diffusion model as another eye to correct and update the erroneous knowledge of the image encoder to improve the overall visual bustness of the multimodal model. However, updating larger image encoders such as CLIP- VIT-L and CLIP- ViT-G et al., 2021) will increase the training overhead, which may limit the ap- plication of our model. We hope that in the future, the diffusion model can completely replace the image encoder further enhance the effectiveness of our method.
B BROADER IMPACTS
The proposed method introduces a novel strategy to enhance the robustness and generalization capa- bilities of multimodal models by leveraging diffusion model as additional eye for large language models. This strategy allows for the correction and updating of potential semantic errors in the im- age encoder, leading to significant improvements in handling out-if-distributtin data and mitigating visual hallucinations. Overall, our contributions provide significant step forward the field of multimodal, offering robust, efficient, and scalable solution for improving the accuracy and liability of multimodal models. The broader impacts of this work include the potential for more intelligent and adaptive systems that can operate effectively in diverse and challenging environ- ments.
Table 6: Zero-shot region-level image captioning results on ReferCOCOg.
Method
Type METEOR
CIDEr
GRIT (Wu et al, 2022)
Box
15.2
71.6
Kosmos-2 (0-shot) (Peng et al., 2023)
Box
12.2
60.3
Kosmos-2 2 (2-shot) (Peng al., 2023)
62.2
Kosmos-2 (4-shot) (Peng et al., 2023)
Box
14.1
62.3
NeXi-Chat (Zhang et al., 2023a)
Box
12.0
79.0
DEEM -Musk
Mask
14.1
71.0
C ADDITIONAL EXPERIMENTS RESULTS
Table 7: Zero-shot text-to-image gener- ation FID on MS-COCO and IN-COCO.
Region-Level Image Captioning. In addition to holis- tic image understanding, we also validate the model's ability take region-level image captioning. As shown in Fig. 3, we use a mask-aware extractor to obtain region-level visual features and address region-level image captioning tasks. We adopt the RefCOCOg (Mao al., 2016) validation set and compare it with other state-of-the-art (SOTA) models, including GRIT (Wu al., 2022), Kosmos-2 (Peng al., 2023), and NeXt- Chat (Zhang et al., 2023a). The CIDEr (Vedantam al., 2015) and METHOR are applied as the evalua- tion metrics. As shown in Table 6, our model is capable of achieving competitive performance on CIDEr and METEOR across all of the compared methods, which shows the superiority of our DEEM.
Method
MS-COCO LN-COCO
Test-to-lmugge Specialisty
Retrieval Reselt
17.97
33.59
DALL-E (Ramesia
CogView View (Ding al. 2021)
27.10
Cog Cog View2 (iing et al., 2022)
24.00)
Stable Dillusion (Rombach al., 2022)
12:43
31,26
GLIDE (Nichul
2021)
12.24
Make- Scene (Gafui
2022)
11.84
DALL- (Ramesh et 2022)
Muse-BB (Ying et al.. 2019)
7.8K
Imagen-3.4 3.B (Sahania et 2022)
7.27
Parti 20B (Yu et al., 2022)
7.23
15.97
Models for Booh Image ouf Textt Genezation
CMB3 13B (Adbajanyan
2022)
29.56
GILL-SB (Koh al., 2024)
Emu-1 33 (Son al., 20236)
11.66
CMBLeon- 7B (Yu
2023)
10.82
DreamLL 7D (Dong et al., 2023)
8.76
22.42
DFEM -78 (Ouns)
8.89
24.13
Text-to-Image Generation. et al., 2014) and LN-COCO (Pont-Tuse Tuset et al., 2020). On MSCOCO, we sample 8 images per text condition and use CIIP--VI-LL (Radford et al., 2021) to rerank based text-image similarity. CLIP
evaluate text-conditional image generation on MS-COCO (Lin
18

---

Published as a conference paper at ICLR 2025
Table 8: Ablation study of input image resolution and coefficient À with 2k training steps and 16 batch size.
SFT resolution À
RVQA-A RVQA-R RVQA-V POPE-R POPE-P POPE-A OK-VQA
256
51.6
52.0
49.6
48.5
50.0
50.0
18.9
X
256
5
51.9
54.3
52.7
48.4
50.1
50.0
20.7
x
256
10
51.7
52.7
51.9
48.7
50.3
50.3
20.1
256
5
51.5
59.1
57.9
85.9
77.1
76.4
38.7
/
448
5
52.5
57.6
58.1
86.0
79.2
77.1
41.0
reranking is not used for LN-COCO. FID (Heusel al., 2017) is used evaluate both datasets. As shown in Table 7, our model shows competitive text-to-image generation compared to existing image and text generation models. qualitative results on text-to-image synthesis in Fig. 10 in Appendix F.
D ADDITIONAL ABLATION STUDY
we provide more ablation studies for DEEM in this section, all which share the same settings, All the code, models, and data tools will be released soon.
D.1 ABLATION STUDY OF INPUT IMAGE RESOLUTION
In addition the aforementioned exploration, also scale up the input image resolution for perfor- mance gain. The performance gain becomes larger when further increasing the input image resolu- tion from 256 to 448 image-text instruction fine-tuning, shown in Table 8. Such results indicate our method could better exploit the additional information gained from high resolution. Moreover, we conduct an ablation study on coefficient À in loss function. As shown in Table setting À = achieves a better balance between robustness and hallucination empirically.
D.2 ABLATION STUDY TRAINING RECIPES
We also conduct ablation study to control the trainability of different training modules. As shown Table 10, we found that freezing the DM (Diffusion Model) while not freezing the VFM (Visual Foundation Model) during training yields the best robustness and hallucination results.
E ADDITIONAL IMPLEMENTATION DETAILS
E. DATASET CONSTRUCTION
As shown in Fig. 5, we first convert the original ImageNet-A (Hendrycks et al., 2021b), ImageNet- R (Hendrycks et al., 2021a), and ImageNet-V V2 (Recht et al., 2019) data into a VQA format that
Table 9: Comparison of different VQA formats. Questions in the or no format can well eval- uate the performance of the models on the RobustVQA benchmark, while questions in the multiple- choice format are very random, and MM-interleaved tend to output the first option. Therefore, we adopt yes or no format in our experimental settings. More details about the new benchmark Ro- bustVQA design can be found in Appendix E.I.
Prampe
RebusVQA-A RobunVQQ-R RabuiVQA-V
mikiipe--hhiie "Kowt te the main object this Inage?"
"chose fron the iists Ifaise cateoorv 1tbel, catacerv label]."
miltiple choice
"Kast in the main object in this inage?"
84.00
90.16
82.92
"Coose the Jists estegory Labe., category labell."
"Ia eategory label] the abjeec this inage?"
yes
"please answes yeu no."
"Is Itolse category label] the main octect in this inags?"
"please answes yeu no."
19

---

Published as a conference paper at ICLR 2025
the multimodal model can evaluate. Specifically, we use the CLIP-ViTH model for hard exam- ple mining, predicting the incorrect category label with the highest confidence score apart from the ground truth category label. We then use a pre-defined prompt as: "Is [category iabel] the ma main n ob object in this image? Please answer yes or no. " to simultane- ously construct pair of positive and negative example samples, allowing the model to answer "yes" or "no". By using this design, we can evaluate the robustness of multimodal models in an unbiased manner with the new benchmark called RobustVQA, facilitating both assessment and comparison. It is worth noting that, as shown in Table 9, we find that the yes or no format is more stable than the multiple-choice format and can better evaluate the robustness of mulli-modaa models.
ImageNet-like Format
VQA Format
Category Collection: /Sealiinn, 'Drake', 'Goose', KKiller whale', ......)
Scalion
Drake
Number of C'Cleggories
A photo of Goose
Incorrect category
Highest confidence
Is Sealion the main object in this image?
Negutive
Please answer yes or no.
no
C.
In Goose the main object this image? yes
Positive
Pleuse answer yes or no.
Example Mining
208
0144
Soite ON ICA
Figure 5:
Robustness dataset construction process. We use the CLIP-ViT-L model for hard ex- ample mining and then transform them into question-answer pairs via pre-defined template.
E.2 IMAGE-TEXT ALIGNMENT PRE-TRAINING
We use MMC4-Core (Zhu et al., 2024), LAION-400M (Schuhmann et al., 2021). SBU (Ordonez et al., 2011), and CC-12M (Changpinyo al., 2021) as the pre-training dataset. For LAION-400M (Schuhmann al., 2021), SBU (Ordonez et al., 2011), and CC- 12M (Changpinyo et al., 2021), instead of utilizing the original annotations, we use the version filtered by the pre-trained BLIP-2 model (Li et al., 2023a). For simplicity, refer to it as BLIP- LCS hereafter. "LCS" abbreviates the LAION, CC, and SBU datasets. Text prompts with lengths shorter than 10 are also filtered out. Due to network constraints, only collect approximately 6M of MMC4-Core and 20M of BLIP-LCS data. The sampling probability of MMC4 is twice that of BLIP-LCS. The images are inserted before or after the corresponding text sentence with equal probability. Specifically, images with a CLIP similarity score below 0.24 will be discarded, and only 6 images most will be kept for each document in MMC4-Core. We also exclude 100% of all documents that do not contain any images, and 50% of documents that contain only 1 image. For amage-text-pait BLIP-LCS datasets, we randomly sample multiple image-text pairs from the same dataset and concatenate them to the maximum context length (i.e., 2048) during pre-training. For interleaved image and text MMC4-Core (Zhu et al., 2024) datasets, we also split and concatenate the documents to form the training samples. Such a concatenation strategy can utilize the full context window of Large Language Models and thus achieve high data efficiency. Besides that, for image generation, we ignore the training loss of images which are the first element in the sequence. The text condition of the rest images is dropped with a 10% probability to improve classifier free guidance sampling. The detailed hyper-parameters of image-text alignment pre- pre-training listed in Table 11.
Table 10: Ablation study of training recipe in image-text alignment pre-training with 10k training steps and 128 batch size.
VPM DM
RVQA-A
RVQA-R
RVQA-V
POPE-R
POPE-P
POPE-A
frecze unfreeze
53.2
56.1
$4.7
50.6
52.0
51.9
unfreeze freeze
56.8
70.3
unfreeze unfreeze
50.3
52.5
53.1
54.8
56.3
56.1
E.3 IMAGE-TENT INSTRUCTION FINE-TUNING
We utilize public available datasets for supervised fine-tuning, including LLaVA-665K(Liu et al., 2024a), COCO Caption (Chen et al., 2015), VQAv2 (Goyal et al., (007)).extCCpp (Sidorov et al., 2020), -OR-VQAA (Mishra et al., 2019), GQA (Hudson & Manning, 2019), -KA (Marino et al., 2019), TextVQA (Singh et al., 2019), and AOK-VQA (Schwenk et al., 2022). We use the fol- lowing prompt template "Based on the image, please answer the question. (image) (question). The answer is: (answer) to convert the data into mix-
20

---

Published as a conference paper at ICLR 2025
ture of instruction following forms, resulting in approximately 800K instruction data for the second- stage image-text instruction fine-tuning. The detailed syper-parametery of image-text instruction fine-tuning are listed in Table 11.
E.4 MASK-TEL TEXT INSTRUCTION FINE-TUNING
We collect short text and pixel-level mask pairs from the publicly available object-level datasets (COCO, RefCOCO, RefCOCO+) and part-level datasets (Pascal Part, Part Imagenet), then transform them into instruction following data. Moreover, Visual Genome (VG) and Visual Commonsense Reasoning (VCR) datasets are employed to add more multiple region understanding data, resulting in approximately 200K instruction data for the third-stage mask-text instruction fine-tuning. See more cyyper-parameterr details in Table 11.
E.5 EVALUATION
As shown in Fig. 6, DEEM achieves the best results on both hallucination and robustness bench- marks even the smallest scale, demonstrating the efficiency and effectiveness of our approach. In addition to visual robustness and hallucination, we also use various benchmarks and datasets, such image caption, visual question answering, text-to-image generation and so on, assess the image-text comprehension capabilities. All these evaluation tasks and metrics are listed in Table 12. The prompt templates for each task are listed Fig.
RobustVQA-A
Model
MM-Interleaved Stage1 (VIT-L/Micuaa 13B)
NEXT-Chat (VIT-LMicuns 7B)
POPE-A
RobustVQA-R
Shikra (VT-L/LLaMA 7B)
LLaVa (Vi--L/icuna 138)
DEEM (ConwNext-BVVicuna 7B)
POPE-P
RobustVQA-V
POPE-R
Figure 6: Performance on visual robustness and hallucination benchmark. DEEM achieves the best results robustness benchmark and competitive performance hallucination even the smallest scale, demonstrating the efficiency and effectiveness of our approach.
F ADDITIONAL VISUALIZATION EXAMPLES
F.1 SEMANTIC IMAGE SYNTHESIS
Dynamic Semantic Bias Frasure. We demonstrate the dynamic semantic bias elimination pro- cess through three iterations on the same sample, providing an illustration of the original im- age alongside its version reconstructed in real-time according to semantic conditions, shown Fig. 9. Our method, DEEM, gradually mitigates potential erroneous semantics within the visual encoder through multiple iterations, ultimately enhancing the perceptual capabilities of MLLMs.
Consistency Semantic Image Synthesis We visualize consistency semantic image synthesis and display both the original images and their reconstructed versions in Fig. 11. DEEM accurately recovers the features of the original images without causing distortion.
21

---

Published as a conference paper at ICLR 2025
Visual input example, Goose:
User
Based on the image, please answer the question. What is the main object in this image?
Chose from the list: ['Scalion', 'Drake', 'Goose', 'Killer whale"].
Let's think step by step.
DEEM (Ours)
A carved goose sitting flat surface. So the answer is goose
LLaVA
The main object in this image is drake. X
NExT-Chat
A sculpture of a duck sitting on a table. So the answer is duck. X
MM-Interlcaved
Sealion X
Figure 7: Case Comparison. Compared to other SOTA models, including LLaVA, NeXt-Chat, and MM-Interleaved, when encountering out-of-distribution data, their models are affected by incorrect semantics from the image encoder and cannot output the correct answer. However, DEEM can output the correct answer via generative feedback.
Table 11: Training recipes for DEEM The three training stages are introduced in Section 2.3. Stage I: Image-Text Alignment Pre-training, Stagg II: Image-TTex Text Instruction Fine-tuning,
Stage III: Mask-Text Instruction Fine-tuning.
Stage
Stage
Stage
Phase
Image- Text Alignment
Image-Texe Text
SFT SET
Mask-Toxt SFT
Training Sxpee-PPorometer
Input image resolution
256x
448x4488
448 x448
Output image resolution
512x512
512x512
$12x512 CLIP.ConwNext-B Vicuna-7B v1.5
VFM
CLIP-ComwNext-B
CLIP-ConwNext-B
LLM
Vicuna-7B v1.5
Vicuna-7B v1.5
DM
Stable Diffusion v2.1
Stable Diffusion v2.1 5
Stable Diffusion v2.1
A
2o-5 (image encoder&deccoder) 1c-4 (others) Adam
1e-6 (language model) le-5 (others) AdamW
le-6 (language model) le-5 (others) Adamm 0.9, 0.999, 0.05 10k
Leamiig Rate
Optimizer
Optimizer hyper pare-parameters
91, 92,4 1.995.1e-6
0.999.1e-8
Ar
Weight Decay
0,05
0.05
Training leerations
10k
10k
Warmup steps
1k
500
500
Leamiing Rate Scheduler
Cosine
Cosine 16
Cosine 2
Batch Per GPU
Maximum Token Length
2048
2048
2048
Augmentation
CenterCrop
Unfreeze LLM
/
Unfreeze
X
Unfreeze VFM
x
Training Data
MMC4 @ BLIP-LCS
① LLaVA Mix 665K
2 VQA-Mixture
@ Bascal-Par/Put--lnagee @ VG/VRC
Datasel
& COCO Caption
Data Size
-26 Interleave/Pair
800K Instruction
~00K Instruction
Data Type
Trnining
GPU Device
32X A100
32x NVIDIA A100 66
NVIII A
Training Time
300
~3h
22

---

Published as a conference paper at ICLR 2025
Table 12: Overall descriptions of the evaluation benchmarks for evaluating capabilities, includ- ing image-level captioning, image-level visual question answering, text-to- ext-to-image generation, region- level image captioning, visual robustness, comprehension, pereeption and hallucination.
Dutaset
Task description
Eval Split
Metric
CAP
00C0 (Chen et al, 2015)
Scene description
test
CIDEr(t) (Vedantam al., 201.5)
InageeParagrah (Krause al., 2017)
description
teat
CIDEr(") (Vedantam al. 2015)
VQAv2 (Goyal
2017)
understanding
teat-dev
VQA Ace(f) (Antol
2015)
VOA.
OKVQA (Marino et al, 2019)
External knowledge QA
vall
VQA Acc(f) (Antol al., 2015)
GQA (Hudson & Manning, 2019)
Scene understanding QA
teat-dev
VQA Acc(f) (Antol al., 2015)
VizWiz (Gurari al., 2018)
Scene understanding
teat-cev
VQA Acc(1) (Antol 2015)
VisDial (Das et al., 2017)
Image dialogue
val
NDCG(1)
SYN
MS-COCO (Lin et al. 2014)
Text- Conditional Image Synthesis
va1-30K val
FID(.) (Heusel et al., 2017)
LN-COCO (Pont-Tusee Tuset al., 2020)
Text-C Cot-Coondtional Image Synthesis
FID()) (Heusel et al., 2017)
REF.
ReiCOCO+ (Mao al,, 2016)
RefCOCO (Kazemzadeh et al., 2014)
Region-Ievel scene description
Region level scene description
val
val
CIDEr(T) (Vedantam al., 2015)
CIDEr(1) (Vedantam al., 2015)
RefCOCOg (Mao et al., 2016)
Region-level scene description
val
CIDEn(") (Vedantam et al., 2015)
RobustVQA-V
cittof-Distribution Robustness
val Acc(f)
OOD.
RoA-stVQA-R
Out-of Distribution Robustness
Acc(*)
RobustVQA-A
Distr--Dittibttioonn Robustness
val Acc(1)
POPE-R (Li et 2023c)
Visual Hallucination
val Acc(f)
PCP. CPH Hall
POPE-P (Li al., 2023c)
Visual Hallucination
val
Acc(1)
POPE-A (Li al., 2023c)
Visual Hallucination
Acc(")
MMBench
2024)
Visual Comprehension
val Acc(")
MMVet (Yu al., 2024)
Visual Comprehension
Acc(1)
MMVP (Tong 2024)
Visual Perception
Acc(1)
System Prompt
helpfful assistant.
(6) VQA
USER:
image,
[IMAGEHQUESTION] Please provide within word.
ASSISTANT:
The answeris
<ANSWER>
(b) Image Coptioning
USER: [IMAGE] yould a description Image?
ASSISTANT: Aphoto of <ANSWER>
(e) Referiing
USER: [IMAGE] Provide short description for this <ref>region1</ref>
[MASK]
ASSISTANT: <ANSWER>
(d) Hallacination
USER: Based image,
Is there a <OBJECT> In Image? Please answer yes no.
<ANSWER>
Robustness
USER:
Based image,
question.
(IMAGE)
Is <OBJECT> the main object In this Image? Please answer yas or no.
ASSISTANT:
The answer
<ANSWER>
Figure 8: Prompt template used for evaluation. (a) VQA includes VQAv2, VizWiz, OKVQA, GQA, VisDial, and MMVP. (b) Image Captioning includes COCO. Image2Paragraph. (c) Region- level Image Captioning includes RefCOCOg. (d) Visual hallucination includes POPE. (e) Visual Robustness includes Rob,stVQA-A, RobustVQA-R, and RobustVQA-V.
IMAGE >denotes the input image representation. < MASK > denotes the mask-level image representation, < QUESTION >denotes each specific question, < ANSWER is the generated answer, and < OBJECT > is the specific object name in question of POPE and RobustVQA.
F.2 TEXT CONDITION IMAGE SYNTHESSI
In Fig. 10, we present some text-to text-to-image synthesis examples from DEEM, demonstrating its capa- bility to generate corresponding images based on given prompts.
23

---

Published as a conference paper at ICLR 2025
loss
duck
goose sed duck
gposs gea
Figure 9: Dynamic semantic bias elimination process through three iterations on the same sample, diffusion process is conducted by adding 65% noise to the original image as the initial condition.
F.3 ROBUSTNESS COMPARISON
In Fig. 7, we present comparative analysis of visual robustness results between model, DEEM and other state-of-the-art models: LLaVA (Liu et al., 2024a), NeXt-Chat (Zhang et al., 2023a), and MM--Iterleaved (Tian et al., 2024). When encountering natural adversarial samples or out-of- distribution samples, the image encoder in their models will output incorrect semantic information, leading incorrect category answers. In contrast, our method uses diffusion model the eyes of the large language model to inspect and correct the output features of the image encoder. This process eliminates incorrect semantic outputs from the image encoder, ultimately allowing the large language model to produce the correct category answer. This simple yet effective approach signifi- cantly enhances the model's s robustness and generalization capabilities.
F.4 IMAGE- IMAGE-TEXT MULTIMODAL DIALOGUE
In Fig. 12, show the image-text dialogue examples of DEEM Our model input any interleaved layout of text-image data and simultaneously understand and generate text-image outputs any interleaved layout, representing the future of next-generation multimodal dialogue.
F.5 MASK- TEXT MULTIMODAL DIALOGUE
In addition to image-level input, DEEM also supports mask-text input to perform fine-grained region-level reasoning tasks. As shown in the Fig. 13, DEEM can accurately extract region se- manties of the image based the input mask and complete the corresponding instruction tasks.
24

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
(e)
Figure 10: image-to-image generation examples with the outputs of image encoder. (a.c,e) are original images and (b,d,f) are synthesis images based on the image embeddings of original images.
25

---

Published as a conference paper at ICLR 2025
Small bird with a pale yellow underside light brown crown and back gray tail and wing tips tip of tail feather bright yellow black eyes and black strip over eyes
An armchair in the shape of an avocad
A couple of glasses are sitting on a tabl
long wooden bench in front of a brick wall
Downtown Beijing at sunrise. detailed ink wash
A beat-up truck in the desert
Figure 11: Text-to-image generation examples with prompts. DEEM can generate vivid images based on input text conditions.
26

---

Published as a conference paper at ICLR 2025
Photograph A:
deem
Photograph B:
Deenn
What is written in photo A and B? What is the difference between these two photos? Answer detail about the difference written words and styles.
pink In Photograph A, the word \"DEEMY" written background with hearts placed around This creates an eye- catching and unique display for this particular image of roses against another colorful rose patterned backdrop that complements DEEM's bright colors. On the other hand, in Photograph B, the same words are displayed using vintage typography style and printed
light blue gradient. The overall effect looks more nostalgic or dreamy compared to Photograph but has similar visual elements such as flowers arranged beneath DEEM. These differences highlight how design choices can impact mood and perception while retaining essential components like textual content.
help me mix the styles the above two images create a new image that combines both modern and romantic styles featuring the word "DEEM":
Deem
This image merges the sleek, contemporary elements with soft, pastel romantic touches, creating unique blend of both aesthetics.
Figure 12: Examples of image-text multimodal dialogue between human and DEEM Text and image can be used as inputs outputs, and multi-round dialogue is shown.
27

---

Published as a conference paper at ICLR 2025
Can you provide me with a brief descripiin of region in the picture marked by reif--eiionn//eef please?
Give short description <reforegian</ref-.
lake on the bottom
a white heron on the right
I'm curious about the region represented by rref-repion</ref the picture. yould describe
What can you give me about the region outlined <ref-region</ref the photo?
an elophant in the middle
baby elepahnt
Can give brief introduction of reglon <reffrreiion//eet picture?
Could you describe Peeiionns/eee picture concisely?
vases theright
television middle
Can you give me short description of <ref>reglon</eef>
description region mar Red with <ref>region</reh in the image.
an orange table lamp lamp
Figure 13: Examples of mask-text multimodal dialogue between human and DEEM Text and mask can be used as inputs and DEEM outputs the corresponding answer, and multi-round dialogue shown.
28


Reviews:
[{"review_id": "zOaSXYyd2B", "reviewer": "Reviewer_Sncp", "summary": "This paper introduces the diffusion model into the image perception process of large language models to reduce excessive compression of visual details, thereby enhancing the model's robustness and its ability to mitigate hallucinations. An end-to-end multimodal model with interleaved text-image modeling capabilities is trained to perform various multimodal tasks in a unified manner. A new robustness benchmark for large multimodal models is developed to assess their visual robustness capabilities. The extensive experimental results demonstrate the effectiveness of the proposed method.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a new benchmark, RobustVQA, specifically designed for evaluating the visual robustness capabilities of the large multimodal models.\n2. The proposed method is the first to introduce the diffusion model into the image perception of large language models, which is simple but effective.\n3. The proposed method trained end-to-end with interleaved text-image data can perform various multimodal tasks in a unified manner.", "weaknesses": "1. The insight of the paper is the integration of the diffusion model into the image perception process of LLMs while training to correct potential semantic biases in the image encoder. It is crucial to validate the generalization capability of the proposed method across existing LMMs to enhance their image perception. However, the validation is somewhat limited, focusing solely on LLaVA, as shown in Table 5. Could the proposed method be flexibly applied to additional existing architectures, such as BLIP-2 and MiniGPT-4?\n2. The claim that the proposed method can \"correct potential semantic bias in the image encoder and alleviate excessive compression of visual details\" seems intuitive. Are there any validations or qualitative comparisons between existing LMMs trained with and without this method? For instance, maybe comparisons of feature maps for image tokens encoded by the image encoder trained with and without the proposed method.\n3. The contribution of the constructed RobustVQA benchmark seems somewhat limited, as it focuses primarily on evaluating visual robustness. Emphasizing the unique advantages of the RobustVQA benchmark over existing benchmarks through more extensive qualitative or quantitative comparisons with other benchmarks in some aspects would enhance its value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDMEhYeaK1", "reviewer": "Reviewer_ikTv", "summary": "This paper enhances large language models image perception by using diffusion models to align image encoder semantics, improving resilience to out-of-distribution data and reducing visual hallucinations. Without extra modules or parameters, DEEM outperforms on benchmarks like RobustVQA, POPE, and MMVP, using fewer parameters and less data. It strengthens visual tasks such as question answering, captioning, and image synthesis, showcasing robust perception and reduced hallucinations.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper design a new robustness benchmark RobustVQA for visual robustness capabilities of multi modal models.\n2. DEEM is the first approach to enhance image perception in LLMs using diffusion models, making it novel.\n3. This approach is highly versatile, demonstrating effective experimental results across benchmarks.\n4. This paper is well written and easy to follow.", "weaknesses": "1. Since diffusion models need to be utilized during both training and inference, the computational cost will be high.\n2. If this model is used with an LLM that has a much larger number of parameters, how would its performance be affected?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TmgTM0iwkK", "reviewer": "Reviewer_9Tws", "summary": "This paper proposed a method to enhance the image perception ability of large multimodal models (LMMs), by leveraging the generative feedback of diffusion models. This reduced the visual hallucination and strengthen the perception ability in against out-of-distribution samples. Also a benchmark is developed, specifically designed for out-of-distribution or challenging samples. Extensive experiments showed that the proposed method achieves an improvement in a collection of benchmarks and downstream tasks.", "questions": "Q1) Regarding RobustVQA, would it be possible to also run on close-source models (GPT4o/Gemini1.5Pro)? I am curious on how the close-source solutions perform against the proposed benchmark.\n\nQ2) In Table 9, it states \"Questions in the yes or no format can well evaluate the performance of the models on the RobustVQA benchmark, while questions in the multiple choice format are very random, and MM-interleaved tend to output the first option. Therefore, we\nadopt yes or no format in our experimental settings.\" seems a bit unconvincing to me. Would LMM also be easier to guess the solution if there is only 2 choices for yes/no format?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper is well-structured and it shows a good originally. This is a pioneering work in using diffusion models to specifically enhance the image perception capabilities of LMMs.", "weaknesses": "W1) The detail information of RobustQA is missing. This makes the experiments result look less convincing. How many samples are in the benchmark? Please also include a diversity chart to show how robust the benchmark is.\n\nW2) Figure 2 is difficult to follow. Please revise the figure, e.g. adding some step indicators, legends for arrows.\n\nW3) Typo \"obnly\" in the last paragraph of Section 3.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lybhC47Rip", "reviewer": "Reviewer_CASC", "summary": "The DEEM model integrates diffusion models with large multimodal models (LMMs) to enhance image perception capabilities, especially for out-of-distribution data. DEEM employs generative feedback from diffusion models to correct potential semantic biases in image encoders, resulting in more accurate and robust visual perception. Evaluations across benchmarks such as RobustVQA, POPE, and MMVP demonstrated improvements in visual robustness and reduced visual hallucinations compared to traditional methods. This enhancement is achieved without additional training parameters, using fewer resources and a smaller model scale.", "questions": "1. Could the diffusion model’s role be optimized or reduced without compromising robustness? For instance, could a selective application of diffusion-based feedback lower training costs?\n2. Could the impact of scaling down the diffusion model be evaluated further to assess how smaller models might perform in balancing robustness and efficiency?\n3. Have you considered a comparative analysis of DEEM with approaches like \"De-Diffusion Makes Text a Strong Cross-Modal Interface\"?\n\nhttps://openaccess.thecvf.com/content/CVPR2024/papers/Wei_De-Diffusion_Makes_Text_a_Strong_Cross-Modal_Interface_CVPR_2024_paper.pdf", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tInnovation in Multimodal Integration: DEEM introduces diffusion models as “visual auditors” for LMMs, a novel approach for refining visual inputs and reducing errors from biased image encoders.\n\t2.\tEfficiency: The approach reduces dependency on larger, more complex models by using a smaller image encoder (CLIP-ConvNext-B), which lowers computational requirements.\n\t3.\tRobustness: Demonstrates resilience against adversarial and out-of-distribution data, enhancing the model’s suitability for diverse, real-world scenarios.\n\t4.\tExtensive Evaluation: The paper provides a comprehensive analysis using custom (RobustVQA) and widely recognized benchmarks (POPE and MMVP), showing DEEM’s performance gains in both robustness and visual perception.", "weaknesses": "1.\tReliance on Diffusion Models: Although diffusion models enhance robustness, they also introduce computational complexity. Future versions might explore alternative approaches that achieve similar robustness with lower computational costs.\n\t2.\tGeneralization Limits: While DEEM shows strong performance for specific multimodal tasks, its generalization to other modalities (e.g., audio) remains unexplored.\n\t3.\tTraining Overheads: The training process, though optimized, may still be prohibitive for some use cases where computational resources are constrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OYzknJQ7H6", "reviewer": "Reviewer_qAMc", "summary": "This paper introduces DEEM, a method that leverages the generative feedback of diffusion models to align the semantic distributions of image encoders, addressing issues faced by large multimodal models (LMMs) when encountering out-of-distribution data such as orientation, quantity, color, and structure. DEEM improves the resilience of LMMs against such data and reduces visual hallucinations without the need for additional training modules and with fewer training parameters. It demonstrates significant improvements in visual perception performance across several benchmarks (RobustVQA, POPE, and MMVP), exhibiting enhanced robustness and superior capability to mitigate model hallucinations while using fewer trainable parameters and less pre-training data, and being smaller in base model size. Extensive experiments show that DEEM enhances the performance of LMMs across various downstream tasks, including visual question answering, image captioning, and text-conditioned image synthesis.", "questions": "**Questions for the Authors**\n\n1. **Scalability and Computational Efficiency**:\n   - How does the use of diffusion models as an additional component impact the overall computational efficiency and memory usage, especially for larger versions of image encoders?\n   - Have you explored any strategies for optimizing computational resources? If so, could you provide details on these optimization techniques and their effectiveness?\n\n2. **Robustness Knowledge Forgetting**:\n   - You mentioned that DEEM cannot completely eliminate robustness knowledge forgetting. Do you have insights or preliminary results on potential methods for further mitigating this issue?\n\n3. **Failure Case Analysis**:\n   - Could you provide more details or examples of cases where DEEM underperforms compared to other state-of-the-art methods?\n   - What common patterns or factors have you identified in these failure cases, and what potential solutions might address these issues?\n\n4. **Impact of High-Resolution Inputs**:\n   - How does the increase in input image resolution from 256 to 448 pixels impact the performance and training dynamics? Are there diminishing returns at higher resolutions?\n   - Have you evaluated the impact of image resolution on the computational requirements and time to convergence?\n\n5. **Diffusion Model's Role**:\n   - Could you elaborate on how the diffusion model corrects and aligns the semantic outputs of the image encoder during training? Are there specific examples or visualizations that illustrate this process?\n   - How critical is the specific architecture or configuration of the diffusion model in achieving these results?\n\nIn summary, while the paper proposes combining diffusion models with LLMs to enhance the model's perceptual capabilities, this approach is not entirely novel, as seen in works like Transfusion (https://www.arxiv.org/abs/2408.11039). Additionally, if the sole purpose is to enhance the model's visual perception, introducing such a complex diffusion module may seem excessive. If a diffusion module is introduced, the authors should also focus on evaluating image generation capabilities. Alternatively, the authors should attempt to use simpler methods as baselines to highlight the necessity of introducing the diffusion module. The ablation studies in the paper do not seem to cover this aspect.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Strengths**\n\n1. **Originality**:\n   - The paper presents a novel approach, DEEM, which integrates diffusion models to enhance the visual perception capabilities of large language models (LLMs) for multimodal tasks. \n\n2. **Quality**:\n   - The paper provides comprehensive empirical evidence to support the effectiveness of the proposed method. The authors perform extensive evaluations on newly constructed and well-known benchmarks (e.g., RobustVQA, POPE, and MMVP), demonstrating the method's robustness and performance improvements.\n\n3. **Clarity**:\n   - The paper is well-structured and clearly written, providing a detailed description of the DEEM architecture, training process, and experimental evaluation.\n\n4. **Significance**:\n   - DEEM significantly enhances the robustness and visual perception capabilities of LMMs, addressing a critical challenge in the field of multimodal machine learning.", "weaknesses": "**Weaknesses**\n\n1. **Scalability Concerns**:\n   - While DEEM demonstrates improved performance using fewer training parameters and a smaller base model size, the requirement to employ diffusion models as an additional component could potentially increase computational overhead.\n   - As the model size continues to grow, will the computational burden brought by the diffusion models outweigh the performance gains? After all, as the model size increases, its capabilities significantly improve, and computational costs become the primary bottleneck. The introduction of diffusion might bring negative returns.\n\n2. **Robustness Limitation**:\n   - Although DEEM significantly enhances visual robustness, the paper acknowledges that the method cannot completely eliminate but only alleviate robustness knowledge forgetting during subsequent fine-tuning stages. This indicates a potential long-term limitation in maintaining the enhanced robustness through the entire lifecycle of model updates and task-specific fine-tuning.\n\n3. **Limited Analysis of Failure Cases**:\n   - The paper largely focuses on positive results and improvements brought by DEEM. However, it lacks an in-depth analysis of cases where the method might fail or underperform compared to existing approaches. In particular, in the example shown in figure 9 of the supplementary materials, many generated images exhibit significant errors in color, spatial arrangement, and orientation. These generation flaws somewhat undermine the paper's claim that diffusion models can enhance visual understanding, as diffusion models also cannot perfectly restore visual features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)