DEEM: Diffusion models serve as the eyes of large language models for image perception

## Abstract

Abstract The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal comprehension and creation, they often face challenges when confronted with out-of-distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due to their reliance on image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception? In this paper, we propose DEEM , a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP-ViT, thereby enhancing the model’s resilience against out-of-distribution samples and reducing visual hallucinations. Importantly, this is achieved without requiring additional training modules and with fewer training parameters. We extensively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM’s visual perception performance to a large extent (e.g., 4% ↑ on RobustVQA, 6.5% ↑ on MMVP and 12.8 % ↑ on POPE ). Compared to the state-of-the-art interleaved content generation models, DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10%), and a smaller base model size. Extensive experiments demonstrate that DEEM enhances the performance of LMMs on various downstream tasks without inferior performance in the long term, including visual question answering, image captioning, and text-conditioned image synthesis. The code and benchmark are available at https://github.com/RainBowLuoCS/DEEM

## 1 Introduction

With the success of large language models (LLMs), large multimodal models (LMMs) built on LLMs have garnered significant attention. Researchers (Liu et al., ; Zhu et al., ; Dai et al., ; Alayrac et al., ; Chen et al., ) have attempted to build a bridge between large language models and image encoders through simple mapping modules, and have already made significant progress in multimodal understanding tasks such as visual question answering. Subsequent studies (Yu et al., ; Sun et al., ; Dong et al., ; Tian et al., ) utilize extra advanced diffusion models (DMs) (Rombach et al., ) for image generation and train the LMMs on interleaved text-image data in an end-to-end manner. This unified paradigm of multimodal understanding and creation brings various isolated multimodal tasks together, greatly boosting model capabilities and expanding application scenarios.

However, these models commonly rely on encoder architectures like CLIP-ViT (Radford et al., ), which suffers from certain perceptual understanding
limitations due to the contrastive learning paradigm and the noisy image-text pairs used in training, to encode input images. Additionally, these image encoders are typically trained to encode images into features relevant to downstream tasks, thereby disregarding irrelevant details. Consequently, as shown in Fig. 1, when faced with images outside the training scope, they often capture biased semantic features, resulting in erroneous visual information being perceived by subsequent language models. This accumulation of inaccuracies renders the multimodal model unable to comprehend multimodal context effectively. For this reason, this makes it difficult for previous methods to discern subtle details, thereby hindering their ability to handle tasks related to basic visual perception, visual hallucinations, and visual robustness that are very simple for humans.

On the contrary, the goal of diffusion models (Ho et al., ) is to learn a diffusion process that characterizes a probability distribution for a given dataset, without direct training on the downstream task objective. This enables it to capture finer details of images for better handling of out-of-distribution data. However, there have been few efforts to integrate the capabilities of the diffusion model into the image perception of large multimodal models.

In this paper, we propose DEEM, a simple but effective approach to leverage the generative feedback of diffusion models for aligning the semantic distributions of image encoders in an elegant self-supervised manner. Building upon this, we introduce an end-to-end interleaved image-text generative modeling approach, where diffusion models serve as additional eyes of large language models for image perception. This addresses the limitations of previous methods that solely relied on image encoders such as CLIP-ViT (Radford et al., ), enhancing the model’s robustness against out-of-distribution samples and reducing hallucination perception in multimodal scenarios, without the need for additional training modules and with fewer training parameters. To the best of our knowledge, we are the first to apply diffusion models to large multimodal models for image perception.

Specifically, DEEM takes interleaved image-text pairs as input to the model. It starts by encoding images and text using corresponding visual and text encoders, resulting in image tokens and text tokens. These tokens are then organized according to their original layout and inputted into a large language model to generate corresponding hidden state outputs. The model employs autoregressive modeling for the hidden state outputs of text and utilizes the output hidden states of images, along with the image tokens encoded by the image encoder, as diffusion conditions. These conditions are then fed into a diffusion model for image reconstruction. Through end-to-end training, the model not only acquires the capacity to generate text and images but also employs semantic consistency regularization on the semantic information produced by the image encoder during image reconstruction. This compels the image encoder to incorporate more details into the semantic representation of the image, thereby mitigating the issue of semantic bias in image encoding.

DEEM is trained on a mixture corpora of image-text pairs and interleaved image-text sequences data without extra in-house data following previous solution (Li et al., ; ; Dong et al., ; Tian et al., ). To assess the robustness recognition capability of LMMs, we constructed a new robustness benchmark, RobustVQA, based on existing datasets containing natural adversarial samples and out-of-distribution data. RobustVQA is divided into three parts: RobustVQA-A, RobustVQA-R, and RobustVQA-V, based on different data sources, aiming to provide better insights into the performance of LMMs in real-world scenarios. We conducted extensive evaluations of DEEM on RobustVQA and two widely recognized benchmarks, POPE and MMVP, for visual hallucination and perception respectively.
Experimental results indicate that our method exhibits enhanced robustness, a superior capacity to alleviate model hallucinations and better visual perception ability in comparison to the state-of-the-art interleaved image-text modeling model MM-Interleaved (Tian et al., ), using a smaller-scale image encoder (CLIP-ConvNext-B (Liu et al., ) vs. CLIP-ViT-L (Radford et al., )), a smaller-scale language model (Vicuna 7B vs. Vicuna 13B (Zheng et al., )), and less pre-training data (without Laion-coco (Andreas et al., ) & Laion-en (Schuhmann et al., )). DEEM outperforms MM-Interleaved 9.4% on RobustVQA, 17.8% on POPE and 9.1% on MMVP. Moreover, with further enhancement via supervised fine-tuning, DEEM achieves competitive results on various multimodal tasks, including visual question-answering, region-level image captioning, and text-to-image generation.

Before delving into details, we summarize our contributions as follows.

$\bullet$ Robustness Benchmark. We design a new robustness benchmark RobustVQA for LMMs based on publicly available ImageNet-A (Hendrycks et al., ), ImageNet-R (Hendrycks et al., ), and ImageNet-V2 (Recht et al., ) datasets, which can be utilized to effectively assess the visual robustness capabilities of the multimodal models.

$\bullet$ Effective Method. We are the first to introduce the diffusion model into the image perception of large language models, to correct potential semantic bias in the image encoder and alleviate the excessive compression of visual details. This approach enhances the model’s robustness and hallucination mitigation capabilities without the need for additional modules or trainable parameters.

$\bullet$ DEEM Model. Based on the proposed method, we train a multimodal model with end-to-end interleaved text-image modeling capabilities. After supervised fine-tuning, DEEM can perform various multimodal tasks in a unified manner, such as visual question answering, text-to-image generation, and region-level image captioning.

$\bullet$ Comprehensive Experiments. We provide abundant qualitative and quantitative comprehensive experimental results to demonstrate the effectiveness and efficiency of the proposed method.

## 2 Method

In this section, we first present our DEEM , starting with an introduction to the overall architecture in Section 2.1, followed by a description of the pipeline in Section 2.2. Finally, we provide details on the training and inference process in Section 2.3.

### 2.1 Architecture

In this subsection, we present the multi-modal architecture for processing interleaved
image-text data. To excel in both comprehension and creation tasks of text and images, a multi-modal model consists of the following three key components.

VFM-based Image Encoder $\mathcal{E}_{V}$ which encodes each image $x^{V}\in\mathbb{R}^{H\times W\times 3}$ into an image embedding $e^{V}\in\mathbb{R}^{N\times C}$, where $C$ is the channel dimension and $N$ is the number of visual tokens in image embedding. LLM-based Multi-modal Decoder $\mathcal{D}_{\text{LLM}}$ that extracts context features from the interleaved image-text token sequences. Its input sequence $E\in\mathbb{R}^{K\times C}$ is a concatenation of embeddings $(e_{1},e_{2},\dots)$, where $e_{n}$ is either a word embedding $e_{n}^{L}\in\mathbb{R}^{1\times C}$ or an image embedding $e_{n}^{V}\in\mathbb{R}^{N\times C}$. $K$ is the total number of input tokens. DM-based Image Decoder $\mathcal{D}_{\text{DM}}$ that generates the image conditioned on image-text sequences context feature.

To provide the conditional inputs for $\mathcal{D}_{\text{DM}}$ and reduce the number of visual tokens in image embedding $e^{V}$, two different Perceiver Resampler (Alayrac et al., ) are employed to map the output features from multi-modal decoder $\mathcal{D}_{\text{LLM}}$ and image encoder $\mathcal{E}_{V}$ to a fixed number of conditional tokens, respectively. Additionally, we utilize an extra mask-aware visual extractor $\mathcal{E}_{\text{M}}$ for extracting region visual information from image embedding $e^{V}$ via simple mask-aware operation $\mathcal{E}_{\text{M}}(e^{V},\mathcal{M}^{V})$, where $\mathcal{M}^{V}$ is the corresponding binary mask of image $x^{V}$.

Figure: Figure 2: Overview of our DEEM framework. Interleaved documents serve as input, decoded to produce outputs. Both text and images are encoded into sequential, discrete token embeddings for the LMM input. Here, we replace the $<$IMG$>$ token embedding in the text with the image embedding before inputting it into the LLM. The text is predicted in an autoregressive manner and the images are synthesized by the DM-based image decoder conditioned on holistic historical semantics captured by LMM. Besides, the image token embeddings are fed into DM-based image decoder for consistent image restoration. The start of image token $<$SOI$>$ is used to determine the starting position of the image, facilitating the natural autoregressive generation of interleaved text-image layouts. Note that our core architecture is presented without the connectors between modules for simplicity.
Refer to caption: x2.png

### 2.2 Pipeline

As shown in Fig. 2, given an interleaved image-text sequence $X=\{x_{1},x_{2},x_{3},\dots\}$, where each element $x_{n}$ is either a text token (denoted as $x_{n}^{L}$) or a whole image (denoted as $x_{n}^{V}$). Text and images are arranged in the order in which they appear in the original content.
To build an end-to-end generative model for interleaved image-text data, a common practice is to first extract embedding for each text token and each image and then feed them into LLMs, i.e., $e_{n}^{L}=\mathcal{E}_{L}(x_{n}^{L})$ and $e_{n}^{V}=\mathcal{E}_{\text{M}}(\mathcal{E}_{V}(x_{n}^{V}),\mathcal{M}_{n}^{V})$, where $\mathcal{E}_{L}$ denotes word embedding in LLM. $\mathcal{E}_{V}$ is typically an image encoder followed by a Perceiver Resampler (Alayrac et al., ) to map each image to a fixed number of visual tokens. As shown in Fig. 3, we introduce a mask-aware visual extractor $\mathcal{E}_{\text{M}}$ for extracting region visual information from image embedding $e_{n}^{V}$ via simple mask-aware operation $\mathcal{E}_{\text{M}}(e_{n}^{V},\mathcal{M}_{n}^{V})$, where $\mathcal{M}_{n}^{V}$ is the corresponding binary mask of image $x_{n}^{V}$ and the default value is 1. Then, the interleaved generative modeling is trained to maximize the log-likelihood:

$$ $\displaystyle\log p(X)=\sum_{n}\log p(x_{n}|e_{<n})=\sum_{n\in\mathcal{I}_{L}} \underbrace{\log p(x_{n}^{L}|e_{<n})}_{\text{text ~{} prediction}}+\sum_{n\in \mathcal{I}_{V}}\underbrace{\log p(x_{n}^{V}|e_{<n})}_{\text{image ~{} prediction}},$ (1) $$

where $\mathcal{I}_{L}$ and $\mathcal{I}_{V}$ represent the index sets for text tokens and images, respectively. That $<n$ in the subscript represents the abbreviation of $\{1,2,\dots,n-1\}$. The following paragraphs provide explanations of Eq. (1).

Text Generation with Multi-modal Condition.
$\log p(x_{n}^{L}|e_{<n})$ is similar to traditional causal language modeling, except that the condition also includes previous images.
Recent works (Alayrac et al., ; Li et al., ; Liu et al., ) have demonstrated the effectiveness of using LLMs for processing additional visual inputs.
The loss function for text generation is

$$ $\displaystyle\mathcal{L}_{\text{NTP}}(x_{n}^{L}|e_{<n})=-\log p(x_{n}^{L}| \mathcal{D}_{\text{LLM}}(e_{<n})\big{)},$ (2) $$

where $\mathcal{D}_{\text{LLM}}$ denotes the LLM network.

Image Generation with Multi-modal Condition.
Maximizing $\log p(x_{n}^{V}|e_{<n})$ aligns with the diffusion denoising process, which recently achieved widespread success in image generation. Maximizing the log-likelihood is derived as minimizing the diffusion modeling loss as

$$ $\displaystyle\mathcal{L}_{\text{NIP}}(x_{n}^{V}|e_{<n})=\mathbb{E}_{\epsilon,t }~{}||\epsilon-\mathcal{D}_{\text{DM}}\big{(}x_{n,t}^{V},t,\mathcal{D}_{\text{ LLM}}(e_{<n})\big{)}||^{2},$ (3) $$

where $\mathcal{D}_{DM}$ is the diffusion model for denoising process. That $x_{n,t}^{V}$ is the noisy version of the original image at the denoising step $t$, and the denoising network $\mathcal{D}_{DM}$ is trained to predict the noise $\epsilon$.

Consistency Semantic Regularization. In addition to the above text and image generation loss functions, we propose a new consistency semantic constraint term. This term reuses the diffusion model to perform generative checks on the image semantic information extracted by the image encoder, ultimately correcting erroneous knowledge in the pre-trained image encoder. This significantly enhances the out-of-distribution generalization and reduces visual hallucinations in the multi-modal model. The new log-likelihood function can be written as

$$ $\displaystyle\log p^{\star}(X)=\sum_{n\in\mathcal{I}_{L}}\underbrace{\log p(x_ {n}^{L}|e_{<n})}_{\text{text~{}prediction}}+\sum_{n\in\mathcal{I}_{V}} \underbrace{\log p(x_{n}^{V}|e_{<n})}_{\text{image~{}prediction}}+\sum_{n\in \mathcal{I}_{V}}\underbrace{\log p(x_{n}^{V}|e_{n})}_{\text{image~{} restoration}}.$ (4) $$

Similarly, the corresponding log-likelihood function $\log p(x_{n}^{V}|e_{n})$ can be equivalently written as the following loss function used in training:

$$ $\displaystyle\mathcal{L}_{\text{CSR}}(x_{n}^{V}|e_{n})=\mathbb{E}_{\epsilon,t} ~{}||\epsilon-\mathcal{D}_{\text{DM}}\big{(}x_{n,t}^{V},t,e_{n}\big{)}||^{2}.$ (5) $$

Note that the new end-to-end modeling framework brings significant improvements to the generalization performance of the model without altering the original modeling flexibility or introducing additional modules.

Figure: Figure 3: Pipeline of Mask-Aware Extractor. The mask-aware extractor can be used to extract region-level visual features based on the mask-aware operation. A simple dot product is applied between the mask and the image embedding before being fed into the LLM.
Refer to caption: x3.png

### 2.3 Training and Inference

We employ a three-stage training process, consisting of image-text alignment pre-training, image-text instruction fine-tuning, and mask-text instruction fine-tuning. Image-text alignment pre-training and image-text instruction fine-tuning are designed to validate the effectiveness and efficiency of semantic consistency regularization in enhancing the visual perception capabilities of LMMs. Mask-text instruction fine-tuning is used to verify whether the model trained with semantic consistency regularization negatively impacts the performance of fine-tuning on downstream tasks in the long term. The image-text alignment pre-training objective is defined as the sum of the next-text prediction loss in Eq. (2), next-image prediction loss in Eq. (3) and consistency semantic regularization loss in Eq. (5) as $\mathcal{L}_{S_{1}}=\mathcal{L}_{\text{NTP}}+\lambda~{}\mathcal{L}_{\text{NIP}
}+\lambda~{}\mathcal{L}_{\text{CSR}}$,
where $\lambda$ is a coefficient used to determine the relative loss weight between the image and text decoding branches. In order to enable the DEEM to perform general multimodal comprehension and creative tasks following human instructions, we use $\mathcal{L}_{S_{2}}=\mathcal{L}_{\text{NTP}}+\lambda~{}\mathcal{L}_{\text{CSR}}$ to conduct image-text instruction fine-tuning. To further enhance the model’s fine-grained region awareness, we conducted region-level mask-text instruction fine-tuning. Since there is no need to perform text-to-image tasks, we removed the next-image prediction loss and the training objective in mask-text instruction fine-tuning can be defined as $\mathcal{L}_{S_{3}}=\mathcal{L}_{\text{NTP}}$.
The whole framework can be optimized end-to-end during the three stages. During inference, the images and texts are generated in an auto-regressive manner. Text tokens are sampled from the distribution predicted by the multi-modal LLM. When the generated token is <SoI>, the diffusion model is called for generating the next image.

## 3 Experiment

### 3.1 Implementation Details

In this subsection, we first introduce the network of DEEM and then showcase the three-stage training recipes. More details of datasets and hyper-parameters can be found in Table 11.

Network. Similar to previous work, We leverage Vicuna7B (Zheng et al., ) and Stable Diffusion v2.1 (Rombach et al., ) as the large language model, and image decoder, respectively. However, unlike their use of a 427M parameter CLIP-ViT-L as the image encoder, we use a smaller 122M parameter CLIP-ConvNeXt-B(Liu et al., ). For the multi-modal LLM, two different Perceiver Resamplers (Alayrac et al., ) are used to connect diffusion model with image encoder and large language model respectively.

Image-Text Alignment Pre-training. Our model is pre-trained on a mixture of image-text pairs and interleaved image-text sequences, including MMC4-Core (Zhu et al., ), LAION-400M (Schuhmann et al., ), SBU (Ordonez et al., ), and CC-12M (Changpinyo et al., ). For LAION-400M (Schuhmann et al., ), SBU (Ordonez et al., ), and CC-12M (Changpinyo et al., ), instead of utilizing the original annotations, we use the version filtered by the pre-trained BLIP-2 model (Li et al., ). For simplicity, we refer to it as BLIP-LCS hereafter. ”LCS” abbreviates the LAION, CC, and SBU datasets. The sampling probability of MMC4 is twice that of BLIP-LCS. The images are inserted before or after the corresponding text sentence with equal probability. To optimize training efficiency and data utility, multiple image-text pairs or interleaved image-text sequences are concatenated into extended sequences with the maximum context length.

Image-Text Instruction Fine-tuning.
To enable DEEM to perform general multimodal comprehension tasks following human instructions, we utilize publicly available datasets for image-text instruction fine-tuning, including LLaVA-665K (Liu et al., ), COCO
Caption (Chen et al., ), VQAv2 (Goyal et al., ), TextCaps (Sidorov et al., ), OCR-VQA (Mishra et al., ), GQA (Hudson & Manning, ), OK-VQA (Marino et al., ), TextVQA (Singh et al., ), and
AOK-VQA (Schwenk et al., ).

Figure: Figure 4: Examples from ImageNet-R, ImageNet-A, and ImageNet-V2. These examples share similar backgrounds, rare materials, and unusual textures. They serve as natural adversarial examples and out-of-distribution data, which can be used to test the robustness of models.
Refer to caption: x4.png

Mask-Text Instruction Fine-tuning. At this stage, we use a simple mask-aware visual extractor to capture pixel-level region features and then align mask-based region features with language embeddings. We collect short text and pixel-level mask pairs from the publicly available object-level datasets (COCO (Chen et al., ), RefCOCO (Kazemzadeh et al., ),
RefCOCO+ (Mao et al., ), RefCOCOg (Mao et al., )), part-level datasets (Pascal Part (Chen et al., ), Part Imagenet (He et al., )), and multiple region datasets(VCR (Zellers et al., ), Visual Genome (Krishna et al., )). Then we conduct mask-text instruction fine-tuning on the mixture of the above text-mask pairs data, enabling DEEM to complete region-level understanding tasks, such as region-level image captioning.

### 3.2 Experimental Results

In this study, we evaluate our DEEM model by comparing it with current state-of-the-art (SOTA) models on various tasks including visual robustness , hallucination diagnosis, basic visual perception and image-level visual question answering. Please refer to Appendix C for more experimental results about mask-level visual question answering and text-to-image generation. All metrics and data splits are listed in Table 11 in Appendix E.

**Table 1: Zero-shot visual robustness, hallucination and perception evaluation of RobustVQA-A: RVQA-A, RobustVQA-R: RVQA-R, RobustVQA-V: RVQA-V, POPE-Random: POPE-R (Li et al., ), POPE-Popular: POPE-P (Li et al., ), POPE-Adversarial: POPE-A (Li et al., ) and MMVP (Tong et al., ) benchmarks. RobustVQA-A, RobustVQA-R, and RobustVQA-V are robustness benchmarks designed by us in Section E.1. ”AVG” denotes the overall average accuracy of seven benchmarks. ”SFT” denotes the supervised fine-tuning. ”*” denotes baseline model without diffusion feedback. The evaluation metrics for each benchmark are listed in Table 12.**
| Method | SFT | Architecture | RVQA-A | RVQA-R | RVQA-V | POPE-R | POPE-P | POPE-A | MMVP | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models for Text-Generation Only |  |  |  |  |  |  |  |  |  |  |
| Shikra (Chen et al., ) | ✓ | ViT-L/LLaMA 7B | 33.71 | 38.33 | 37.45 | 86.90 | 83.97 | 83.10 | 22.56 | 55.15 |
| NeXT-Chat (Zhang et al., ) | ✓ | ViT-L/Vicuna 7B | 44.82 | 43.67 | 47.30 | 87.70 | 84.57 | 81.93 | 27.41 | 59.62 |
| Models for Text and Image Generation |  |  |  |  |  |  |  |  |  |  |
| MM-Interleaved (Tian et al., ) | ✗ | ViT-L/Vicuna 13B | 50.76 | 52.71 | 50.60 | 64.73 | 65.33 | 65.20 | 23.82 | 53.31 |
| Emu-I (Sun et al., ) | ✓ | ViT-L/Vicuna 7B | 46.40 | 49.12 | 47.36 | 61.28 | 56.79 | 56.01 | 22.69 | 48.52 |
| SEED (Ge et al., ) | ✓ | ViT-G/Vicuna 7B | 52.06 | 59.71 | 57.02 | 69.84 | 56.83 | 59.63 | 25.62 | 54.39 |
| DreamLLM (Dong et al., ) | ✓ | ViT-L/Vicuna 7B | 51.43 | 58.96 | 57.60 | 86.36 | 80.07 | 72.63 | 26.37 | 61.84 |
| SEED-X (Ge et al., ) | ✓ | ViT-G/Vicuna 13B | 52.36 | 60.27 | 59.49 | 86.41 | 81.43 | 74.56 | 29.16 | 63.39 |
| DEEM * | ✗ | ConvNext-B/Vicuna 7B | 53.24 | 56.06 | 54.72 | 50.55 | 52.00 | 51.93 | 20.30 | 48.40 |
| DEEM | ✗ | ConvNext-B/Vicuna 7B | 56.86 | 68.63 | 63.08 | 69.93 | 70.27 | 68.87 | 28.74 | 60.91 |
| DEEM -VQA | ✓ | ConvNext-B/Vicuna 7B | 55.22 | 64.12 | 62.99 | 87.40 | 82.80 | 78.49 | 32.89 | 65.56 |

Visual Perception Diagnose. We explore the impact of diffusion feedback on the visual perception capabilities of LMMs from three dimensions: visual robustness, visual hallucinations, and basic visual perception. To rigorously assess visual robustness of our model, we design a benchmark called RobustVQA for robustness evaluation based on online datasets, including ImageNet-A (Hendrycks et al., ), ImageNet-R (Hendrycks et al., ) and ImageNet-V2  (Recht et al., ). As shown in Fig. 4, these challenging natural adversarial examples and out-of-distribution samples in the original ImageNet dataset can be used to evaluate the neural network robustness of our model. Similar to the POPE and MMVP dataset, we first choose the challenging sample from ImageNet-A, ImageNet-R, and ImageNet-V2 dataset and then convert the them into a VQA format that the multimodal model can evaluate simply and accurately. More details about the new benchmark RobustVQA design can be found in Section E.1.
For a comprehensive visual robustness and hallucination evaluation, we evaluate our model against other open-source state-of-the-art (SOTA) LMMs for text and image generation, including SEED (Ge et al., ), SEED-X (Ge et al., ), MM-Interleaved (Tian et al., ), and DreamLLM (Dong et al., ), on the RobustVQA, POPE  (Li et al., ) and MMVP  (Tong et al., ) dataset with accuracy metric. The results, presented in Table 1, demonstrate that our DEEM model not only exhibits competitive performance compared with existing fine-tuned SOTA models on POPE and MMVP after fine-tuning, but also achieves the best results among visual robustness benchmark only after pre-training. Notably, compared to the larger-scale concurrent SOTA model for interleaved text-image modeling, MM-Interleaved (Tian et al., ), our model achieves better results with a smaller scale. DEEM outperforms MM-Interleaved 9.4% on RobustVQA,
17.8% on POPE and 9.1% on MMVP. To ensure a fair comparison and prove the effectiveness of our method, we also train an MM-Interleaved model with the same experimental setting as a baseline. Compared to this baseline, Our method achieves an 4% average gain on RobustVQA,
12.8% average gain on POPE and 6.5% average gain on MMVP, respectively. The experimental results demonstrate the effectiveness of our method for better LMMs’ visual perception capability.

**Table 2: Multi-modal comprehension evaluation. “ED” denotes using extra in-house data. Benchmarks include COCO (Chen et al., ); I2Para.: Image2Paragraph (Krause et al., ); VQA^v2: VQAv2 (Goyal et al., ); OKVQA (Marino et al., ); GQA (Hudson & Manning, ); VizWiz (Gurari et al., ); VisDial (Das et al., ); MMBench: MMB (Yu et al., ); MMVet (Yu et al., );. The evaluation metrics for each benchmark are listed in Table 12.**
| Model | LLM | VFM | ED | COCO | I2Para. | VQA^v2 | OKVQA | GQA | VizWiz | VisDial | MMB | MMVet |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Models for Text-Generation Only |  |  |  |  |  |  |  |  |  |  |  |  |
| IDEFICS-80B (IDEFICS, ) | LLaMA-65B | ViT-H | ✗ | 91.8 | – | 60.0 | – | 45.2 | 36.0 | – | 27.9 | – |
| IDEFICS-80B-I (IDEFICS, ) | LLaMA-65B | ViT-H | ✗ | 117.2 | – | 37.4 | – | – | 26.0 | – | – | – |
| KOSMOS-1 (Huang et al., ) | MetaLM | ViT-L | ✓ | – | – | 46.7 | – | – | – | – | – | – |
| KOSMOS-2 (Peng et al., ) | KOSMOS-1 | ViT-L | ✓ | – | – | 45.6 | – | – | – | – | – | – |
| Flamingo-9B (Alayrac et al., ) | Chinchilla-7B | ViT-L | ✓ | 79.4 | – | 51.8 | 44.7 | – | 28.8 | 48.0 | 7.9 | 23.3 |
| Flamingo-80B (Alayrac et al., ) | Chinchilla-70B | ViT-H | ✓ | 84.3 | – | 56.3 | 50.6 | – | 31.6 | 52.0 | – | – |
| mPLUG-DocOwl (Ye et al., ) | LLaMA-7B | ViT-L | ✗ | 52.6 | – | – | – | – | – | – | 60.8 | 35.7 |
| BLIP-2 (Li et al., ) | Vicuna-7B | ViT-L | ✗ | – | – | – | – | 38.6 | 25.3 | – | – | – |
| BLIP-2 (Li et al., ) | Vicuna-13B | ViT-L | ✗ | – | – | 41.0 | – | 41.0 | 19.6 | – | – | – |
| InstructBLIP (Dai et al., ) | Vicuna-7B | ViT-L | ✗ | – | – | – | – | 49.2 | 34.5 | – | 68.9 | 33.1 |
| InstructBLIP (Dai et al., ) | Vicuna-13B | ViT-L | ✗ | – | – | – | – | 49.5 | 33.4 | – | – | – |
| Shikra (Chen et al., ) | Vicuna-13B | ViT-L | ✗ | 117.5 | – | 77.4 | – | – | – | – | – | – |
| LLaVA-1.5 (Liu et al., ) | Vicuna-7B | ViT-L | ✗ | – | – | 78.5 | – | 62.0 | 50.0 | – | 53.1 | 32.9 |
| LLaVA-1.5 (Liu et al., ) | Vicuna-13B | ViT-L | ✗ | – | – | 80.0 | – | 63.3 | 53.6 | – | 60.6 | 35.6 |
| Qwen-VL (Bai et al., ) | Qwen-7B | ViT-G | ✗ | – | – | 78.8 | – | 59.3 | 35.2 | – | 32.9 | 13.0 |
| Qwen-VL-Chat (Bai et al., ) | Qwen-7B | ViT-G | ✓ | – | – | 78.2 | – | 57.5 | 38.9 | – | 59.1 | – |
| Models for both Image and Text Generation |  |  |  |  |  |  |  |  |  |  |  |  |
| CM3Leon (Yu et al., ) | – | – | ✓ | 61.6 | 10.5 | 47.6 | 23.8 | – | 37.6 | 22.6 | – | – |
| Emu (Sun et al., ) | Vicuna-13B | ViT-L | ✓ | 112.4 | – | 52.0 | 38.2 | – | 34.2 | 47.4 | – | – |
| Emu-I (Sun et al., ) | Vicuna-13B | ViT-L | ✓ | 117.7 | – | 40.0 | 34.7 | – | 35.4 | 48.0 | – | – |
| Emu2 (Sun et al., ) | LLaMA-33B | ViT-L | ✓ | – | – | 33.3 | 26.7 | – | 40.4 | – | – | – |
| DreamLLM (Dong et al., ) | Vicuna-7B | ViT-L | ✗ | 103.7 | 8.4 | 72.9 | 52.2 | – | 49.3 | – | 58.2 | 36.6 |
| DEEM -VQA | Vicuna-7B | ConvNext-B | ✗ | 115.4 | 22.4 | 68.2 | 53.4 | 55.7 | 50.4 | 42.1 | 60.8 | 37.4 |

Image-Level Visual Question Answering and Captioning. In order to assess multimodal vision and language capabilities of DEEM , we conduct evaluation against current SOTA LMMs including LLaVA-1.5 (Liu et al., ), Qwen-VL (Bai et al., ), DreamLLM (Dong et al., ) and MM-Interleaved (Tian et al., ) across several tasks, including image captioning on COCO  (Chen et al., ), Image2Paragraph (Krause et al., ), visual question answering on VQAv2 (Goyal et al., ), OKVQA (Marino et al., ), GQA (Hudson & Manning, ), VizWiz (Gurari et al., ), and VisDial
(Das et al., ). As demonstrated in Table 2, DEEM exhibits
superior or comparable performance relative to SOTA models. In comparison with models for text generation only, our approach consistently achieves competitive performance across various dataset splits. Against models for both image and text generation, DEEM demonstrates enhanced performance in nine dataset splits. Compared to the current state-of-the-art model DreamLLM, DEEM outperforms DreamLLM in six out of the seven shared evaluation dataset splits. It is noteworthy that DEEM is trained with a significantly smaller image encoder CLIP-ConvNeXt-B (Liu et al., ), comprising only 122M parameters, in stark contrast to baselines such as DreamLLM (Dong et al., ), which utilize larger 427M CLIP-ViT-L (Radford et al., ). These results indicate that our method can enhance the model’s robustness performance without compromising the multimodal vision and language capabilities of our model.

### 3.3 Ablation Study

In this study, we conduct ablation studies on several key components of the model, including consistency semantic regularization, training latency, scalability and the impact of different architectures. Benchmarks include RobustVQA-A:RVQA-A; RobustVQA-R: RVQA-R; RobustVQA-V:RVQA-V; POPE-R (Li et al., ); POPE-P (Li et al., ); POPE-A (Li et al., ); MMVP (Tong et al., ); OK-VQA (Marino et al., ). More additional ablation studies can be found in Appendix D.

**Table 3: Ablation study of $\mathcal{L}_{\text{CSR}}$ and training latency. Using semantic consistency regularization during both the pre-training and supervised fine-tuning phases can significantly enhance the model’s robustness and resistance to hallucinations, while incurring only a marginal additional training cost.**
| SFT | $\mathcal{L}_{\text{CSR}}$ | RVQA-A | RVQA-R | RVQA-V | POPE-R | POPE-P | POPE-A | SPEED |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✗ | ✗ | 53.2 | 56.1 | 54.7 | 50.6 | 52.0 | 51.9 | 8.11 s/step |
| ✗ | ✓ | 57.8 | 69.0 | 64.8 | 69.9 | 70.3 | 68.9 | 9.25 s/step |
| ✓ | ✗ | 51.3 | 56.5 | 57.4 | 85.4 | 78.8 | 76.2 | 2.14 s/step |
| ✓ | ✓ | 53.5 | 57.6 | 58.1 | 86.0 | 79.2 | 77.1 | 2.22 s/step |

Consistency Semantic Regularization and Training Latency. To evaluate the effectiveness of the key elements of our design, we conduct the following ablation experiments. We first pre-train a baseline model without using the consistency semantic regularization term under the same training setting for comparison to demonstrate the effectiveness of our architecture. As we can see from Table 3, during the pre-training phase, using our consistency semantic regularization can significantly enhance the model’s performance on both hallucination and robustness benchmarks. Moreover, we load the weights of the pre-trained model for image-text instruction fine-tuning experiments. In the second phase of image-text instruction fine-tuning experiments, we demonstrate the effectiveness of our model design. As detailed in Table 3, we observe that after fine-tuning with image-text instruction data, the model’s visual hallucination ability improves further, but its visual perception robustness decreases. However, using our consistency semantic regularization can mitigate the robustness degradation while further enhancing the model’s visual hallucination ability. To explore the impact of introducing consistency semantic regularization on the training latency in the two stages of training, we conduct corresponding ablation experiments. We present the result in  Table 3. Employing consistency semantic regularization adds only a marginal increase in training latency, yet it significantly enhances the model’s robustness capabilities.

**Table 4: Ablation study of model scalability. Gradually expanding the training data and model size can further enhance the model’s capabilities, demonstrating the scalability of the approach.**
| Architecture | Training Data | RVQA-A | RVQA-R | RVQA-V | POPE-R | POPE-P | POPE-A | OK-VQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ConvNext-B/Vicuna 7B | 32K | 51.86 | 54.31 | 52.73 | 48.44 | 50.10 | 50.06 | 20.74 |
| ConvNext-B/Vicuna 7B | 96K | 52.31 | 57.43 | 54.06 | 54.42 | 57.22 | 56.35 | 22.33 |
| ConvNext-B/Vicuna 7B | 160K | 52.89 | 58.93 | 55.31 | 60.28 | 60.74 | 59.96 | 23.65 |
| ConvNext-L/Vicuna 7B | 160K | 53.23 | 60.47 | 56.88 | 61.12 | 62.87 | 62.09 | 23.87 |
| ConvNext-B/Vicuna 13B | 160K | 53.92 | 61.27 | 57.02 | 62.60 | 64.26 | 63.19 | 31.11 |

Model Scalability. Although DEEM demonstrates better performance with smaller data count and model sizes, its scalability has yet to be validated. As is well known, scalability is crucial for model performance. We conduct ablation experiments to assess the scalability concerning data count and model size. As shown in Table 4, gradually increasing the training data enables the model to successfully scale while achieving improved results. Additionally, increasing the sizes of both the VFM and LLM leads to sustained performance enhancements, indicating that DEEM possesses good scalability.

**Table 5: Ablation study of different architectures. Our method not only significantly enhances the capabilities of LLMs for text and image generation with marginal additional training costs, but it also improves the performance of LLMs for text generation only, validating the generalization ability of the approach.**
| Name | $\mathcal{L}_{\text{CSR}}$ | MMVP | RVQA-A | RVQA-R | RVQA-V | POPE-R | POPE-P | POPE-A |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA | ✗ | 18.6 | 54.8 | 60.0 | 58.7 | 55.5 | 53.3 | 54.6 |
| LLaVA | ✓ | 25.1 | 56.7 | 66.7 | 61.9 | 67.9 | 68.7 | 65.4 |
| DEEM | ✗ | 20.3 | 53.2 | 56.1 | 54.7 | 50.6 | 52.0 | 51.9 |
| DEEM | ✓ | 28.7 | 56.9 | 68.6 | 63.1 | 69.9 | 70.3 | 68.9 |

Impact of Different Architectures. By cleverly reusing the diffusion model from LMMs for image and text generation, we can significantly enhance the model’s foundational visual perception, visual robustness, and anti-hallucination capabilities with only marginal additional training costs. However, whether DEEM possesses sufficient generalization ability to remain effective for LMMs on text generation obnly has yet to be explored. To validate our hypothesis, we employ the LLaVA (Liu et al., ) architecture and conducted ablation experiments using semantic consistency regularization loss, with results presented in Table 5. We observe that utilizing diffusion feedback to improve the basic perceptual capabilities of LMMs—thus preventing the model from overly compressing visual information and losing sensitivity to subtle details—is a general method that is architecture-agnostic and exhibits good generalization properties. This suggests that the benefits of our approach could extend beyond the specific configurations tested, potentially enhancing a wide range of LMMs in various applications.

## 4 Related Work

### 4.1 Diffusion Models for Representation Learning

Diffusion models have made significant progress in various generative tasks (Song et al., ; Ho et al., ), such as image generation (Betker et al., ), video generation (Ho et al., ), and object tracking (Luo et al., ). In addition to the aforementioned research, many studies focus on leveraging diffusion models for representation learning. Some works utilize the conditional control of pre-trained diffusion models to flexibly address different downstream tasks, including object classification (Xiang et al., ), semantic segmentation (Xu et al., ), and keypoint matching (Nam et al., ). Other studies  (Li et al., ; Song et al., ) design specialized modules and train diffusion models from scratch to further enhance representation capabilities. Although diffusion models have been widely applied in the generative tasks of large multimodal models, the use of diffusion models to optimize the visual representations of large multimodal models has yet to be explored. To our knowledge, we are the first to employ diffusion models in a self-supervised paradigm to optimize the visual representations of large multimodal models, significantly enhancing their perceptual abilities and reliability at minimal cost.

### 4.2 Large Multimodal Model

Image-to-text large multimodal models (LMMs) inject visual information into large language models (LLMs) through vision foundation models (VFMs), allowing the language models to perceive visual inputs and thus generate captions or answer questions based on the given multimodal content. Flamingo (Alayrac et al., ) tries to extract vision features with a resampler, and transfer them into the text features with a cross-attention mechanism. Instead of using cross-attention layers, BLIP-2 (Li et al., ) directly feed the visual features into the LLMs as soft prompts and significantly reduce the training cost by reducing the visual token number. LLaVA (Liu et al., ) and MiniGPT-4 (Zhu et al., ) construct a small-scale instruction tuning dataset to better align the LMM with the expected output format.
Although this unidirectional image-to-text paradigm has achieved tremendous success, it still fails to unify multimodal tasks like text-to-image generation and image-to-text visual question answering, significantly limiting the capabilities of multimodal models.

In order to unify multimodal tasks into a unified manner, some works (Yu et al., ; Koh et al., ; Sun et al., ; Dong et al., ; Tian et al., ; Ge et al., ; ; Luo et al., ) attempt to generate images and text in the interleaved context concurrently. The release of some public large-scale interleaved image-text datasets (Laurençon et al., ; Zhu et al., ) has significantly advanced the development of this field. CM3Leon (Yu et al., ) converts images into discrete tokens, facilitating token-level auto-regressive modeling as traditional language modeling. Although CM3Leon showcases competitive image generation capabilities, it exhibits notable weaknesses in image understanding. Emu (Sun et al., ) and DreamLLM (Dong et al., ) focus on single-stage end-to-end modeling using raw image pixels as input for interleaved image-text generation modeling, but they feed image information at the input of LMMs, which are limited by the problem that fixed number of visual tokens cannot efficiently describe image details. MM-Interleaved (Tian et al., ) addresses this limitation by integrating image details into LMMs via multi-scale visual features. However, when faced with out-of-distribution noisy data, the image encoders used by LMMs often produce incorrect visual information, ultimately leading to erroneous predictions. This significantly limits the application of the models in safety-critical scenarios. Building on an advanced interleaved content modeling mechanism, we propose DEEM , which cleverly reuses DMs to correct the outputs of the VFMs without increasing extra parameter count, thereby enhancing the model’s generalization capabilities and reducing visual hallucinations in a self-supervised manner. Similar to previous work (Liu et al., ; Dong et al., ; Tian et al., ), after supervised fine-tuning, it achieves competitive performance on multiple downstream multimodal tasks with the smallest scale.

## 5 Conclusion

Can diffusion models serve as the eyes of
large language models for image perception? In this paper, we answer the question by proposing a novel method called DEEM , which leverages a diffusion model as the eyes for LLMs. This approach enhances the robustness of the multimodal model for interleaved image-text modeling and reduces visual hallucinations without introducing extra modules. Through comprehensive exploratory experiments, we demonstrate the effectiveness of
the proposed DEEM method. In addition to its advanced robust performance and visual hallucination handling capabilities, we adopt an additional two-stage instruction fine-tuning process to broaden the application scenarios of our DEEM . This enables DEEM to handle a variety of multimodal tasks, including visual question answering, image captioning, and region-level image reasoning. Besides, this work initiates the first step towards visual robustness via generative feedback in a multimodal model. In the future, we will continue to enhance the model’s ability to conduct better multimodal comprehension and creation tasks. As an end-to-end framework, we hope it will spur further research in the multimodal robustness field, such as multimodal agents that can handle complex tasks that require safety abilities.

## Appendix A Limitation

Although our method significantly enhances the visual robustness of interleaved image-text modeling multimodal models after image-text alignment pre-training, it, unfortunately, cannot eliminate but only alleviate the robustness knowledge forgetting issue caused by subsequent fine-tuning, as shown in the Table 3. Additionally, our model requires using a diffusion model as another eye to correct and update the erroneous knowledge of the image encoder to improve the overall visual robustness of the multimodal model. However, updating larger image encoders such as CLIP-ViT-L and CLIP-ViT-G(Radford et al., ) will increase the training overhead, which may limit the application of our model. We hope that in the future, the diffusion model can completely replace the image encoder to further enhance the effectiveness of our method.

## Appendix B Broader Impacts

The proposed method introduces a novel strategy to enhance the robustness and generalization capabilities of multimodal models by leveraging a diffusion model as an additional eye for large language models. This strategy allows for the correction and updating of potential semantic errors in the image encoder, leading to significant improvements in handling out-of-distribution data and mitigating visual hallucinations. Overall, our contributions provide a significant step forward in the field of multimodal, offering a robust, efficient, and scalable solution for improving the accuracy and reliability of multimodal models. The broader impacts of this work include the potential for more intelligent and adaptive AI systems that can operate effectively in diverse and challenging environments.

**Table 6: Zero-shot region-level image captioning results on ReferCOCOg.**
| Method | Type | METEOR | CIDEr |
| --- | --- | --- | --- |
| GRIT (Wu et al., ) | Box | 15.2 | 71.6 |
| Kosmos-2 (0-shot) (Peng et al., ) | Box | 12.2 | 60.3 |
| Kosmos-2 (2-shot) (Peng et al., ) | Box | 13.8 | 62.2 |
| Kosmos-2 (4-shot) (Peng et al., ) | Box | 14.1 | 62.3 |
| NeXt-Chat (Zhang et al., ) | Box | 12.0 | 79.0 |
| DEEM -Mask | Mask | 14.1 | 71.0 |

## Appendix C Additional Experiments Results

Figure: Table 7: Zero-shot text-to-image generation FID on MS-COCO and LN-COCO.

Region-Level Image Captioning. In addition to holistic image understanding, we also validate the model’s ability to take region-level image captioning. As shown in Fig. 3, we use a mask-aware extractor to obtain region-level visual features and address region-level image captioning tasks. We adopt the RefCOCOg (Mao et al., ) validation set and compare it with other state-of-the-art (SOTA) models, including GRIT (Wu et al., ), Kosmos-2 (Peng et al., ), and NeXt-Chat (Zhang et al., ). The CIDEr (Vedantam et al., ) and METEOR are applied as the evaluation metrics. As shown in Table 6, our model is capable of achieving competitive performance on CIDEr and METEOR across all of the compared methods, which shows the superiority of our DEEM .

Text-to-Image Generation. we evaluate text-conditional image generation on MS-COCO (Lin et al., ) and LN-COCO (Pont-Tuset et al., ). On MSCOCO, we sample 8 images per text condition and use CLIP-ViT-L (Radford et al., ) to rerank based on text-image similarity. CLIP reranking is not used for LN-COCO. FID  (Heusel et al., ) is used to evaluate both datasets. As shown in Table 7, our model shows
competitive text-to-image generation compared to existing
image and text generation models. See
qualitative results on text-to-image synthesis in Fig. 9 in Appendix F.

## Appendix D Additional Ablation Study

we provide more ablation studies for DEEM in this section, all of which share the same settings. All the code, models, and data tools will be released soon.

### D.1 Ablation Study of Input Image Resolution

**Table 8: Ablation study of input image resolution and coefficient $\lambda$ with 2k training steps and 16 batch size.**
| SFT | resolution | $\lambda$ | RVQA-A | RVQA-R | RVQA-V | POPE-R | POPE-P | POPE-A | OK-VQA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ✗ | 256 | 1 | 51.6 | 52.0 | 49.6 | 48.5 | 50.0 | 50.0 | 18.9 |
| ✗ | 256 | 5 | 51.9 | 54.3 | 52.7 | 48.4 | 50.1 | 50.0 | 20.7 |
| ✗ | 256 | 10 | 51.7 | 52.7 | 51.9 | 48.7 | 50.3 | 50.3 | 20.1 |
| ✓ | 256 | 5 | 51.5 | 59.1 | 57.9 | 85.9 | 77.1 | 76.4 | 38.7 |
| ✓ | 448 | 5 | 52.5 | 57.6 | 58.1 | 86.0 | 79.2 | 77.1 | 41.0 |

In addition to the aforementioned exploration, we also scale up the input image resolution for performance gain. The performance gain becomes larger when further increasing the input image resolution from 256 to 448 in image-text instruction fine-tuning, as shown in Table 8. Such results indicate our method could better exploit the additional information gained from high resolution. Moreover, we conduct an ablation study on coefficient $\lambda$ in loss function. As shown in Table 8, setting $\lambda$ = 5
achieves a better balance between robustness and hallucination empirically.

### D.2 Ablation Study of Training Recipes

We also conduct an ablation study to control the trainability of different training modules. As shown in Table 10, we found that freezing the DM (Diffusion Model) while not freezing the VFM (Visual Foundation Model) during training yields the best robustness and hallucination results.

## Appendix E Additional Implementation Details

**Table 9: Comparison of different VQA formats. Questions in the yes or no format can well evaluate the performance of the models on the RobustVQA benchmark, while questions in the multiple-choice format are very random, and MM-interleaved tend to output the first option. Therefore, we adopt yes or no format in our experimental settings. More details about the new benchmark RobustVQA design can be found in Section E.1.**
| Format | Prompt | RobustVQA-A | RobustVQA-R | RobustVQA-V |
| --- | --- | --- | --- | --- |
| multiple-choice | “What is the main object in this image?” | 44.88 | 58.88 | 46.86 |
| “Chose from the list: [false category label,gt category label].” |  |  |  |  |
| multiple-choice | “What is the main object in this image?” | 84.60 | 90.16 | 82.92 |
| “Chose from the list: [gt category label,false category label].” |  |  |  |  |
| yes or no | “Is [gt category label] the main object in this image?” | 50.76 | 52.71 | 50.60 |
| “Please answer yes or no.” |  |  |  |  |
| “Is [false category label] the main object in this image?” |  |  |  |  |
| “Please answer yes or no.” |  |  |  |  |

### E.1 Dataset Construction

As shown in Fig. 5, we first convert the original ImageNet-A (Hendrycks et al., ), ImageNet-R (Hendrycks et al., ), and ImageNet-V2 (Recht et al., ) data into a VQA format that the multimodal model can evaluate. Specifically, we use the CLIP-ViT-L model for hard example mining, predicting the incorrect category label with the highest confidence score apart from the ground truth category label. We then use a pre-defined prompt as: “Is [category label] the main object in this image? Please answer yes or no.” to simultaneously construct a pair of positive and negative example samples, allowing the model to answer “yes” or “no”. By using this design, we can evaluate the robustness of multimodal models in an unbiased manner with the new benchmark called RobustVQA, facilitating both assessment and comparison. It is worth noting that, as shown in Table 9, we find that the yes or no format is more stable than the multiple-choice format and can better evaluate the robustness of multi-modal models.

Figure: Figure 5: Robustness dataset construction process. We use the CLIP-ViT-L model for hard example mining and then transform them into question-answer pairs via a pre-defined template.
Refer to caption: x5.png

### E.2 Image-Text Alignment Pre-training

Figure: Table 10: Ablation study of training recipe in image-text alignment pre-training with 10k training steps and 128 batch size.

We use MMC4-Core (Zhu et al., ), LAION-400M (Schuhmann et al., ), SBU (Ordonez et al., ), and CC-12M (Changpinyo et al., ) as
the pre-training dataset. For LAION-400M (Schuhmann et al., ), SBU (Ordonez et al., ), and CC-12M (Changpinyo et al., ), instead of utilizing the original annotations, we use the version filtered by the pre-trained BLIP-2 model (Li et al., ). For simplicity, we refer to it as BLIP-LCS hereafter. ”LCS” abbreviates the LAION, CC, and SBU datasets. Text prompts with lengths shorter
than 10 are also filtered out. Due to network constraints, we only collect approximately 6M of MMC4-Core and 20M of BLIP-LCS data. The sampling probability of MMC4 is twice that of BLIP-LCS. The images are inserted before or after the corresponding text sentence with equal probability. Specifically, images with a CLIP similarity score below 0.24 will be discarded, and only 6 images at most will be kept for each document in MMC4-Core. We also exclude 100% of all documents that do not contain any images, and 50% of documents that contain only 1 image. For image-text-pair BLIP-LCS
datasets, we randomly sample multiple image-text pairs from the same
dataset and concatenate them to the maximum context
length (i.e., 2048) during pre-training. For interleaved image and text MMC4-Core (Zhu et al., ) datasets, we also split and concatenate the documents to form the training samples. Such a concatenation strategy can utilize the full context window of Large Language Models and thus achieve high data efficiency. Besides that, for image
generation, we ignore the training loss of images which are
the first element in the sequence. The text condition of the
rest images is dropped with a 10% probability to improve
classifier-free guidance sampling. The detailed hyper-parameters of image-text alignment pre-training are listed in Table 11.

### E.3 Image-Text Instruction Fine-tuning

We utilize public available datasets for supervised fine-tuning, including LLaVA-665K(Liu et al., ), COCO Caption (Chen et al., ), VQAv2 (Goyal et al., ),TextCaps (Sidorov et al., ), OCR-VQA (Mishra et al., ), GQA (Hudson & Manning, ), OK-VQA (Marino et al., ), TextVQA (Singh et al., ), and AOK-VQA (Schwenk et al., ). We use the following prompt template ‘‘Based on the image, please answer the
question. {image} {question}. The answer is: {answer}
" to convert the data into a mixture of instruction following forms, resulting in approximately 800K instruction data for the second-stage image-text instruction fine-tuning. The detailed hyper-parameters of
image-text instruction fine-tuning are listed in Table 11.

### E.4 Mask-Text Instruction Fine-tuning

We collect short text and pixel-level mask pairs from the publicly available object-level datasets (COCO, RefCOCO,
RefCOCO+) and part-level datasets (Pascal Part, Part Imagenet), then transform them into instruction following data. Moreover, Visual Genome (VG) and Visual Commonsense
Reasoning (VCR) datasets are employed to add more multiple region understanding data, resulting in approximately 200K instruction data for the third-stage mask-text instruction fine-tuning. See more hyper-parameters details in Table 11.

### E.5 Evaluation

As shown in Fig. 6, DEEM achieves the
best results on both hallucination and robustness benchmarks even at the smallest scale, demonstrating
the efficiency and effectiveness of our approach. In addition to visual robustness and hallucination, we also use various benchmarks and datasets, such as
image caption, visual question answering, text-to-image
generation and so on, to assess the image-text comprehension capabilities. All these evaluation tasks and metrics are listed in Table 12. The prompt templates for each task are listed in Fig. 8.

Figure: Figure 6: Performance on visual robustness and hallucination benchmark. DEEM achieves the best results on robustness benchmark and competitive performance on hallucination even at the smallest scale, demonstrating the efficiency and effectiveness of our approach.
Refer to caption: x6.png

Figure: Figure 7: Case Comparison. Compared to other SOTA models, including LLaVA, NeXt-Chat, and MM-Interleaved, when encountering out-of-distribution data, their models are affected by incorrect semantics from the image encoder and cannot output the correct answer. However, DEEM can output the correct answer via generative feedback.
Refer to caption: x7.png

**Table 11: Training recipes for DEEM . The three training stages are introduced in Section 2.3. Stage I: Image-Text Alignment Pre-training, Stage II: Image-Text Instruction Fine-tuning, Stage III: Mask-Text Instruction Fine-tuning.**
|  | Stage I | Stage II | Stage III |
| --- | --- | --- | --- |
| Phase | Image-Text Alignment | Image-Text SFT | Mask-Text SFT |
| Training Hyper-Parameters |  |  |  |
| Input image resolution | 256$\times$256 | 448$\times$448 | 448$\times$448 |
| Output image resolution | 512$\times$512 | 512$\times$512 | 512$\times$512 |
| VFM | CLIP-ConvNext-B | CLIP-ConvNext-B | CLIP-ConvNext-B |
| LLM | Vicuna-7B v1.5 | Vicuna-7B v1.5 | Vicuna-7B v1.5 |
| DM | Stable Diffusion v2.1 | Stable Diffusion v2.1 | Stable Diffusion v2.1 |
| $\lambda$ | 5 | 5 | 5 |
| Learning Rate | 2e-5 (image encoder&decoder) | 1e-6 (language model) | 1e-6 (language model) |
| 1e-4 (others) | 1e-5 (others) | 1e-5 (others) |  |
| Optimizer | AdamW | AdamW | AdamW |
| Optimizer hyper-parameters | $\beta_{1},\beta_{2},\epsilon=0.9,0.995,$1e-6 | $\beta_{1},\beta_{2},\epsilon=0.9,0.999,$1e-8 | $\beta_{1},\beta_{2},\epsilon=0.9,0.999,$1e-8 |
| Weight Decay | 0.05 | 0.05 | 0.05 |
| Training iterations | 10k | 10k | 10k |
| Warmup steps | 1k | 500 | 500 |
| Learning Rate Scheduler | Cosine | Cosine | Cosine |
| Batch Size Per GPU | 4 | 16 | 2 |
| Maximum Token Length | 2048 | 2048 | 2048 |
| Augmentation | CenterCrop | - | - |
| Unfreeze LLM | ✗ | ✓ | ✓ |
| Unfreeze DM | ✗ | ✗ | ✗ |
| Unfreeze VFM | ✓ | ✗ | ✗ |
| Training Data |  |  |  |
| Dataset | ➀ MMC4 | ➀ LLaVA-Mix-665K | ➀ COCO/ReferCOCO/ReferCOCO+ |
| ➁ BLIP-LCS | ➁ VQA-Mixture | ➁ Pascal-Part/Part-ImageNet |  |
|  | ➂ COCO Caption | ➃ VG/VRC |  |
| Data Size | $\sim$26M | $\sim$800K | $\sim$200K |
| Data Type | Interleave/Pair | Instruction | Instruction |
| Training Cost |  |  |  |
| GPU Device | 32$\times$NVIDIA A100 | 32$\times$NVIDIA A100 | 32$\times$NVIDIA A100 |
| Training Time | $\sim$30h | $\sim$6h | $\sim$3h |

**Table 12: Overall descriptions of the evaluation benchmarks for evaluating capabilities, including image-level captioning, image-level visual question answering, text-to-image generation, region-level image captioning, visual robustness, comprehension, perception and hallucination.**
|  | Dataset | Task description | Eval Split | Metric |
| --- | --- | --- | --- | --- |
| CAP. | COCO (Chen et al., ) | Scene description | test | CIDEr($\uparrow$) (Vedantam et al., ) |
| Image2Paragraph (Krause et al., ) | Scene description | test | CIDEr($\uparrow$) (Vedantam et al., ) |  |
| VQA. | VQAv2 (Goyal et al., ) | Scene understanding QA | test-dev | VQA Acc($\uparrow$) (Antol et al., ) |
| OKVQA (Marino et al., ) | External knowledge QA | val | VQA Acc($\uparrow$) (Antol et al., ) |  |
| GQA (Hudson & Manning, ) | Scene understanding QA | test-dev | VQA Acc($\uparrow$) (Antol et al., ) |  |
| VizWiz (Gurari et al., ) | Scene understanding QA | test-dev | VQA Acc($\uparrow$) (Antol et al., ) |  |
| VisDial (Das et al., ) | Image dialogue | val | NDCG($\uparrow$) |  |
| SYN. | MS-COCO  (Lin et al., ) | Text-Conditional Image Synthesis | val-30K | FID($\downarrow$) (Heusel et al., ) |
| LN-COCO (Pont-Tuset et al., ) | Text-Conditional Image Synthesis | val | FID($\downarrow$) (Heusel et al., ) |  |
| REF. | RefCOCO (Kazemzadeh et al., ) | Region-level scene description | val | CIDEr($\uparrow$) (Vedantam et al., ) |
| RefCOCO+ (Mao et al., ) | Region-level scene description | val | CIDEr($\uparrow$) (Vedantam et al., ) |  |
| RefCOCOg (Mao et al., ) | Region-level scene description | val | CIDEr($\uparrow$) (Vedantam et al., ) |  |
| OOD. | RobustVQA-V | Out-of-Distribution Robustness | val | Acc($\uparrow$) |
| RobustVQA-R | Out-of-Distribution Robustness | val | Acc($\uparrow$) |  |
| RobustVQA-A | Out-of-Distribution Robustness | val | Acc($\uparrow$) |  |
| Hall. | POPE-R (Li et al., ) | Visual Hallucination | val | Acc($\uparrow$) |
| POPE-P (Li et al., ) | Visual Hallucination | val | Acc($\uparrow$) |  |
| POPE-A (Li et al., ) | Visual Hallucination | val | Acc($\uparrow$) |  |
| CPH. | MMBench (Yu et al., ) | Visual Comprehension | val | Acc($\uparrow$) |
| MMVet (Yu et al., ) | Visual Comprehension | val | Acc($\uparrow$) |  |
| PCP. | MMVP (Tong et al., ) | Visual Perception | val | Acc($\uparrow$) |

Figure: Figure 8: Prompt template used for evaluation. (a) VQA includes VQAv2, VizWiz, OKVQA, GQA, VisDial, and MMVP. (b) Image Captioning includes COCO, Image2Paragraph. (c) Region-level Image Captioning includes RefCOCOg. (d) Visual hallucination includes POPE. (e) Visual Robustness includes RobustVQA-A, RobustVQA-R, and RobustVQA-V. $<\text{IMAGE}>$denotes the input image representation, $<\text{MASK}>$ denotes the mask-level image representation, $<\text{QUESTION}>$denotes each specific question, $<\text{ANSWER}>$ is the generated answer, and $<\text{OBJECT}>$ is the specific object name in a question of POPE and RobustVQA.
Refer to caption: x8.png

## Appendix F Additional Visualization Examples

### F.1 Consistency Semantic Image Synthesis

We visualize some consistency semantic image synthesis and display both the original images and their reconstructed versions in Fig. 10. DEEM accurately recovers the features of the original images without causing distortion.

Figure: Figure 9: image-to-image generation examples with the outputs of image encoder. (a,c,e) are original images and (b,d,f) are synthesis images based on the image embeddings of original images.
Refer to caption: x9.png

### F.2 Text Condition Image Synthesis

In Fig. 9, we present some text-to-image synthesis examples from DEEM , demonstrating its capability to generate corresponding images based on given prompts.

Figure: Figure 10: Text-to-image generation examples with prompts. DEEM can generate vivid images based on input text conditions.
Refer to caption: x10.png

### F.3 Robustness Comparison

In Fig. 7, we present a comparative analysis of visual robustness results between our model, DEEM , and other state-of-the-art models: LLaVA (Liu et al., ), NeXt-Chat (Zhang et al., ), and MM-Interleaved (Tian et al., ). When encountering natural adversarial samples or out-of-distribution samples, the image encoder in their models will output incorrect semantic information, leading to incorrect category answers. In contrast, our method uses a diffusion model as the eyes of the large language model to inspect and correct the output features of the image encoder. This process eliminates incorrect semantic outputs from the image encoder, ultimately allowing the large language model to produce the correct category answer. This simple yet effective approach significantly enhances the model’s robustness and generalization capabilities.

### F.4 Image-Text Multimodal Dialogue

In Fig. 11, we show the image-text dialogue case examples of DEEM . Our model can input any interleaved layout of text-image data and simultaneously understand and generate text-image outputs in any interleaved layout, representing the future of next-generation multimodal dialogue.

Figure: Figure 11: Examples of image-text multimodal dialogue between human and DEEM . Text and image can be used as inputs or outputs, and multi-round dialogue is shown.
Refer to caption: x11.png

### F.5 Mask-Text Multimodal Dialogue

In addition to image-level input, DEEM also supports mask-text input to perform fine-grained region-level reasoning tasks. As shown in the Fig. 12, DEEM can accurately extract region semantics of the image based on the input mask and complete the corresponding instruction tasks.

Figure: Figure 12: Examples of mask-text multimodal dialogue between human and DEEM . Text and mask can be used as inputs and DEEM outputs the corresponding answer, and multi-round dialogue is shown.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: 1. The paper introduces a new benchmark, RobustVQA, specifically designed for evaluating the visual robustness capabilities of the large multimodal models.
2. The proposed method is the first to introduce the diffusion model into the image perception of large language models, which is simple but effective.
3. The proposed method trained end-to-end with interleaved text-image data can perform various multimodal tasks in a unified manner.
summary: This paper introduces the diffusion model into the image perception process of large language models to reduce excessive compression of visual details, thereby enhancing the model's robustness and its ability to mitigate hallucinations. An end-to-end multimodal model with interleaved text-image modeling capabilities is trained to perform various multimodal tasks in a unified manner. A new robustness benchmark for large multimodal models is developed to assess their visual robustness capabilities. The extensive experimental results demonstrate the effectiveness of the proposed method.
weaknesses: 1. The insight of the paper is the integration of the diffusion model into the image perception process of LLMs while training to correct potential semantic biases in the image encoder. It is crucial to validate the generalization capability of the proposed method across existing LMMs to enhance their image perception. However, the validation is somewhat limited, focusing solely on LLaVA, as shown in Table 5. Could the proposed method be flexibly applied to additional existing architectures, such as BLIP-2 and MiniGPT-4?
2. The claim that the proposed method can "correct potential semantic bias in the image encoder and alleviate excessive compression of visual details" seems intuitive. Are there any validations or qualitative comparisons between existing LMMs trained with and without this method? For instance, maybe comparisons of feature maps for image tokens encoded by the image encoder trained with and without the proposed method.
3. The contribution of the constructed RobustVQA benchmark seems somewhat limited, as it focuses primarily on evaluating visual robustness. Emphasizing the unique advantages of the RobustVQA benchmark over existing benchmarks through more extensive qualitative or quantitative comparisons with other benchmarks in some aspects would enhance its value.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please refer to weaknesses.
soundness: 3
strengths: 1. This paper design a new robustness benchmark RobustVQA for visual robustness capabilities of multi modal models.
2. DEEM is the first approach to enhance image perception in LLMs using diffusion models, making it novel.
3. This approach is highly versatile, demonstrating effective experimental results across benchmarks.
4. This paper is well written and easy to follow.
summary: This paper enhances large language models image perception by using diffusion models to align image encoder semantics, improving resilience to out-of-distribution data and reducing visual hallucinations. Without extra modules or parameters, DEEM outperforms on benchmarks like RobustVQA, POPE, and MMVP, using fewer parameters and less data. It strengthens visual tasks such as question answering, captioning, and image synthesis, showcasing robust perception and reduced hallucinations.
weaknesses: 1. Since diffusion models need to be utilized during both training and inference, the computational cost will be high.
2. If this model is used with an LLM that has a much larger number of parameters, how would its performance be affected?

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: Q1) Regarding RobustVQA, would it be possible to also run on close-source models (GPT4o/Gemini1.5Pro)? I am curious on how the close-source solutions perform against the proposed benchmark.

Q2) In Table 9, it states "Questions in the yes or no format can well evaluate the performance of the models on the RobustVQA benchmark, while questions in the multiple choice format are very random, and MM-interleaved tend to output the first option. Therefore, we
adopt yes or no format in our experimental settings." seems a bit unconvincing to me. Would LMM also be easier to guess the solution if there is only 2 choices for yes/no format?
soundness: 4
strengths: The paper is well-structured and it shows a good originally. This is a pioneering work in using diffusion models to specifically enhance the image perception capabilities of LMMs.
summary: This paper proposed a method to enhance the image perception ability of large multimodal models (LMMs), by leveraging the generative feedback of diffusion models. This reduced the visual hallucination and strengthen the perception ability in against out-of-distribution samples. Also a benchmark is developed, specifically designed for out-of-distribution or challenging samples. Extensive experiments showed that the proposed method achieves an improvement in a collection of benchmarks and downstream tasks.
weaknesses: W1) The detail information of RobustQA is missing. This makes the experiments result look less convincing. How many samples are in the benchmark? Please also include a diversity chart to show how robust the benchmark is.

W2) Figure 2 is difficult to follow. Please revise the figure, e.g. adding some step indicators, legends for arrows.

W3) Typo "obnly" in the last paragraph of Section 3.3.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the diffusion model’s role be optimized or reduced without compromising robustness? For instance, could a selective application of diffusion-based feedback lower training costs?
2. Could the impact of scaling down the diffusion model be evaluated further to assess how smaller models might perform in balancing robustness and efficiency?
3. Have you considered a comparative analysis of DEEM with approaches like "De-Diffusion Makes Text a Strong Cross-Modal Interface"?

https://openaccess.thecvf.com/content/CVPR2024/papers/Wei_De-Diffusion_Makes_Text_a_Strong_Cross-Modal_Interface_CVPR_2024_paper.pdf
soundness: 3
strengths: 1.	Innovation in Multimodal Integration: DEEM introduces diffusion models as “visual auditors” for LMMs, a novel approach for refining visual inputs and reducing errors from biased image encoders.
	2.	Efficiency: The approach reduces dependency on larger, more complex models by using a smaller image encoder (CLIP-ConvNext-B), which lowers computational requirements.
	3.	Robustness: Demonstrates resilience against adversarial and out-of-distribution data, enhancing the model’s suitability for diverse, real-world scenarios.
	4.	Extensive Evaluation: The paper provides a comprehensive analysis using custom (RobustVQA) and widely recognized benchmarks (POPE and MMVP), showing DEEM’s performance gains in both robustness and visual perception.
summary: The DEEM model integrates diffusion models with large multimodal models (LMMs) to enhance image perception capabilities, especially for out-of-distribution data. DEEM employs generative feedback from diffusion models to correct potential semantic biases in image encoders, resulting in more accurate and robust visual perception. Evaluations across benchmarks such as RobustVQA, POPE, and MMVP demonstrated improvements in visual robustness and reduced visual hallucinations compared to traditional methods. This enhancement is achieved without additional training parameters, using fewer resources and a smaller model scale.
weaknesses: 1.	Reliance on Diffusion Models: Although diffusion models enhance robustness, they also introduce computational complexity. Future versions might explore alternative approaches that achieve similar robustness with lower computational costs.
	2.	Generalization Limits: While DEEM shows strong performance for specific multimodal tasks, its generalization to other modalities (e.g., audio) remains unexplored.
	3.	Training Overheads: The training process, though optimized, may still be prohibitive for some use cases where computational resources are constrained.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: **Questions for the Authors**

1. **Scalability and Computational Efficiency**:
   - How does the use of diffusion models as an additional component impact the overall computational efficiency and memory usage, especially for larger versions of image encoders?
   - Have you explored any strategies for optimizing computational resources? If so, could you provide details on these optimization techniques and their effectiveness?

2. **Robustness Knowledge Forgetting**:
   - You mentioned that DEEM cannot completely eliminate robustness knowledge forgetting. Do you have insights or preliminary results on potential methods for further mitigating this issue?

3. **Failure Case Analysis**:
   - Could you provide more details or examples of cases where DEEM underperforms compared to other state-of-the-art methods?
   - What common patterns or factors have you identified in these failure cases, and what potential solutions might address these issues?

4. **Impact of High-Resolution Inputs**:
   - How does the increase in input image resolution from 256 to 448 pixels impact the performance and training dynamics? Are there diminishing returns at higher resolutions?
   - Have you evaluated the impact of image resolution on the computational requirements and time to convergence?

5. **Diffusion Model's Role**:
   - Could you elaborate on how the diffusion model corrects and aligns the semantic outputs of the image encoder during training? Are there specific examples or visualizations that illustrate this process?
   - How critical is the specific architecture or configuration of the diffusion model in achieving these results?

In summary, while the paper proposes combining diffusion models with LLMs to enhance the model's perceptual capabilities, this approach is not entirely novel, as seen in works like Transfusion (https://www.arxiv.org/abs/2408.11039). Additionally, if the sole purpose is to enhance the model's visual perception, introducing such a complex diffusion module may seem excessive. If a diffusion module is introduced, the authors should also focus on evaluating image generation capabilities. Alternatively, the authors should attempt to use simpler methods as baselines to highlight the necessity of introducing the diffusion module. The ablation studies in the paper do not seem to cover this aspect.
soundness: 2
strengths: **Strengths**

1. **Originality**:
   - The paper presents a novel approach, DEEM, which integrates diffusion models to enhance the visual perception capabilities of large language models (LLMs) for multimodal tasks. 

2. **Quality**:
   - The paper provides comprehensive empirical evidence to support the effectiveness of the proposed method. The authors perform extensive evaluations on newly constructed and well-known benchmarks (e.g., RobustVQA, POPE, and MMVP), demonstrating the method's robustness and performance improvements.

3. **Clarity**:
   - The paper is well-structured and clearly written, providing a detailed description of the DEEM architecture, training process, and experimental evaluation.

4. **Significance**:
   - DEEM significantly enhances the robustness and visual perception capabilities of LMMs, addressing a critical challenge in the field of multimodal machine learning.
summary: This paper introduces DEEM, a method that leverages the generative feedback of diffusion models to align the semantic distributions of image encoders, addressing issues faced by large multimodal models (LMMs) when encountering out-of-distribution data such as orientation, quantity, color, and structure. DEEM improves the resilience of LMMs against such data and reduces visual hallucinations without the need for additional training modules and with fewer training parameters. It demonstrates significant improvements in visual perception performance across several benchmarks (RobustVQA, POPE, and MMVP), exhibiting enhanced robustness and superior capability to mitigate model hallucinations while using fewer trainable parameters and less pre-training data, and being smaller in base model size. Extensive experiments show that DEEM enhances the performance of LMMs across various downstream tasks, including visual question answering, image captioning, and text-conditioned image synthesis.
weaknesses: **Weaknesses**

1. **Scalability Concerns**:
   - While DEEM demonstrates improved performance using fewer training parameters and a smaller base model size, the requirement to employ diffusion models as an additional component could potentially increase computational overhead.
   - As the model size continues to grow, will the computational burden brought by the diffusion models outweigh the performance gains? After all, as the model size increases, its capabilities significantly improve, and computational costs become the primary bottleneck. The introduction of diffusion might bring negative returns.

2. **Robustness Limitation**:
   - Although DEEM significantly enhances visual robustness, the paper acknowledges that the method cannot completely eliminate but only alleviate robustness knowledge forgetting during subsequent fine-tuning stages. This indicates a potential long-term limitation in maintaining the enhanced robustness through the entire lifecycle of model updates and task-specific fine-tuning.

3. **Limited Analysis of Failure Cases**:
   - The paper largely focuses on positive results and improvements brought by DEEM. However, it lacks an in-depth analysis of cases where the method might fail or underperform compared to existing approaches. In particular, in the example shown in figure 9 of the supplementary materials, many generated images exhibit significant errors in color, spatial arrangement, and orientation. These generation flaws somewhat undermine the paper's claim that diffusion models can enhance visual understanding, as diffusion models also cannot perfectly restore visual features.

Meta-review: This paper has received ratings of 8, 8, 8, 6, 6, where the reviewers have generally rated the paper positively, highlighting its innovative approach to enhancing visual robustness and mitigating hallucinations in multimodal models. 

The authors has proposed a framework DEEM, leveraging diffusion models to enhance the visual perception of large multimodal models (LMMs). The authors demonstrate that integrating generative feedback from diffusion models can address limitations of traditional image encoders, improving performance on tasks such as visual question answering, image captioning, and text-conditioned image synthesis.

Strength:
- Innovation in Methodology: The proposal to use diffusion models as a corrective mechanism for image encoders is novel and effective.
- Empirical Results are also promising. The model achieves state-of-the-art performance on several benchmarks, including RobustVQA, POPE, and MMVP, with notable improvements in robustness and hallucination mitigation.
- Scalability and Efficiency: The method demonstrates competitive performance with fewer parameters and less pre-training data compared to contemporary models.

Area for improvements:
- The computational requirements for integrating diffusion models may limit practical applications.
- The broader applicability of the approach to larger image encoders and other multimodal tasks requires further exploration.

The manuscript makes a significant contribution to the field of multimodal modeling by advancing the use of diffusion models in enhancing image perception. The empirical results are strong, and the methodology is robust and generalizable. While the computational overhead and scalability concerns are valid, they do not outweigh the novelty and impact of the work.

Decision: Accept (Spotlight)