Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models

## Abstract

Abstract Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs.
However, these models are prone to parametric knowledge conflicts, which arise from inconsistencies of represented knowledge between their vision and language components.
In this paper, we formally define the problem of cross-modality parametric knowledge conflict and present a systematic approach to detect, interpret, and mitigate them.
We introduce a pipeline that identifies conflicts between visual and textual answers, showing a persistently high conflict rate across modalities in recent LVLMs regardless of the model size.
We further investigate how these conflicts interfere with the inference process and propose a contrastive metric to discern the conflicting samples from the others.
Building on these insights, we develop a novel dynamic contrastive decoding method that removes undesirable logits inferred from the less confident modality components based on answer confidence.
For models that do not provide logits, we also introduce two prompt-based strategies to mitigate the conflicts.
Our methods achieve promising improvements in accuracy on both the ViQuAE and InfoSeek datasets.
Specifically, using LLaVA-34B, our proposed dynamic contrastive decoding improves an average accuracy of 2.24%.
Code and data are available at https://github.com/luka-group/vlm-knowledge-conflict .

## 1 Introduction

Large Vision-Language Models (LVLMs; OpenAI ; Anil et al. ; Liu et al. ) have demonstrated potent capabilities for perceiving and understanding information across different modalities.
These models typically consist of a visual encoder and a large language model (LLM), aligned by a projection layer (Li et al., ; Alayrac et al., ; Liu et al., ).
This alignment and collaboration mechanism between language and vision components allows users to input text and images simultaneously, breeding some of the wildest applications, including retrieving information based on a combination of textual and visual queries (Karthik et al., ) and accomplishing complex real-world tasks with multimodal agents (Zhang & Zhang, ; Zheng et al., ).

However,
the disentangled training processes and distinct learning resources leveraged by the vision and language components of an LVLM, respectively, inherently bring along inconsistencies in their learned representations, captured knowledge, as well as their influence during inference (Bartsch et al., ; Rabinovich et al., ).
Given that the visual encoder and the LLM are separately trained on different datasets with distinct training objectives, their parametric knowledge across language and vision modalities is susceptible to conflicts (Wang et al., ), potentially leading to hallucinations (Ji et al., ) and inconsistencies in prediction (Chang & Bergen, ).
As illustrated in Figure 1, we present a conflict case from an LVLM.
When asked a question about the same entity presented in two different modalities, the LVLM provides two contradictory answers.
Even though the visual encoder is able to recognize the Sydney Opera House
, the model still fails to integrate this information coherently across modalities.
This phenomenon reveals a crucial challenge: the disparity between the knowledge captured by the vision and language components of LVLMs.
However, there has been limited research on parametric knowledge conflicts within these models, especially concerning cross-modality conflicts.
Thus, in this paper, we systematically investigate the phenomenon of cross-modality parametric knowledge conflict as defined in Section 3.
We aim to address several principled research questions, as further detailed below:

Figure: Figure 1: A conflict case of different input modalities with the same information. The conflict still happens even when the visual components recognize the Sydney Opera House.
Refer to caption: x1.png

*RQ1: How to detect cross-modality parametric knowledge conflicts?*

In Section 4, we introduce a pipeline for detecting such conflicts using a multiple-choice question answering format focused on named entities.
Specifically, we present each named entity in different modalities and pose the same question about it.
The resulting answers derived from the knowledge of each modality are then compared to determine if a conflict exists.
Our findings reveal a persistently high conflict rate across various model scales and architectures, indicating that increasing model size alone does not resolve these conflicts.

*RQ2: How can cross-modality parametric knowledge conflicts be interpreted, especially how they intervene the inference process?*

Given the severity of knowledge conflicts in LVLMs, this intriguing question arises.
One might initially assume that such cross-modal conflicts would reduce the prediction confidence in the original answer due to conflicting parametric knowledge.
However, our analyses demonstrate that confidence cannot reliably distinguish between correct and incorrect answers, necessitating a more nuanced interpretation of these conflicts.
To address this issue, we propose a contrastive metric in Section 5 that more effectively identifies conflicting samples.
This metric suggests that cross-modality knowledge conflicts actually widen the information gap embedded in the tokens.

*RQ3: What strategies can be introduced to mitigate cross-modality knowledge conflicts at inference?*

Having gained an understanding of how these conflicts affect the inference, we seek to address this question.
Inspired by the strong discriminatory power of the contrastive metric, we propose a dynamic contrastive decoding method in Section 6.
This method selectively removes undesired logits inferred from the less reliable modality based on answer confidence.
Additionally, we propose two prompt-based strategies to mitigate cross-modality knowledge conflicts in cases where the model does not provide logits.
Our dynamic contrastive decoding method provides more consistent improvements.

In summary, the main contributions of this paper are threefold:
1) To the best of our knowledge, this is the first-of-its-kind work to define and study cross-modality parametric knowledge conflicts in LVLMs.
2) We propose a practical pipeline for detecting such conflicts, along with a metric that distinguishes conflicting samples from non-conflicting ones.
3) We introduce a dynamic contrastive decoding method to mitigate these conflicts, as well as two prompt-based strategies for resolving conflicts in closed-source models.

## 2 Related Work

Knowledge Conflict.
Knowledge conflict is a critical problem in context-specific tasks, such as machine reading comprehension (Longpre et al., ; Zhou et al., ; Wang et al., ) and information extraction (Wang et al., ; Fang et al., ; Xu et al., ; Wang et al., ; )
In the realm of LLMs, recent studies can be categorized into context-memory conflict, inter-context conflict, and intra-memory conflict (Xu et al., ).
The context-memory conflict and the inter-context conflict are concerned mainly in the process of Retrieval Augmented Generation (RAG).
They find that LLMs tend to overly rely on their own parametric memory when facing contradictory evidence (Xie et al., ; Wu et al., ).
The intra-memory conflict, on the other hand, is rooted in the pre-training corpus which contains inaccurate and misleading information (Bender et al., ; Lin et al., ; Kandpal et al., ).
The inconsistency of knowledge causes LLMs to generate outputs that are contradictory to each other when given different prompts with the same information (Elazar et al., ; Grosse et al., ), undermining their reliability.
In this context, prior work has not systematically studied this problem for LVLMs, which is exactly the focus of this work.

Robustness Issues of LVLMs.
Although LVLMs have demonstrated significant potential in understanding and reasoning over multimodal inputs, they also face several robustness challenges, including language bias (Niu et al., ; Zhang et al., ; Wang et al., ), hallucinations (Huang et al., ; Zhu et al., ), and the visual perception gap (Ghosh et al., ).
Language bias refers to the tendency of LVLMs to rely on language patterns learned during LLM pretraining (Niu et al., ; Zhang et al., ; Wang et al., ).
Hallucinations, which originate from LLMs, pertain to the discrepancies between generated contents and facts from either real-world or user inputs. (Huang et al., ; ).
The visual perception gap refers to the phenomenon that the LVLMs demonstrate proficient knowledge and visual recognition abilities but fail to link their visual recognition to this knowledge (Lee et al., ; Ghosh et al., ).
These issues often overlook the potential conflicts between the visual and textual components of the LVLM, which may contribute to the aforementioned challenges.

Inference-time Intervention.
Inference-time intervention encompasses a range of techniques designed to influence the inference or generation process of LLMs (Damera Venkata & Bhattacharyya, ; Li et al., ).
These techniques either directly manipulate the logits of the generated tokens or adjust the parameters of the LLM during inference.
One of the most notable strategies in this context is contrastive decoding (Li et al., ; Leng et al., ; Zhang et al., ), which mitigates undesired distributions by removing them from the original distribution.
Another approach involves modifying specific layers of the LLMs.
For instance, ITI (Li et al., ) adjusts model activation during inference by following a set of directions across several attention heads, enhancing the truthfulness of LLMs.
These methods provide a means for training-free adjustments to LVLMs, significantly reducing the cost compared to readjusting model parameters.

## 3 Preliminaries

Before diving into parametric knowledge conflicts in LVLMs, we will first outline key definitions relevant to our analysis and provide an overview of the general experimental setup.

### 3.1 Definitions

To ground our analysis, we need to define 1) a typical LVLM architecture, and 2) cross-modality parametric knowledge conflicts.

LVLM Architecture.
We focus on the general architecture that is adopted by a variety of LVLMs, including LLaVA (Liu et al., ), Blip (Li et al., ), and Qwen-VL (Bai et al., ).
Typically, these models consist of a visual encoder $V$, a projector $F$, and a language model LM.
Given a multimodal input $x_{m}=\{x_{v},x_{t}\}$, where $x_{v}$ is the visual input and $x_{t}$ is the textual input, LVLM first processes $x_{v}$ with $V$, resulting in $p_{v}=V(x_{v})$.
Then, through the projector $F$, $p_{v}$ is projected into the textual embedding space: $e_{v}=F(p_{v})$.
Finally, $x_{t}$ is embedded into the embedding space by the embedding layer of the LM, resulting in $e_{t}=\text{embed}(x_{t})$.
The language model then generates the output by the probability $p_{\text{LM}}(y|e_{v},e_{t})$.
So, a contemporary LVLM can be defined as $p_{\text{LM}}(y|F(V(x_{v})),\text{embed}(x_{t}))$.

Cross-Modality Parametric Knowledge Conflict.
Since training a large model from scratch is prohibitively costly,
LVLMs typically align a vision encoder onto an existing language model.
For example, LLaVA (Liu et al., ) aligns the pre-trained CLIP visual encoder ViT-L/14 (Radford et al., ) with Vicuna (Chiang et al., ), which have been separately trained on different data distributions, leading to potential inconsistent parametric knowledge (Grosse et al., ).

To elicit parametric knowledge, we propose to use answers from different modalities as the indicators of the specific parametric knowledge from each modality.
Specifically, given a multimodal input $x_{m}=\{x_{v},q\}$, where $q$ is the question regarding the entity in the image $x_{v}$, the output $y_{m}$ is generated by $p_{\text{LM}}(F(V(x_{v})),\text{embed}(q_{m}))$, which we define as the *visual answer*.
On the contrary, given a textual input $x_{t}=\{x_{e},q\}$, where $x_{e}$ is the textual description of a named entity and $q$ is the question to the named entity, the output $y_{t}$ is generated by $p_{\text{LM}}(\text{embed}(q_{t}))$, which we define as the *textual answer*.
If $y_{m}\neq y_{t}$, then a parametric knowledge conflict is identified.

### 3.2 Experimental Setup

#### 3.2.1 Datasets Construction

Original Datasets.
Following prior studies (Xie et al., ; Wu et al., ), we adopt the multiple-choice question answering (QA) as the form of evaluating cross-modality parametric knowledge conflicts.
We choose two tasks of knowledge-based visual question answering
about named entities:

- •
ViQuAE (Lerner et al., ) is a semi-automatically constructed dataset comprising 3.7K questions about named entities grounded in a visual context, built upon TriviaQA (Joshi et al., ).
The named entity in the original question is replaced with an image depicting it, requiring the model to answer the question based on the visual context provided.
- •
InfoSeek (Chen et al., ) is a dataset containing 1.3M questions about over 11K visual entities, designed to evaluate the performance of LVLMs in processing visual content while acquiring relevant knowledge.
The dataset is automatically constructed from templates of over 300 relations in Wikidata, ensuring a diverse set of questions.

**Table 1: Statistics of the constructed multiple-choice QA dataset.**
|  | ViQuAE | InfoSeek |  |  |
| --- | --- | --- | --- | --- |
|  | Original | MCQA | Original | MCQA |
| #samples | 3,697 | 3,010 | 73,620 | 3,000 |

Multiple Choices Construction.
Given that the original datasets are free-form question answering, we synthesize distractor choices for each question.
These distractor choices must be relevant to the questions to some extent but factually incorrect, to effectively evaluate the model’s ability to discern the correct answers.
To this end, we employ LLaMA-3-8B (AI@Meta, ) to synthesize relevant but incorrect distractor choices.
The prompt used in this process is listed in Appendix Section B.2.
We also down-sample the InfoSeek dataset to match the sample size of the ViQuAE dataset.
The statistics of the datasets are presented in Table 1.

#### 3.2.2 Evaluation Metrics

Since we adopt MCQA as the evaluation form, we can directly calculate the accuracy:

$$ $\text{Acc}=\frac{1}{N}\sum_{i=1}^{N}{\mathbbm{1}(y_{i}=\hat{y_{i}})},\vspace{- 2mm}$ (1) $$

where $N$ is the number of samples and $\hat{y_{i}}$ is the gold answer.
Moreover, to investigate parametric knowledge conflicts, we define the inconsistency between the generated answers as flip rate (FR):

$$ $\text{FR}=\frac{1}{N}\sum_{i=1}^{N}{\mathbbm{1}(y_{v_{i}}\neq y_{t_{i}})}, \vspace{-2.5mm}$ (2) $$

where $y_{v_{i}}$ is the visual answer and $y_{t_{i}}$ is the textual answer.
This metric indicates how many samples encounter conflicting answers between textual and visual modalities.

#### 3.2.3 Models

Following prior works regarding LVLMs (Zhang et al., ; Zhu et al., ), we choose the LLaVA series (Li et al., ) for evaluation, as they provide strong performance and a full range of model scales.
Moreover, to evaluate how the architecture of LVLMs affects the phenomenon of knowledge conflicts, we adopt InstructBlip (Dai et al., ) and Qwen-VL (Bai et al., ).

## 4 Detecting Parametric Knowledge Conflicts

In this section, we discuss the pipeline for detecting parametric knowledge conflicts in LVLMs and evaluate the severity of these conflicts.

**Table 2: Results of detecting cross-modality parametric knowledge conflict. We report accuracy (Acc), recognized accuracy (R. Acc), flip rate (FR) and conflict rate (CR).**
| Model | ViQuAE | InfoSeek |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc$\uparrow$ | R. Acc$\uparrow$ | FR$\downarrow$ | CR$\downarrow$ | Acc$\uparrow$ | R. Acc$\uparrow$ | FR$\downarrow$ | CR$\downarrow$ |  |  |
| LLaVA-7b | Textual | 75.65 | 78.43 | 41.68 | 21.36 | 52.74 | 54.55 | 70.13 | 42.85 |
| Visual | 53.26 | 58.11 | 22.11 | 27.27 |  |  |  |  |  |
| LLaVA-13b | Textual | 75.65 | 69.63 | 36.47 | 28.10 | 56.31 | 55.41 | 58.44 | 38.53 |
| Visual | 58.57 | 61.26 | 31.33 | 35.50 |  |  |  |  |  |
| LLaVA-34b | Textual | 82.46 | 82.32 | 24.90 | 20.53 | 66.02 | 64.07 | 43.72 | 28.57 |
| Visual | 69.14 | 77.95 | 44.35 | 48.92 |  |  |  |  |  |
| InstructBlip-7b | Textual | 81.73 | 80.42 | 55.35 | 20.56 | 50.53 | 53.68 | 59.74 | 40.16 |
| Visual | 43.09 | 45.63 | 35.17 | 38.10 |  |  |  |  |  |
| Qwen2-VL-7b | Textual | 79.30 | 78.56 | 28.65 | 22.46 | 63.24 | 62.77 | 22.51 | 20.35 |
| Visual | 67.97 | 72.37 | 61.69 | 60.61 |  |  |  |  |  |

### 4.1 Method

Inputs.
As defined in Section 3.1, the visual answer is generated by asking a question about the entity presented in the image, while the textual answer is induced by replacing the image with the textual description of the named entity.
To ensure that equal information is provided across modalities, we design distinct inputs for each, as illustrated in Figure 1.
Specifically, given a multimodal input $x_{m}=\{x_{v},q\}\in\mathcal{D}$, where $\mathcal{D}$ is the dataset, $x_{v}$ is the image containing the named entity, and $q$ is the question to the named entity in $x_{v}$, the visual answer is generated by:

$$ $y_{v}\sim p_{\text{VLM}}(x_{v},q)=p_{\text{LM}}(F(V(x_{v})),\text{embed}(q)).$ (3) $$

To generate the textual answer, we add an indicator prompt $p$ before the original question, informing the language model about the named entity in the question.
$p$ is written as This is an image of $named_entity.
Thus, the input of the textual answer becomes $x_{t}=p+q$.
The textual answer is then generated by:

$$ $y_{t}\sim p_{\text{VLM}}(x_{t})=p_{\text{LM}}(\text{embed}(x_{t})).$ (4) $$

Irrelevant Factor Mitigation in Conflict Detection.
The results generated from the aforementioned inputs can be regarded as the elicited parametric knowledge from LVLMs.
However, these answers are influenced by various other factors.
For example, the visual perceiver $V$ might fail to recognize the entity in $x_{v}$, resulting in a random guess.
These potential issues impede our ability to accurately detect cross-modality parametric knowledge conflicts.
To mitigate irrelevant factors, we first instruct the LVLM to identify the entity depicted in $x_{v}$.
If the model correctly predicts the named entity, we assume that the knowledge related to the named entity is stored in the parametric memory of $V$ and $F$, implying that any such conflict is not due to a lack of knowledge in $V$ and $F$.

Figure: Figure 2: Relationship of conflicting samples.
Refer to caption: x2.png

### 4.2 Metric

Despite efforts to mitigate irrelevant factors in the process of detecting cross-modality parametric knowledge conflict, certain factors remain difficult to disentangle.
For instance, the visual perceiver $V$ might recognize the entity in $x_{v}$, but be unable to link it to the parametric knowledge within the LVLMs through the projector $F$ (Ghosh et al., ).
Alternatively, the LVLM may be limited in its reasoning ability to relate the recognized named entity to the question.
We classify these potential limitations as the *performance gap*.
The relationship between conflict cases and performance gap cases is illustrated in Figure 2.
The performance gap leads to failures in generating the correct answer, resulting in an overall performance decline, which can be quantified by the accuracy difference $\Delta\text{Acc}=\text{Acc}_{\text{textual}}-\text{Acc}_{\text{visual}}$.
The number of flip samples attributable to the performance gap can be calculated as $N_{p}=N\times\Delta\text{Acc}$, while the total number of flip samples is $N_{f}=N\times\text{FR}$, where $N$ represents the total number of samples.
To assess the severity of the conflicts, we calculate its lower bound as $N_{kc}\geq N_{f}-N_{p}$.
Accordingly, the lower bound of the parametric knowledge conflict rate can be expressed as:

$$ $\text{CR}=\frac{N_{kc}}{N}\geq\frac{N_{f}-N_{p}}{N}=\text{FR}-\Delta\text{Acc} .\vspace{-2mm}$ (5) $$

### 4.3 Analysis

We conduct experiments with LVLMs following the aforementioned procedure, and the results are presented in Table 2.
We report the accuracy (Acc) on the complete evaluation set and the recognized accuracy (R. Acc) on the subset of the evaluation set recognized by the LVLM.
Additionally, we calculate the flip rate (FR) and the conflict rate (CR) based on the recognized evaluation set.

Performance.
For both datasets, the LLaVA-34b model demonstrates the highest accuracy for both textual and visual inputs.
However, a significant performance gap exists between the textual and visual answers.
The most pronounced performance gap in the LLaVA family is observed in the LLaVA-7b model, where the accuracy difference exceeds 20%.
This performance gap is attributed to the cross-modality parametric knowledge conflict and the aforementioned reasons.
Furthermore, there is a notable improvement in the recognized accuracy (R. Acc) across all models compared to the overall accuracy (Acc).
This indicates that the models perform better on recognized entities and that the recognition process effectively mitigates potential factors influencing the final performance.

Conflict Rate.
The flip rate (FR) decreases with increasing model size on both datasets, ranging from 55.35% to 24.90% on the ViQuAE dataset.
Concurrently, the $\Delta\text{Acc}$ also declines with larger model sizes, decreasing from 20.32% to 4.37% on the ViQuAE dataset.
This trend likely results from the improved ability of larger models to link visual perception to parametric knowledge and their enhanced reasoning ability, rather than a reduced likelihood of parametric knowledge conflicts in larger models.
When calculating the lower bound of the parametric knowledge conflict rate CR, a consistent pattern emerges across the datasets: LLaVA-7b/13b/34b exhibits values of 21.36%, 28.10%, and 20.53%, respectively.
This pattern suggests that regardless of the model’s scale and architecture, the likelihood of parametric knowledge conflicts remains relatively constant.

## 5 Interpreting parametric knowledge conflicts

The constantly large conflict rate across datasets highlights the phenomenon caused by cross-modality knowledge conflicts.
In this section, we will take a closer look, through the sample-wise perspective, at how parametric knowledge in visual components, i.e., the visual encoder $V$ and the projector $F$, causes cross-modality parametric knowledge conflict by intervening the inference process of the LLM.
In particular, we explore how these conflicts influence answer confidence and propose a metric that can serve as an indicator of the presence of such conflicts.

### 5.1 Is probability a reliable indicator of answer correctness?

Method.
Since the answer probability reflects the model’s confidence in a given response, it is natural to consider how parametric knowledge conflicts might affect this probability.
For instance, such conflicts may either reduce confidence in the original answer or introduce a more confident alternative answer.
Given that $\text{embed}(x_{e})$ and $F(V(x_{v}))$ might encapsulate different knowledge, this discrepancy can affect the probability distribution over possible answers, resulting in a shift in confidence in the final output.
To investigate how cross-modality parametric knowledge conflict influences answer confidence, we design experiments to determine whether the answer confidence can serve as an indicator of conflict and whether it can suggest the correctness of the answer.

To elicit the answer probability, we calculate the textual answer probability $p_{t}$ and the visual answer probability $p_{v}$ using Equation 3 and Equation 4.
Since we adopt MCQA as the task format, we extract the logits of the answer token, i.e. “A,” “B,” “C,” and “D” and apply the softmax function to them.
Thus, the extracted confidence can be presented as $c=\text{softmax}(\log(p[\text{A}]),\log(p[\text{B}]),\log(p[\text{C}]),\log(p[
\text{D}]))$, where $p[\text{A}]$ indicates the probability of token “A,” and so on.
Then, we use the following strategies to understand how visual components influence the inference:

- 1.
*Max confidence*: $\max(c_{t}[y_{t}],c_{v}[y_{v}]),$ where the most confident answer is considered correct.
- 2.
*Max confidence shift*: $\max(c_{t}[y_{t}]-c_{t}[y_{v}],c_{v}[y_{v}]-c_{v}[y_{t}]),$ where $y_{t}$ is the textual answer and $y_{v}$ is the visual answer, indicating that the modality with the most significant influence on the answer is deemed the dominant modality for the question.
- 3.
*Min variance*: $\min(\sigma(c_{t}[y_{t}]),\sigma(c_{v}[y_{v}])),$ where the answer with the least variance under disturbance is considered the final answer. We introduce disturbance through two methods: writing diverse prompts and applying the Monte Carlo dropout (Gal & Ghahramani, ).

**Table 3: Testing different answer correctness indicators based on answer confidence.**
| Method | ViQuAE |  |
| --- | --- | --- |
| Acc | R. Acc |  |
| Textual Answer | 75.65 | 78.43 |
| Visual Answer | 53.26 | 58.11 |
| Max Confidence | 54.22 | 60.14 |
| Max Confidence Shift | 54.29 | 60.14 |
| Min Variance Prompt | 55.51 | 61.41 |
| Min Variance Dropout | 46.51 | 50.72 |

Results.
The results of three strategies are listed in Table 3, and the complete experimental setup is described in Section B.1.
From these results, it is evident that none of the strategies based on token probability reliably selects the correct answer when conflicts arise between the textual and visual answers.
This suggests that:
1) Confidence is not necessarily reduced by conflicts.
The presence of a cross-modality parametric knowledge conflict does not inherently lower the confidence level of the answer.
Instead, the conflict often introduces an alternative answer with higher confidence, overshadowing the original, potentially correct answer.
This observation indicates that high confidence alone is not a reliable indicator of answer correctness in the presence of such conflicts.
2) Confidence shifts are not indicative of reliability.
The results show that a greater shift in confidence between the textual and visual answers does not necessarily correlate with the reliability of the final answer.
3) Cross-modality parametric knowledge conflict is not an uncertainty issue.
The table also reveals that methods based on variance do not contribute to the performance.
Although these methods attempt to select the more stable answer by selecting the answer with minimum variance in token probability, the results show reductions in accuracy.
This implies that minimizing variance does not effectively address the underlying knowledge conflicts.

### 5.2 Contrastive metric as indicator of conflicts

Method.
To effectively understand how conflicting knowledge affects the inference, we utilize the concept of Contrastive Decoding (Li et al., ).
Its objective, which subtracts an undesired distribution from the original distribution, serves as a metric for evaluating the degree of divergence between the two distributions.
Given that we are using MCQA as the task format, our focus is specifically on the distribution of the answer token, particularly the first token.

Specifically, given a multimodal input $x_{m}=\{x_{v},q\},$ where $x_{v}$ is the image and $q$ is the question, and a textual input $x_{t}=\{x_{e},q\}$, where $x_{e}$ is the textual description of the named entity in $x_{v}$, the predicted first token distribution of answers for each modality can be represented as Equations 3 and 4.
The contrastive objective can then be written as:

$$ $\log(p_{cd})=\log(p_{v})-\log(p_{t})=\log(\frac{p_{\text{VLM}}(y_{v}|x_{v},q)} {p_{\text{VLM}}(y_{t}|x_{e},q)})=\log(\frac{p_{\text{LM}}(y_{v}|F(V(x_{v})), \text{embed}(q))}{p_{\text{LM}}(y_{t}|\text{embed}(x_{e}),\text{embed}(q))}).$ (6) $$

Ideally, if $F(V(x_{v})$ and $\text{embed}(x_{e})$ provide the same information for $q$, Equation 6 should be equal to 0.
However, due to the parametric knowledge conflict between the visual components and the LLM, $V(F(x_{v}))$ may not embed the same knowledge as $\text{embed}(x_{e})$, leading to $log(p_{cd})\not\approx 0$.
Thus, $|\log(p_{cd})|$ can be interpreted as the degree of difference between $V(F(x_{v}))$ and $\text{embed}(x_{e})$.
Additionally, the contrastive decoding objective also allows us to elicit visual memories by eliminating the influence of textual knowledge.
The analyses of the elicited memories are listed in Appendix A.

Figure: Figure 3: Distribution of the contrastive metric on all samples, samples with modality-consistent answers, and samples with modality-conflict answers. The dashed lines indicate the medians.
Refer to caption: x3.png

Result.
In Figure 3, we present the distribution of the contrastive metric, specifically separating samples with consistent answers across modalities from those with conflicting answers.
The figure reveals a significant disparity between the consistent and conflicting samples.
Most consistent samples fall within the range of 0-0.6, while conflicting samples exhibit greater variability, with an average median of 1.46.
This similar trend suggests that the extent of conflicts, as measured by the contrastive metric, is relatively consistent across different models, despite variations in model scales and architectures, implying that the cross-modality parametric knowledge conflicts are not solely dependent on the model’s architecture or size but are intrinsic challenges that persist across current training datasets.
The figure also suggests that the contrastive metric is effective in distinguishing between consistent and conflicting answers.
From the perspective of the contrastive metric, it quantifies the divergence between the knowledge encoded in the visual components and the LLM.
Thus, the misaligned knowledge leads to the information gap embedded in the tokens of different modalities, which is ultimately presented by the conflicting answer.

## 6 Mitigating parametric knowledge conflicts at inference time

Having established an understanding of cross-modality parametric knowledge conflicts, we now shift our focus to strategies for mitigating these conflicts.
Since the contrastive metric has proven effective in distinguishing conflicting samples from consistent ones, we first propose a strategy that leverages the principles of contrastive decoding.
Moreover, we also design an alternative approach based on prompting for models that do not provide access to logits during inference.

### 6.1 Dynamic Contrastive Decoding

Method.
In an ideal application of contrastive decoding, we would have an a priori knowledge of the logits, which enables us to define the undesired logits.
That is to say, to resolve cross-modality parametric knowledge conflicts, the logits from the incorrect, conflicting modality should be excluded from those of the correct modality.
However, in real-world scenarios, without external validation, it is impossible to definitively determine the correctness of an answer.
Therefore, we propose using the model’s answer confidence as a trend for correctness, also treating it as a scaling factor for the original logits.
We then apply these scaled logits to the contrastive decoding algorithm, formulating the dynamic contrastive decoding (DCD).
This approach adjusts the contrastive decoding objective by incorporating confidence as a dynamic factor to more accurately measure the difference in information embedded by the textual and visual components.

Specifically, given the textual answer $y_{t}$ with its probabilities $p_{t}(y_{t}|x_{e},q)$ and the visual answer $y_{v}$ with its probabilities $p_{v}(y_{v}|x_{v},q)$, we first calculate the confidence for each answer as follows:

$$ $\displaystyle c_{t}=\max(\text{softmax}(\log(p_{t}[\text{A}]),\log(p_{t}[\text {B}]),\log(p_{t}[\text{C}]),\log(p_{t}[\text{D}]))),$ (7) $\displaystyle c_{v}=\max(\text{softmax}(\log(p_{v}[\text{A}]),\log(p_{v}[\text {B}]),\log(p_{v}[\text{C}]),\log(p_{v}[\text{D}]))),$ (8) $$

where $p[\text{A}]$ indicates the probability for token “A,” and similarly for other tokens.
Next, the scaled logits are computed as $s_{t}=c_{t}\times\log(p_{t})$ and $s_{v}=c_{v}\times\log(p_{v})$.
To assess which modality is more likely to provide the correct answer, we view the confidence as the likelihood, selecting the modality with the higher confidence.
However, as discussed in Section 5.1, confidence alone is insufficient to determine correctness.
Therefore, we subtract the scaled logits of the less confident modality from those of the more confident one.
This leads to the application of contrastive decoding on the scaled logits, conditioned by the answer confidence:

$$ $\log(p_{cd}(y|x))=\begin{cases}c_{t}\times\log(p(y_{t}|x_{e},q))-c_{v}\times \log(p(y_{v}|x_{v},q)),&\text{if }c_{t}>c_{v}\\ c_{v}\times\log(p(y_{v}|x_{v},q))-c_{t}\times\log(p(y_{t}|x_{e},q)),&\text{ otherwise.}\end{cases}\vspace{-2mm}$ (9) $$

**Table 4: Results of the dynamic contrastive decoding compared to the baselines. Bold indicates best results and underline indicates second bests.**
| Model | Method | ViQuAE | InfoSeek |  |  |
| --- | --- | --- | --- | --- | --- |
| Acc | R. Acc | Acc | R. Acc |  |  |
| LLaVA-7b | Textual Answer | 75.65 | 78.43 | 52.74 | 54.55 |
| Visual Answer | 53.26 | 58.11 | 22.11 | 27.27 |  |
| DCD | 76.49 (+0.84) | 79.51 (+1.08) | 54.90 (+2.16) | 58.87 (+4.32) |  |
| LLaVA-13b | Textual Answer | 75.65 | 69.63 | 56.31 | 55.41 |
| Visual Answer | 58.57 | 61.26 | 31.33 | 35.50 |  |
| DCD | 76.58 (+0.93) | 74.14 (+4.51) | 58.03 (+1.72) | 56.52 (+1.11) |  |
| LLaVA-34b | Textual Answer | 80.99 | 82.32 | 66.02 | 64.07 |
| Visual Answer | 69.14 | 77.95 | 44.35 | 48.92 |  |
| DCD | 83.35 (+2.36) | 85.33 (+3.01) | 68.14 (+2.12) | 67.72 (+3.65) |  |
| InstructBlip-7b | Textual Answer | 81.73 | 80.42 | 50.53 | 53.68 |
| Visual Answer | 43.09 | 45.63 | 35.17 | 38.10 |  |
| DCD | 82.47 (+0.74) | 80.59 (+0.17) | 50.53 (+0.00) | 54.38 (+0.70) |  |
| Qwen2-VL-7b | Textual Answer | 79.30 | 78.56 | 63.24 | 62.77 |
| Visual Answer | 67.97 | 72.37 | 61.69 | 60.61 |  |
| DCD | 80.76 (+1.46) | 80.59 (+2.03) | 64.30 (+1.06) | 63.34 (+0.57) |  |

Results.
Table 4 presents the accuracy and the recognized accuracy for different methods across the ViQuAE and InfoSeek datasets.
Across both datasets and all model sizes, DCD consistently outperforms both the textual and visual answers.
For instance, in the LLaVA-7b model, DCD improves the accuracy from 75.65% to 76.49% on the ViQuAE dataset.
Similarly, on the InfoSeek dataset, accuracy increases from 52.74% to 54.90%.
These improvements are even more pronounced in the larger models.
For example, in the LLaVA-34b model, DCD increases accuracy by 2.36% on the ViQuAE dataset and by 2.12% on InfoSeek, indicating its potential in models with larger scales.

DCD demonstrates particularly significant gains in recognized accuracy (R. Acc).
For instance, on the InfoSeek dataset, the recognized accuracy for the LLaVA-34b model increases by 3.65% when using DCD compared to the textual answer.
This trend is consistent across all model sizes, indicating that DCD is particularly effective in improving the performance on recognized entities.
The improvement in recognized accuracy is likely due to the fact that the visual answers within the recognized set are expected to contain more relevant information than those in the unrecognized set, as the visual components have some prior knowledge of these entities.
Consequently, the DCD can more effectively leverage this information to discern which option is correct.

### 6.2 Prompting Strategy

Method.
Since not all models provide the logits of the generated contents, we propose two prompt-based improvement strategy for those models.
To address cross-modality parametric knowledge conflict, we design two types of prompts and the details of these prompts are provided in Section B.2.

- 1.
*Reminder prompt.*
Once a knowledge conflict is detected ,
the model is prompted to regenerate the answer, but this time with a reminder that highlights the presence of conflicting knowledge.
- 2.
*Answer prompt.*
Since both textual and visual answers are already generated during the detection process, this prompt asks the model to determine which one is correct.

Results.
Table 5 presents the results of prompt-based improvements using two strategies across two datasets and different model sizes.
The effectiveness of these strategies varies depending on the model size.
For smaller models, both prompts negatively impact performance across both datasets, with accuracy dropping by at least 1.07% on the ViQuAE dataset and 0.86% on the InfoSeek dataset.
This suggests that smaller models may struggle to handle prompts reminding them of potential knowledge conflicts, as they seem unable to discern which answer is correct.
Furthermore, presenting smaller models with conflicting answers seems to introduce additional confusion, as evidenced by the more substantial accuracy declines.
In contrast, larger models are more effective at processing the information provided in the prompts, demonstrating an accuracy gain of 4.48% on the ViQuAE dataset and 8.08% on the InfoSeek dataset.
These results indicate that larger models are better equipped to interpret and respond to the information in the prompt, likely due to their more advanced reasoning and understanding capabilities, which enable them to determine which modality is more reliable in resolving the conflict.
Overall, these findings indicate that the effectiveness of prompt-based conflict resolution strategies improves with model scale, particularly when the prompt provides the model with both conflicting answers, aiding in conflict resolution.

**Table 5: Results of the prompt-based strategies compared to the baselines. Since the inputs of this experiment are the same as the one of the visual answer except for the prompt, we compare them to the results of the visual answer. Bold indicates best results and underline indicates second bests.**
| Method | ViQuAE | InfoSeek |  |  |
| --- | --- | --- | --- | --- |
| Acc | R. Acc | Acc | R. Acc |  |
| LLaVA-7b |  |  |  |  |
| Visual Answer | 53.26 | 58.11 | 22.11 | 27.27 |
| Reminder Prompt | 53.99 (-1.66) | 57.25 (-2.53) | 21.25(-0.86) | 27.99 (+0.72) |
| Answer Conflict Prompt | 54.58 (-1.07) | 58.51 (-1.27) | 20.23 (-1.88) | 27.39 (+0.12) |
| LLaVA-13b |  |  |  |  |
| Visual Answer | 58.57 | 61.26 | 31.33 | 35.50 |
| Reminder Prompt | 58.57 (+0.00) | 61.26 (+0.00) | 35.53 (+4.20) | 38.10 (+2.60) |
| Answer Conflict Prompt | 57.59 (-0.98) | 59.67 (-1.59) | 34.27 (+2.94) | 39.06 (+3.56) |
| LLaVA-34b |  |  |  |  |
| Visual Answer | 69.14 | 77.95 | 44.35 | 48.92 |
| Reminder Prompt | 72.99 (+3.85) | 79.28 (+1.33) | 45.15 (+0.80) | 49.62 (+0.70) |
| Answer Conflict Prompt | 73.62 (+4.48) | 79.66 (+1.71) | 52.43 (+8.08) | 53.68 (+4.76) |

## 7 Conclusions

In this paper, we introduce the concept of cross-modality parametric knowledge conflicts in LVLMs, a significant issue arising from the misalignment between visual and textual modalities.
We propose a systematic approach to detect these conflicts, revealing a persistently high conflict rate across all model sizes.
Our findings indicate that simply scaling up models does not resolve these conflicts, highlighting the need for targeted intervention strategies.
To address these challenges, we propose the contrastive metric, which effectively identifies conflicting samples by measuring the information gap between modalities.
Building on this, we introduce the dynamic contrastive decoding (DCD), which selectively removes unreliable logits to improve answer accuracy.
For models without access to logits, we propose two prompt-based strategies.
These approaches collectively improve model performance.
On LLaVA-34B, the dynamic contrastive decoding achieves an accuracy improvement of 2.36% on the ViQuAE dataset and 2.12% on the InfoSeek dataset.
Our contributions advance the understanding of cross-modality parametric knowledge conflicts in LVLMs and provide practical solutions to mitigate these conflicts, leading to more robust and accurate multimodal inference.

Future work may extend our method to broader multimodal data including multi-images and videos (Wang et al., ; Li et al., ), or apply our method to enhance multimodal in-context learning (Xu et al., ) and retrieval-augmented generation (Chen et al., ). Additionally, our method may contribute to ensuring trustworthiness of LVLMs in risk-sensitive domains, such as the biomedical field (Li et al., ; Chaves et al., ).

## Ethics Statement

Our study highlights a critical concern in recent LVLMs: the parametric memories of the vision and language components are prone to conflicts.
This issue underscores the potential limitations of these models, as they may produce inconsistent or unreliable outputs if these conflicts are not properly addressed.
As researchers, our goal is to mitigate these risks while maximizing the benefits.

## Reproducibility Statement

Our experiments are conducted using five open-source LVLMs to ensure reproducibility.
To facilitate replication of our results, we have provided the prompts used in our experiments in Section B.2.
Additionally, the datasets utilized in our study are included in the supplementary materials for further reference.

## Appendix A Interpreting Cross-modality Knowledge Conflicts

**Table 6: Examples of elicited textual and visual memories using the contrastive decoding objective.**
| Question | Textual Memory | Visual Memory |
| --- | --- | --- |
| In what city did Bruce Lee grow up? | Hong Kong. | San Francisco, California, USA. |
| George Harrison was deported from which city where the early Beatles had long term engagements, because of his youth? | George Harrison was deported from Liverpool, England because of his youth. | George Harrison of the Beatles was deported from Hamburg because he lied about his age when trying to enter Germany legally. |
| What species of fly has the Latin name calliphora vomitoria? | Calliphora vomitoria is a species of fly commonly known as the ”vomiting fly” or ”fruit fly.” It is a small, yellowish-brown insect with a short life cycle and a strong ability to reproduce quickly. The species is widely distributed throughout the world and is often found near fruit or vegetable sources. | Calliphora vomitoria is commonly known as vomiting fly or green bottle fly. It belongs to family Calliphoridae which includes blowflies and horseflies. These insects play an essential role in decomposition processes by feeding on decaying organic matter… |
| What is the name of Bob Marley’s greatest hits album? | The name of Bob Marley’s greatest hits album is “Bob Marley and the Wailers: Greatest Hits.” | Live Forever Volume Two by Various Artists featuring Bob Marley & The Wailers includes performances captured live during concerts throughout his career culminating with performances shortly before his passing in May of 1981. |

The contrastive decoding objective described in Section 5 offers a valuable tool for examining the memory embedded within the visual components of LVLMs.
Specifically, the contrastive decoding metric can be reformulated in an autoregressive form:

$$ $p_{cd}(y|x)=\prod_{i=1}^{n}p_{cd}(y_{i}|x,y_{<i})=\prod_{i=1}^{n}\frac{p_{ \text{LM}}(y_{v}|F(V(x_{v})),\text{embed}(q),y_{<i})}{p_{\text{LM}}(y_{t}| \text{embed}(x_{e}),\text{embed}(q),y_{<i})},$ (10) $$

where $x$ is the inputs from both modalities and $y_{<i}$ indicates the tokens generated before step $i$.
This autoregressive form of contrastive decoding metric allows us to elicit visual memory from the visual components by removing the influence of textual knowledge.
We accomplish this by transforming the question into a free-form query without predefined options and then examining the elicited memory of the visual components.
The examples of the elicited memories are listed in Table 6.

From these memories, several observations can be made:

- 1.
LLM is better at memorizing date and location.
This aligns intuitively with the nature of the LLM’s training process, where such factual knowledge frequently appears in the text corpora.
It corresponds well with the expectation that language models acquire structured knowledge from reading-based data.
- 2.
Visual components are better at memorizing the correlation between an entity and its names and the relationship among entities.
For example, when asked the common name for Calliphora Vomitoria, the LLM fails to answer correctly, while the visual answer is correct.
This is likely due to the training objective of aligning visual components with the LLM, during which visual components learn entity-specific knowledge by mapping images to the language space.

## Appendix B Experimental Details

### B.1 Experimental Setup

Confidence Analysis.
We will describe the experimental setup of the *Min variance* strategy in Section 5.1.
For both settings, we sample 10 times with disturbance.
For the prompt disturbance, we ask the LLaMA-3-8b (AI@Meta, ) to rephrase the original prompt to obtain 10 different prompts and generate the answer with each of them.
For the dropout disturbance, we set the dropout rate to 0.1 and sample 10 times.
Then we extract the confidence of the gold answer and calculate the variance.

### B.2 Prompts

The details of the prompts used in our experiments are listed here.
The prompt to generate false options is in Table 7.
The reminder prompt to mitigate knowledge conflicts is in Table 8.
The answer conflict prompt to mitigate knowledge conflicts is in Table 9.

**Table 7: Prompt for generating false options to construct the multiple-choice question answering datasets.**
| Given the question and its gold answer, please generate a multiple choice version of this question. Note that the wrong choices should be relevant to the question and the gold answer should be exactly copied from what is given. You can randomly put the gold answer wherever you want. Please output as a json format: {“A”: Answer A, “B”: Answer B, “C”: Answer C, “D”: Answer D}. No further explanation or note. |
| --- |

**Table 8: Reminder prompt to mitigate cross-modality parametric knowledge conflicts.**
| You are an expert at question answering. Given the question, please output the answer. No explanation and further question. Be aware that your visual memory might differ from your textual memory, causing a conflict in your knowledge. |
| --- |

**Table 9: Answer conflict prompt to mitigate cross-modality parametric knowledge conflicts.**
| You are an expert at question answering. Given the question, please output the answer. No explanation and further question. Be aware that your visual memory might differ from your text memory, causing a conflict in your knowledge. Your text memory is: {textual answer} and your visual memory is: {visual answer}. |
| --- |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Why was an LLM (e.g., LLaMA) used instead of a VLM to generate incorrect distractor choices during dataset construction? The prompts in the appendix indicate that the LLM generates incorrect yet relevant choices. However, as LLaMA is a single-modality model without image input capabilities, it implies the model does not have access to the complete original question when synthesizing relevant choices.

2. What is the rationale behind down-sampling the InfoSeek dataset to match the size of the ViQuAE dataset? Comparisons should focus on models, not datasets.

3. Does the flip rate account for cases where both the visual and text answers are incorrect (but not equal)? For subsets entirely consisting of such cases, N_p=N×0=0 and N_f=N×1=N resulting in CR=1. This scenario is more likely attributable to a lack of knowledge or weak reasoning abilities in the LLM rather than knowledge conflicts.

4. In Table 2, why is the R.Acc lower than the Acc for the Qwen2-VL-7b model on the InfoSeek dataset? Could the authors provide a possible explanation for this observation?

5. In Section 4.3, the authors claim that the knowledge conflict rate (CR) is consistent across datasets, regardless of model scale. However, Table 2 shows that for the InfoSeek dataset, as the LLaVA model size increases, the CR decreases, with the reduction becoming more pronounced with larger models. This observation appears to contradict the authors' claim.

6. What is the meaning of the proposed contrastive metric? Section 5.2 states that the contrastive metric is designed to differentiate between consistent and conflicting answers. However, Equation 6, which calculates the metric, uses the logits of both the text and visual answers. If the logits for both approaches are already available, why not directly decode the final answers and determine conflicts by checking their equivalence?

7. In Section 5.1, the authors mention that confidence is unreliable for indicating the more reliable modality. Yet, in Section 6.1, confidence is used as a parameter to scale the logits, followed by a subtraction operation (Equation 7). Could the authors elaborate on why confidence is assumed to be reliable in this context and the significance of this subtraction?

8. The explanation of why larger models benefit more from the prompting strategy is unconvincing. The assumption that incorrect predictions stem from cross-modality knowledge conflicts contrasts with the demonstration that such conflicts exist across various model scales. If these conflicts consistently occur across modalities, what enables larger models to implicitly discern which modality's knowledge is more reliable?
soundness: 2
strengths: 1. The study addresses a generally prevalent and practically valuable problem and proposes a new perspective on the causes of hallucinations in vision-language models.

2. The presentation is clear and comprehensible, with intuitive and illustrative figures effectively conveying the proposed ideas.

3. The experiments are thorough and comprehensive, covering a variety of model sizes and architectures.

4. The proposed method demonstrates strong performance, resulting in significant improvements over baseline models.
summary: This work introduces the concept of knowledge conflict in vision-language models (VLMs) and designs a pipeline to detect such conflicts, tested across various VLMs. Additionally, it proposes a decoding method to mitigate the impact of these conflicts through denoising during the decoding process, thereby enhancing the performance of VLMs on visual question-answering (VQA) tasks.
weaknesses: 1. The definition of the problem is somewhat unclear, with certain edge cases omitted, which could pose potential issues (see Question 3).

2. Some result analyses are inconsistent with the quantitative data presented in the tables (see Questions 4 and 5).

3. The motivation behind the proposed method is not sufficiently explained, particularly regarding the rationale for its design (see Questions 6 and 7).

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: - The main motivation of this paper is to study conflicts between the LLM knowledge and VLM knowledge about the same topic. However, I am not completely convinced by the term 'conflict' for this scenario because the visual modality merely adds new knowledge to the LLM. Any conflict that arises from this addition can be solely because the training process could not bridge the gap between visual modality and language modality correctly, which has been termed as performance gap in this paper. What other reasons could there possibly be for this conflict, which is being captured in the Conflict rate? I would appreciate more discussion on this point in the rebuttal. Apologies if I have missed the point of this paper.

- How exactly is $\Delta$ accuracy calculate? Is it calculated based on the accuracies on the complete evaluation set? If yes, it might change my interpretation of the Conflict Rate outlined under Weaknesses.

- Suggestion to also report $\Delta$ accuracy in the table for easier interpretation of the results.
soundness: 1
strengths: **Well-written paper**: This paper is well-written with a clear motivation for exploration on this topic, well-structured sections and flow of contents. Experimental results are conveyed clearly and hypothesis are discussed extensively in the main text.


**Important Topic**: Topics related to parametric knowledge are always an important research topics, with those pertaining to VLMs becoming even more important as VLMs become mainstream in day-to-day use.
summary: This paper studies the phenomenon of cross-modality parametric knowledge conflict in vision language models. Specifically, when a VLM provides different answers with and and without the vision modality (with effective textual substitute for visual input), those instances are considered as cross-modality knowledge conflicts. To study this scenario, the authors construct a multiple-choice question answering dataset from two datasets: ViQUAE and InfoSeek where the questions are based on named entities presented in the accompanying image. They first extract answers based on both image and text input (visual accuracy), and then extract answers from the VLM with textual input only (textual accuracy). Their exploration shows that there can be large gaps between the textual accuracy and visual accuracy of contemporary VLMs. The accuracy on the subset of data where the VLM succeeds at recognizing the named entity is used for further evaluations. The QA accuracy gap on this subset can be attributed to knowledge conflicts and performance gaps, according to the authors. Performance gaps refer to the inability of the model to connect the visual encodings to the underlying parametric knowledge in LLMs in order to answer the question correctly. The conflict rate is calculated as the difference between flip rate (cross-modality inconsistency) and performance gap (calculated as diff. Of textual and visual acc. On the subset). Further exploration shows that confidence measured using logits is not a good indicator of instances with parametric knowledge conflict. However, a contrastive interpretation of the logits serves as a good indicator of instances where the modalities lead to knowledge conflict. Further, this paper proposes a dynamic contrastive decoding method and prompting method to mitigate cross-modality knowledge conflicts and shows significant improvements in QA accuracy for VLMs.
weaknesses: **Soundness of the cross-modality parametric knowledge conflict explanation**: 

The authors outline an explanation of the cross-modality knowledge conflict in Sec. 4.2. Following the provided explanations, these are the four categories of samples present in the evaluation set of recognized accuracy (also represented in Fig. 2):

1. Correct Text, Correct Visual
2. Correct Text, Incorrect Visual
3. Incorrect Text, Correct Visual
4. Incorrect Text, Incorrect Visual

According to the provided definitions in 4.1,

Flip Rate = size(2) + size(3)

$\Delta$Acc. = size(1) + size(2) - size(1) - size(3) = size(2) - size(3)

So, conflict rate = Flip Rate - $\Delta$Acc. = size(3) * 2

Please correct me if my understanding seems wrong. This seems like an inadequate measure for the cross-modality conflict since it only captures the size of category 3. Specifically, I think that the samples that are being attributed to performance gap (primarily category 2) should also be counted towards conflict since at least a part of this category implies that the visual-language knowledge is at odds with the language-only knowledge.

**Dynamic Decoding Method**: While this method does achieve significant improvements, this method lacks practical use because it requires that the user knows the named entity being shown via image to the VLM. If the user already has this information, then it would make more sense to simply pose this question to a capable LLM.

**Limited Generalizability of Results**: The results presented in this paper are specific to MCQ questions only. The MCQ format does make it easy to study this phenomena, however, it may not generalize to open-domain natural language answers which are the dominant format of interaction with current LLMs and VLMs.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Adding discussion of knowledge conflicts on how to multimodally generate tasks will be much helphul for the community.
soundness: 3
strengths: - The paper introduces and formally defines a critical, underexplored issue in LVLMs, cross-modality parametric knowledge conflict, and its implications for model inference.
- A well-structured pipeline for conflict detection, interpretation, and mitigation provides a comprehensive framework for handling multimodal inconsistencies.
summary: This paper addresses the issue of parametric knowledge conflicts within LVLMs. It defines this cross-modality conflict problem and develop a systematic pipeline to detect, interpret, and mitigate such conflicts, revealing a consistently high conflict rate across modelse. They introduce a contrastive metric to differentiate conflicting samples and propose a dynamic contrastive decoding method that selectively removes low-confidence logits, which improves model accuracy.
weaknesses: This work is compelling, as multimodal knowledge plays a vital role in advancing LVLMs. However, a fundamental challenge remains: for LVLMs, the core focus is still the generation task. The approach in this article appears limited to classification and token generation, focusing mainly on calibrating output probability. Addressing this challenge more directly is crucial for the current stage of LVLM development.

In particular, you may need to consider Image Caption or CoT Reasoning tasks, like Coco, ScienceQA, M3CoT.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weaknesses above.

(1) Can the authors provide more evaluations beyond knowledge intensive tasks?

(2) More justification about why choosing this decoding method?

(3) Is the problem solvable with improving LVLM training?
soundness: 3
strengths: The paper raises and formally defines the phenomenon of cross-modality parametric knowledge conflicts in LVLMs.

The authors not only detect and quantify the conflicts but also delve into their interpretation and propose practical mitigation strategies.

The introduction of a dynamic contrastive decoding method shows promise in effectively reducing knowledge conflicts during inference without the need for additional training, which is resource-efficient.

The paper provides experimental results demonstrating the persistence of knowledge conflicts across various model scales and architectures, highlighting the significance of the problem.
summary: This paper addresses the issue of cross-modality parametric knowledge conflicts in Large Vision-Language Models (LVLMs). The authors identify that inconsistencies between the visual encoder and the language model components can lead to contradictory outputs when processing the same entity presented in different modalities. They propose a pipeline for detecting these conflicts, introduce a contrastive metric to interpret their impact on inference, and suggest mitigation strategies like dynamic contrastive decoding.
weaknesses: The study evaluates the proposed methods on only two knowledge-intensive datasets. This narrow scope raises concerns about the generalizability of the findings across different domains and types of data. The paper should consider adding other evaluations like MMBench, Vibe Eval, and Mistral evals to justify that the method can generalize well.

The introduction and explanation of the contrastive metric could be more detailed. Additional clarification on how it effectively distinguishes conflicting samples would strengthen the paper.

While the paper proposes using a decoding method to solve the problem, more analysis should be given to study whether the problem comes from model training or inference. While a decoding based is lightweight, I wonder whether improving LVLM training can solve the problem too.