Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention Reallocation
Authors: Wenbin An
Jiahao Nie
Feng Tian
Haonan Lin
mingxiang cai
Yaqiang Wu
QianYing Wang
Xiaoqin Zhang
Shijian Lu
Venue: NeurIPS 2025 oral ()
Keywords: Multimodal Large Language Models
Multimodal Retrieval Augmented Generation
Primary area: deep_learning

Abstract: Despite their recent progress, Multimodal Large Language Models (MLLMs) often struggle in knowledge-intensive tasks due to the limited and outdated parametric knowledge acquired during training. Multimodal Retrieval Augmented Generation addresses this issue by retrieving contextual knowledge from external databases, thereby enhancing MLLMs with expanded knowledge sources. 
However, existing MLLMs often fail to fully leverage the retrieved contextual knowledge for response generation. We examine representative MLLMs and identify two major causes, namely, attention bias toward different tokens and knowledge conflicts between parametric and contextual knowledge. To this end, we design Adaptive Logits Fusion and Attention Reallocation (ALFAR), a training-free and plug-and-play approach that improves MLLM responses by maximizing the utility of the retrieved knowledge. Specifically, ALFAR tackles the challenges from two perspectives. First, it alleviates attention bias by adaptively shifting attention from visual tokens to relevant context tokens according to query-context relevance. Second, it decouples and weights parametric and contextual knowledge at output logits, mitigating conflicts between the two types of knowledge. As a plug-and-play method, ALFAR achieves superior performance across diverse datasets without requiring additional training or external tools. Extensive experiments over multiple MLLMs and benchmarks show that ALFAR consistently outperforms the state-of-the-art by large margins. Our code and data are available at https://github.com/Lackel/ALFAR.

Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention Reallocation
Wenbin 22
Jiahao N
Feng T:2
Haonan Lin Mingxiang Cai
Yaqiang Wu
Qianying Wang*1 Xiaoqin Zhang Shijian Lu
Xi' an Jiantong University
National Engineering Laboratory for Big Data Analytics
Nanyang Technological University *Lenovo Research *Zhejiang University of Technology
senbinan&stu.co
jiahao007@e att.enuuueee edu.
fengtian@mail www.:xwt..uuuu.....oooooo. edu mangqyallenove com, shijian. lu@ntu. edu.
Abstract
Despite their recent progress, Multimodal Large Language Models (MLLMs) often struggle in knowledge-intensive tasks due to the limited and outdated parametric knowledge acquired during training. Multimodal Retrieval Augmented Generation addresses this issue by retrieving contextual knowledge from external databases, thereby enhancing MLLMs with expanded knowledge sources. However, exist- ing MLLMs often fail to fully leverage the retrieved contextual knowledge for response generation. We examine representative MILMs and identify two major causes, namely, attention bias toward different tokens and knowledge conflicts between parametric and contextual knowledge. To this end, we design Adap- tive Logits Fusion and Attention Reallocation (ALFAR), training-frres and plug- and-play approach that improves MLLM responses by maximizing the utility of the retrieved knowledge. Specifically, ALFAR tackles the challenges from two perspectives. First, it alleviates attention bias by adaptively shifting attention from visual tokens relevant context tokens according query-context rele- vance. Second, it decouples and weights parametric and contextual knowledge output logits, mitigating conflicts between the two types of knowledge. As a plug-and-play method, ALFARR achieves superior performance across diverse datasets without requiring additional training or external tools. Extensive exper- iments over multiple MILMs and benchmarks show that ALFAR consistently outperforms the state-of-the-art by large margins. Our code and data are available at hhttps::/ www.
Introduction
Building upon powerful Large Language Models (LLMs) 12345675 S Multimodal Large Language Models (MLLMs) 9 13 14 15
16 17 have achieved impressive performance
over a wide range of vision-centric tasks such as image captioning [8]
19 20), visual question
answering 21 22 etc. Nevertheless, MLLMs often struggle to handle wewwledge-nttnsive vision-
language tasks [23
24 primarily due to the limited and outdated parametric knowledge acquired
during training
25 26 Multimodal Retrieval Augmented Generation (MRAG) [27 28 29],
prevalent approach that attempts to resolve this issue, retrieves contextual knowledge from external
Equual contribution *Coreepooning author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

data to empower MLLMs for accurate response generation. However, the way of exploiting the contextual knowledge remains ender-explored. undermining the effectiveness of MRAG.
We examined representative MILMS with MRAG and found that while MRAG can improve MILM performance when high-quality contextual knowledge is retrieved (as shown in Fig. D. MILMs often fail to make full use of the retrieved knowledge, even when ground-truth knowledge is available. We identify two primary causes, namely, attention bias among visual and context tokens and conflicts between MLLMs parametric knowledge and retrieved contextual knowledge. For the attention bias, MLLMs tend to allocate more attention to image tokens over context tokens, especially in shallow layers that are critical for knowledge extraction and exchange [30]. Since images often do not provide sufficient information for knowledge-intensse intensive questions [23 311, the attention bias hinders the effective utilization of contextual knowledge and leads to inaccurate MLLM responses. In addition, MLLMM allocate attention uniformly across context tokens without prioritization, which dilutes the contributions of query-relevant knowledge and tends to introduce inaccurate MILM responses.
Knowledge conflicts typically arise from the discrep- ancy between contextual and parametric knowledge. We observe that MLLMs tend to rely excessively on their parametric knowledge even when accurate contextual knowledge is present, leading under-utilization of con- textual knowledge and counterfactual responses. Such a phenomenon is well aligned with observations in pre- vious LLM studies 33 34 and findings in psychology research 35 36). both underscoring a clear preference toward intrinsic instead of retrieved knowledge. On the other end, the preference for the parametric knowl- edge does help when the contextual knowledge is unreli- able 228 37 38 This can be observed Fig. [ where low-quality contextual knowledge significantly degrades performance. Therefore, striking a balance between parametric and contextual knowledge while leveraging their complementary strengths critical for generating accurate responses.
75.00
71.12
70.00
LLaVA
Accuracy
60.00
65.00
65.47
LLaV + MRAG
ALFAR
55.00
45.10
41 41.53
30.00
Ground- Inuth
1st lettneved
2nd Ratteeved Knowledge
Srd leetnaved Knowlodge
Know edge
Knowledge Contextual Knowledge for Infecence
VQA accuracy of LLaVA- 1.5 [14 on the multi-choice InfoSeck dataset 32] with respect to the quality of contextual knowledge. ALFAR fully exploits the retrieved knowledge, consis- tently improving MRAG performance re- gardless of knowledge quality.
Figure
In this work, we propose Adaptive Logits Fusion and Attention Reallocation (ALFAR), a training- free and plug-and-play approach that enables effective utilization of MRAG-retrieved contextual knowledge for accurate MLLM responses. ALFAR addresses attention bias and knowledge conflicts by dynamically adjusting attention allocation and balancing the parametric and contextual knowledge, respectively. Specifically, ALFAR adaptively shifts attention from image tokens to relevant context tokens based on retrieval scores and query-context relevance, enabling MLLMs to focus on more pertinent information. In addition, ALFAR decouples parametric and contextual knowledge at output logits and weights them according to the attention distribution, enabling a balanced and synergistic integration of the two types of knowledge. Extensive experiments across multiple representative MLLMs and benchmarks demonstrate ALFAR's superior and broad applicability without involving additional training or external tools.
The contributions of this work can be summarized in three major aspects. First, we dive deeply into knowledge utilization in MLLMs, identifying attention bias and knowledge conflicts as two key factors that impede the effective utilization of the retrieved knowledge. These findings provide valuable insights for advancing knowledge utilization in MLLMs. Second, we design ALFAR, a training- free and plug-and sad--anddlyyy -lay approach that reallocates attention and balances parametric and contextual knowledge effectively. Third, Extensive experiments over multiple generative and discriminative benchmarks validate ALFAR's effectiveness and versatility, demonstrating its superior performance and broad applicability across various multimodal tasks.
Related Work
2.1 Multimodal Large Language Models
The rapid advancements in Large Language Models (LLMM)) 2 propelled the development of Multimodal Large Language Models (MLLMs)
4
5
7
39
have greatly
9
10
B
4
15
2

---

40]. To align visual and textual modalities, prior studies explore different approaches such as visual encoders with linear projectors [14 41 40), Q-former [8]] [O], and Perceivers 42 which transform
image patches into visual tokens that are compatible with L.Mss. Most MLLMS conduct training in two stages, namely, pre-training for feature alignment and instruction-based fine-tuning
43 10, Despite these
enabling impressive performance across diverse multimodal tasks [44 45
due to the 46
47
advancements, MLLMs often face challenges in knowledge-intensive tasks
23
limitations of their parametric knowledge acquired during training.
2.2 Multimodal Retrieval Augmented Generation
49], Multimodal Retrieval Augmented Generation (MRAG) has been widely explored for enhancing MILMS with MRAG retrieves relevant knowledge from a multimodal database and incorporates the retrieved knowledge as the context of the input. For instance, Wiki-LLa (27] broadens the knowledge scope of LLaVA [[4] by incorporating the retrieved Wikipedia articles in training. EchoSight [29] leverages a fine-tuned Q-Former [18] to filter retrieved knowledge and enhance retrieval recall. ReflectiVA 28 and MR AG [50] introduce trainable reflection mechanism to assess the necessity of retrieval and the relevance of retrieved knowledge. In the LLM domain, several training- free methods have been proposed to better utilize the retrieved knowledge to enhance generation quality. For instance, CAD 226 employs contrastive to increase the faithfulness of generation. Moreover, AdaCAD 52,, Entropy [53], and COIECD 54 extend CAD (26) by introducing JS divergence, entropy, and information constraints, respectively. Despite the improved faithfulness toward the retrieved context, these methods struggle to balance parametric and contextual knowledge (55], resulting in sub-optimal performance when the contextual knowledge noisy.
Inspired by the concept of Retrieval Augmented Generation (RAG) for LLMs 48
more comprehensive and up-to-date knowledge (27 28
29
50
decoding 51
3 Preliminary and Motivation
3.1 Multimodal Retrieval Augmented Generation
Given textual query q and query image I, an MLLM MM parameterized by 0 is expected to gener- ate a reliable answer y. To enrich MILLMs with external knowledge, MRAG employs a multimodal retriever RS to fetch relevant knowledge from a multimodal knowledge base = {[T,, ()) where T, and o; represent image and its corresponding textual knowledge, respectively. The retriever R. measures the similarity between the query pair (g. 7) and multimodal knowledge pair (I, c) based the cosine similarity between their image embeddings:
Ro(1). (T) [T)( (1),(
The textual knowledge e with the highest retrieval similarity a selected as the input context for MLLMs. Consequently, the output distributions of the MILM with MRAG at the time step are:
p(y+) ~ softmax(Ms (ye q, I, c, y<t))
where V<r represents the sequence of generated tokens before the time step t.
3.2 Self-attention Mechanism in MILMs
Image tokens
Context tokens
Attention Distribution
MLLMs generate responses ruto-regressively using Transformer blocks [56]. Specifically, the input image, query, and context tokens are concatenated and projected into three distinct vectors: the query vector Q, the key vector K, and the value vector V. through three linear layers, W,. WE. and W.. The self-attention mechanism computes the relevance of each token other tokens follows:
0.5
04
0.
Layer Index
Figure 2: Proportions of attention weights that are assigned to image and context tokens at different shallow layers of LLa [4]
A Q K
+ M
3
3

---

where A E R is the attention weight matrix, M is a causal mask, d is the feature dimension, and is the number of input tokens. The output O can then be calculated by:
O softmax(A)
The attention weight matrix A reflects the importance of different input tokens in generating new tokens. This property makes a valuable tool to analyze the contribution of different types of tokens to the generated responses.
3.3 Attention Bias in MRAG
To generate the response token Mn+1, MLLMs perform self-attention over all input tokens which correspond to the n-th row of the attention weight matrix A. We analyze the contributions of the input image and context using an importance score, defined as the total attention weights assigned to these tokens. For the input image, the importance score at layer is calculated by: 5* ( jer A Similarly, for the input context, the importance score at layer i is determined by: As illustrated in Fig. 2 MILLMs tend to allocate more attention to image tokens than context tokens, particularly in shallow layers that are pivotal for extracting and exchanging information from distinct tokens [30]. This issue affects the effective utilization of contextual knowledge since images often do not capture sufficient information for Knowledge-intensiv questions [23 31.. Moreover, MLLMs assign attention uniformly across different parts of the context without highlighting query- relevant segments. Such indiscriminate distribution increases the distraction of irrelevant knowledge, ultimately leading to inaccurate or misleading responses for MLLMs.
jEC Any
3.4 Knowledge Conflicts in MRAG
Table I: Experiments with VLa [4]. Conffict Ratio:
Ratio of discrepancy between parametric and contextual knowledge. Performance drop: Accuracy decline due to knowledge conflicts. Details of the two metrics pro- vided in Appendix A3
After knowledge retrieval, MLLMs in- tegrate the retrieved contextual knowl- edge with their internal parametric knowledge to generate responses. How- ever, similar to LLMs B3
34
52
Infoseek
ViQuAE
MLLMs often encounter knowledge conflicts due the discrepancy between the two types of knowledge, affecting the effectiveness of the model in various practical tasks. Worse still, MIL.Ms tend to prioritize their parametric knowledge even when perfect contextual knowledge is provided, leading to under-utilization of contextual knowledge and factually inconsistent answers. By assuming the accessibility of the ground-truth contextual knowledge, we evaluate the conflict rate the two types knowledge and the resultant performance degradation on the multi-choice InfoSeek 32 and ViQuAE 241 32) datasets with VL. [4]. As shown in
Conflict Ratio
60.85%
48.94%
Performance Drop
28.87%
27.02%
Tab. [ around half of the samples exhibit knowledge conflicts, which lead to an up to 30% perfor- mance drop, highlighting the necessity of mitigating such conflicts to enhance MLLM performance on snowledge-intensive tasks.
4 Method
The proposed framework consists of two branches for effective handling of parametric and contextual knowledge as illustrated in Fig. E Within the contextual branch, we design an attention reallocation mechanism that tackles the attention bias and improves the utilization of contextual knowledge by adaptively adjusting model attention toward relevant context tokens based on query-context relevance (Sec. I..) In addition, the network fuses the parametric and contextual knowledge adaptively in the output logits, mitigating knowledge conflicts under the guidance of the model attention that dynamically captures the relative importance the two types of knowledge (Sec.
4.1 Attention Reallocation
As analyzed in Sec. 3.3 the attention bias results from two major factors, namely, attention preference toward image tokens and uniform attention to context tokens. We address the attention preference
We average all attention heads and omit the notation for simplicity.

---

MRAG
Tokenization
Large Language
Decoding
Viston Encoder
Projector
Regular x
logils,
imagee
WA
iinage
polmme
Waiis
Aduthe Loits Fusko
quary tokomn
When wast the buildinng the imager constructed?
1849
Database
Tokonizer
Modified Attn. Bik. x
query
logits.
The Exfiet Tower Parts, which desigund Biffel completed constreated 1989.
contcai tnken
Wa
Ara.
Atheorion Sealocatton
context
Figure 4: The overview of Adaptive Logits Fusion and Attention Reallocation (ALFAR).
by adaptively adjusting model attention illustrated in Fig. based the retrieval similarity a in Eq. that reflects the reliability of the retrieved context. Specifically, less attention is allocated to image tokens if the context more reliable with a high retrieval similarity. The attention reallocation can be formulated as follows:
A (1 B) Ani, E S₁
where 3 k the scaled retrieval similarity with scaling factor Å is the modified attention weight matrix, is the number of all input tokens and S, is the index set of image tokens. Ani corresponds to the attention weight in the n-th row and i-th column of A.
In addition, we introduce a query-context relevance score to mitigate the uniform attention to all context tokens and allow MILMS focus more query-relevant context. We derive the relevance score from attention weights assigned to query tokens by j-th context token as follows:
Image attentonn weight
Query aniemio
Contest ancetion weight
Cansal mask
Decrosse attention
Increase atternion
weight
Scaled query relevance 8 Scaled renieval similarity
Logiss fusion weights
sum
reallocated weight
Ajk KE
Σ
WJ
Σ
Σ An" s.t. E S. A
(6)
Jof
Jul
Ferp
reallocation coefficient
elemezt-wiie muliipiicttinn
original wright
where Sy and Se index sets of query and context tokens, respectively. With the relevance scores, adaptively increase MLLMs' attention to context tokens:
Attention Weight Matrix
Attention Reallocation
Figure 3: Attention reallocation by adjusting the last row of the attention weight matrix (i.e., An).
Ang = (1+7)) Anj. s.t. j S. (7)
where 7/ = k- wj is the scaled query-conter relevance with a scaling factor k. After attention reallocation, we apply softmax to redistribute the attention in Eq, to compute the output hidden states. This repeats auto-regressively for each subsequent token prediction.
4.2 Adaptive Knowledge Fusion
As analyzed in Sec. 3.4, parametric knowledge could hinder the utilization of contextual knowledge, leading to inaccurate responses. However, parametric knowledge brings benefits when the retrieved context is unreliable. Therefore, striking a balance between parametric and contextual knowledge based on their reliability is essential for generating accurate and reliable responses for MILMs. Nevertheless, parametric knowledge implicitly embedded and the two types of knowledge are entangled during inference, making it hard to explicitly represent and utilize them separately. We address this issue by disentangling two types of knowledge and fuse them at output logits. Specifically, we represent parametric knowledge at step by using the output logits that have only query and image as inputs:
logit, Molyrlg.L,vel
(8)
5

---

Table VQA Accuracy comparison on generative freeffrm VQA datasets over three runs. Regular and Parametric denote that MLLMs generate answers with and without retrieved knowledge, respec tively. The best performance is marked in bold.
Human
Validation 23
Model
Decoding
Unseen
Unseen
Unseen Question
Unseen Entity
Overall
Overall
Question
Entity
Regular
14
7.59 (+0.08)
(.74() (2005))
29.98(20.02)
10.00)
19. 19.78 (4000)))
Parametric
6.27 (10.0))
6.26 (10.9)
6.26 (10.0))
7. 7.14(10.08)
(.28
6. 6.0000
CD 51
7.39 (10.21)
7.31 (11006)
T.35
20.02
19.90;
(1.10
AdaCAD
(4:0.02)
8.07 (+0.03)
21.23 (.940))
0000
20.91 (+0.02)
21.07 (+0.08)
L.a
Entropy 53
7.98[10.00)
8.34
21.97 (.100)
02)
21.91 21.85;10.01)
(10.00)
CAD 26
(+0.5))
8. (20.00))
8.08 (+t0.2)
21.68;=0.01 010)
21.30 20.02)
(2000)))
COIECD
8. (1..00)
(10.01))
22.43 (.71(10.01)
100005
21.73;10.05)
(2.00
AGLA 57
8. (40.28))
9. (
(.94(+0.08)
22.34 02)
21 21.88
22. (+0.02)
VCD 58
9.22 (10.00)
9. (.2000)
9.24(10.02)
10.00)
22.38;
11.003
22, (10.03)
ALEAR (ours)
12.80, (0.08)
(=0.00)
11.96;
=1.96|=0.02)
23.82,
=0.00)
23.75
(=0.5
23.78 (0.01))
Regular 10
(4:000)
3. (20.8)))
(.02()).01)
(282000000) (.60))
3.71 (+7.00)000)
Parametric 10
(0000)
3. (20.00))))
(.84(40.01)
(.30
1.92 (.92
2. (22.000
CD 51
4.52 4.52
3.56
3.98
3.59 (10000))
(.0000
(.79
AdaCAD
4.57 (4:0005)
(00100
(4090))008)
3.7 (+0.02)
(.35
4.01 4.01 (+0.01)
Entropy 53
1.000)
4.14(1
4.34(10.02)
(.0.01)
4.39 (10.00
4.0800
InstructBLIIP
CAD 26
4.52 4.52
3.55
(.98(4).01)
(+77
4.43 4.43 (20002)
(.08
COIECD
(10.00)
(10.02)
4.07 (10.0)
(.5.000.000)
(.30000000)
AGLA [57
4.8 (4000)
4.28
(+0.00)
4.52
3.74 (.74()
4. 4.10 (+0.01)
3.91 (=9.0)
VCD 58
4.70
4.14
(.40010.00)
3.62 (10.01)
4. 2.12
3.85
ALFAR (ours)
5.98 (0.00)
5.67 (+0.00)
5.82 (+0.00)
4.55
5.68, (0.00)
5.05 (.0.00
Regular 40
(1..00)
(20.00)
(.0022)002
(1.93
11.78(40.01)
(1.85
Parametric 40
5. 100
6. 6.10 (20.07)
(.92(20.06)
(.61
6.25
6. (.86))
CD 51
8.2 (10.00)
(10001)
T.64(110.01)
12.41
11.00
(2.14
AdaCAD
(+300
7.1 (+0.00)00
7.60(+0.00)
(2.87(=0003)
(0.03
(2.70(+0.02)
Entropy 53
(1000)
(1731)
8.01(10.05)
13.78( 020)
13.55 13.33
Shikra
CAD 26
8. (.0.00))
7. (21002))02)
7.02 (+t.000
12.99; 003
12. (2.51
(2.75
COIECD
8.32 (10.02)
7.73(10.08)
(10.00)003)
10.02)
14.21(10.03)
(4.03(10002)
AGLA 57
8 (.24
(2.05)
7.88 (20.01)
14.29
02)
13.91
14.08(+0.00)
VCD 58
8. 3.13
7.4 (10.03)
(.75(10004)
13.71 (10.03)
13.81 (1..03)
13.76
ALFAR (ours)
8.61 (10.01)
8.04 (10.01)
(.0.01).0))
(10.01)
15.25 (10000)
15.18/10.00 (..10
15.18
(1000))
Regular [6
(.0.o)
3. (20022)
12.69 (.56(21002)
12.38(10.02)
(2.53
Parametric 16
2.34
2. 2200
2.21
(20.01)
3.93
(229
CD 51
4.28
(.0.00)000)
14.49
14.43;40000)
(4.46(20.01)
AdaCAD
4.7 (2:00000
(203
(1000))000)
14.82; .2)
14.06;
(4.02)
Entropy 53
4. 1.807)
2.91
(16.00
14.66, 002)
14.66;10.01)
14.66
MiniGPT4
CAD 26
(20.0))
3.44
4.07 (20.00))
14.83;
14.94;+0.00;
(4..8
COIECD
4.57 (1:000)
(.40
3.90(10.00)
14.77 14.67;11.02) (4.87(10.01)
AGLA 57
4.67 (+0.02)
(..0))
4.09 (.09(20.01)
14.31;
.02)
13.92
14.11(40.00)
VCD SS
4.52
3.43
14.46; 3.90(
o013
14.30.
(20.0))
ALFAR (ours)
5.05 (10.02)
3.87 (10.00)
(.0.00
15.16
(5.27
15.05 15.05
Similarly, we represent contextual knowledge by using the output logits that have context as additional inputs and perform attention reallocation to better utilize the context:
logit. = Mo (g. I, T,c,
(9)
where Ma is the MLLM with attention reallocation. The reliability of the parametric and contextual knowledge can thus be measured by the attention weights that are assigned the image and context tokens capturing the correlation among tokens [59] 66 61]:
X
Σ A FESt
X
Σ Aij
(10)
JES.

---

Table 3: VQA Accuracy comparison on discriminative multi-choice VQA datasets over three runs.
Model
Decoding
TnfoSeek
ViQuAE
Model
Decoding
InfoSeek
ViQuAF
Regular
[4
51.97 (20.42))
53.32
Regular 110
23. 23.44(44.89)
19. 00.822 (400.1))
Parametric
14
(0.10
(1.06(20.110)
Parametric 10
(.73
6.53 20.as)
CD [51]
20.20
62.56((2n.03)
CD [51]
22.95 22.95
(1006)))
CAD 20
52.99 (2.08
CAD 26
26.56(40.58)
23. 18.18 (40.0
AdaCAD 52
(2.99( (0.00
AdaCAD 52
(7007(+0.05)
23.6 23.64 (40.08)
LLV
InstructBLIP
Entropy 53
=0.07)
(4.20(()
Entropy 53
23. 23.19 27.50(
(40.30))
COIECD
:2.08
(2.00(
COIECD 25.65(10.05)
20 (20.0)
VCD 58
53.87 (2:0007)
(5.13
VCD 58
(3.31(11.0r)
20.28(10055)
AGLA 57
53.53, (3.53
(20.00)
54.
54.24((10.21)
AGLA 57
(10.4(
16.77 (1.00
ALFAR (ours)
58.35 (10021)
(5.91(2)
ALFAR (ours)
(10.00)
24.11
Regular 40
19.41 (=0.12)))
(7.73
Regular 16
25.83
(4006)+)
Parametric
40
(+05
(00.0)
Parametric
(9.57)
(+1
CD 61
24.83
(1.33(+0005)
CD [ST
20. 20.46 (6.55(++.31)
(40.7))
CAD 20
24.51 (=0.06)
21 21068(+0.15)
CAD 26
23.39 27.58(+0-43)
(+0.07)
AdaCAD 52
23.95 18)
21.5 (=00.01)
AdaCAD 52
28.01 17)
23 23.05 (00001))
Shikra
MiniGPT-4
Entropy 5
24.12
21. (1099(
Entropy
28.84(+0:08)
22.56 (2.59(+0000)) (+0007)
COIECD
24.18 (4.18(10.11)
21. 21.68
COIECD S4
29.44(10.01)
25.9 (1094(10.18)
VCD 58
25.76
23. 23.11(40.72)
VCD 58
25. 25.45 28.74(10.11)
AGLA 55
(.0.29)
22. (20.2(
AGLA 55
29.28(10.87) (10.87)
(10.0
ALFAR (ours)
28.11 (10001)
(3042(+0:22))
ALFAR (ours)
30.88
32.28 (0.88(10.12) (10.12)
Regular 62
63.90
55.44(220.12)
Regular 63
53.00 18)
53. (328-0(20001)
Parametric 62
42.50 (20.12
(3.94(220.11)
Parametric 63
45.96(10.32)
53. (30.28
CD [11
51.43
(1.38
CD [51
56.23 25)
50 11,3 (10.1))
CAD 26
53.83;
52, (2.04)
CAD 26
58.67
(1..2
AdaCAD 52
55.60;10.34)
(410.,1
AdaCAD 52
(7.87())
60 (0.89
LLaVA-Next
Qwen2.5-VL.
Entropy 53
34..4((10.20)
55. 100.10
Entropy [53
(80.6((1.13))
(10.10))
COIECD 54
53.93
54.17
COIECD 54
50.23
60.56 (0.02)
VCD 5x
53.27 (20.27)
(+..20
VCD 58
58.56(+0.32)
60. (20.88)
AGL 57
54.03, 54.03 (0025))
54. (000.4))
AGL. 57
59.13 (+..2)
61.08 (+0.18)
AL.FAR (ours)
58.47 (+02)))
59.29, (=0.23)
AL.FAR (ours)
61.57
63.22 61.57 (+0.18)
Finally, based on the adaptive weights, the two types of knowledge are fused dynamically at each decoding step t:
p(ı)) ~ softmax (1+
X X
logite -(1
logitp
(11)
X
Experiment
5.1 Experimental Settings
Datasets. We conduct experiments over three types of knowledge-intensive datasets: (1) Free- form generative datasets including Human 23 high-quality info-seeking dataset curated and verified by experts, and INFOSEEK...e 23 which encompasses diverse entities from Wikidata. For INFOSEEK we adopt its Validation set for evaluations to aligned with prior studies
(2) Multi-choice discriminative datasets including Infoseek [23] 32 and ViQuAE (24 32 which are both multi-choice knowledge-intenss
datasets that are collected for assessing cross-modality knowledge conflicts as described in (22 (3) Knowledge-bass
datasets including OK-VQA (64],
AOK-VQA (65] and Encyclopedic VQA (E-VQA) [31], which are widely adopted for evaluations of tasks that require commonsense knowledge. Additional details about the datasets and knowledge
bases are listed in the Appendix A5
MILMM baselines and SOTA methods. We perform evaluations by using four representative MLLMS
as backbones: LLaVA-1.5 (7B and 13B) [14, InstructBLIP (7B and 13B) [0]. Shikra (7B) 40).
MiniGPT-4 (7B) [6]. LLaVA-Ness (7B) 62) and Qwen2.5-Vi (3B) 63 For benchmarking, we
select several SOTA training-fres free decoding methods that aim to mitigate knowledge conflicts in LLMs:
Contrastive Decoding (CD) [51), Adaptive Context-Aware Decoding (AdaCAD) (52, Entropy-based
decoding (Entropy) (33, Context-Aware Decoding (CAD) (6) and COntextual Information--nttonn
Constraint Decoding (COIECD) [54]. In addition, also benchmark with two representative
7

---

hallucination mitigation methods, including Visual Contrastive Decoding (VCD) 58 and Assembly of Global and Local Attention (AGLA) [57].
Implementation details. For knowledge retrieval, we employ the vision encoder of CLIP-ViT-L/1- 336 [66] the retriever and append the first retrieved knowledge to the prompt as context. The scaled factor k is set to 0.4 to avoid excessive adjustment. For the knowledge base, we use Wikipedia dumps provided by [23] and select items with associated images for retrieval. Multinomial sampling serves as the decoding strategy. We denote MLLM inference with retrieved knowledge as Regular and without retrieved knowledge as Parametric. We follow prior studies [26] 53 58 and adopt adaptive plausibility constraints [51] for fair comparisons. All experiments are conducted on four NVIDIA RTX 3090 GPUs. All compared methods are reproduced by us according their released codes or original papers.
5.2 Experimental Results
Experiments on free-form datasets. Tab. 2 shows experimental results of four representative
MLLMM 44 I0 40 [6] over two free-form generative knowledge-intenne datasets [23]. We can
see that the proposed ALFAR consistently outperforms the Regular decoding strategy by substantial margins (averaged around 2.5% in overall accuracy) across all MLLMs and datasets. Additionally, ALFAR surpasses tlate-of-the-an decoding methods as well, demonstrating its effectiveness in the better utilization of contextual knowledge.
Experiments on multi-choice datasets. Tab. presents experimental results of six MLLMS 14
40 16
over two multi-choice discriminative datasets (32 24). Notably, ALFAR achieves an average improvement of 6.6% over Regular decoding and consistently surpasses state-of-the-art decoding strategies by substantial margins, underscoring its effectiveness in diverse tasks. Moreover, we observe that LLaVA-1.5 [1] demonstrates stronger instruction-folloiingg capability compared to other models. enabling it to produce more correctly formatted outputs.
62 63]
Experiments on knowledge-based datasets. In addition knowledge-basee based datasets [32] [24],
Table 4: VQA Accuracy comparison on the knowledge-basse
entity
VQA datasets with LL. [14
over three runs.
Model
OK-VQA
AOK-VQA
E-VQA
conduct experiments on common- sense knowledge-bard -baeed datasets, OK-
Regular IE]
(6.17
44.13
19.14 14(42.2)))))
VQA (64), AOK-VQA (65) and En-
Parametric
14
(5.13
43.23 (..02
5.34(10.01)
cyclopedic VQA (E-VQA) 31 with LLaVA-1.5 [14]. As shown in Tab. 4
CD 51
(5.00
51.67 51.67 (10.44)
28, (2(.88)))
CAD 26
26.44
03.5)
28. 28.62(41.70)
ALFAR surpasses Regular decoding by 15.2% and consistently outper- forms state-of-the-art decoding strate- gies, underscoring its effectiveness in addressing a broader range of -oowledge-intensvve knowledge intensive tasks.
AdaCAD
52
(7.10
54.40(
28.33 (10.42)
Entropy 53
56.27
53.93 (3.93
29. 29.24 (10.1)
COIECD S4
56.43
53.93
28.24
VCD [58
(7.80
(7.40
27.71 (10.33)
AGLA S7
(0000))
55.40 10063)
28.29 (10.6))
ALFAR (ours)
60.83 (.001)
59.93 (10.02)
29.57 (10.08)
Discussion
6.1
Ablation study
We conduct ablation studies both multi-choice and commonsense knowledge-based datasets [32]
Table 5: Experimental results of ablation study with different model variants.
65 to assess the effectiveness of each design in the proposed ALFAR model with LLaVA-1. [14).
Variants
InfoSeck
AOK-VQA
As shown in Tab. 5 the Attention Reallocation enables MILMs to better utilize retrieved knowl- edge, thereby enhancing overall performance. The Logits Fusion mitigates knowledge conflicts, al- lowing MILMS to integrate retrieved knowledge and improve overall performance effectively. Moreover, applying Adaptive Weights during logits fusion helps MILMM better leverage both parametric and contextual knowledge while reducing the impact of noise from the retrieved context, further contributing performance gains.
Regular 14
51.97
44.13
+ Attention Reallocation
53.42
46.30
+ Logits Fusion
55.83
55.90
+ Adaptive Weights
58.35
59.93
8

---

6.2 Effect of different retrievers
Table Experimental results with different CLIP retrievers.
TT means using the imput query for knowledge retrieval.
We investigate the impact of differ- ent retrievers and retrieval strategies on recall and model performance on the InfoSeck dataset [32]. As shown in 6 our model consistently en- hances performance across all con-
Retriever
Recall
LLaVA
Ours
CLIP-B/16 66
38.27
45.93
51.60
CLIP-L/14
56.53
51.23
57.60
CIJP-I/14-336
666
58.37
51.97
58.35
figurations, demonstrating its robust- ness to retrieval noise. Additionally, retrieving knowledge based on the in- put query yields low recall due to its limited information about entities in the image. Despite this limitation, our model achieves consistent under low retrieval recall, owing its adaptive fusion strategy that balances parametric and contextual knowledge.
CLIP-B/I6-T 66
5.56
32.97
35,43
CLIP--L/1-T 666
6.00
33.40
36,23
CIIP-I/14-336-1
[66
5.73
33.37
36.63
performance improvements
6.3 Inference Efficiency
In this section, we conduct a detailed analysis the inference efficiency of the proposed model in comparison with representative baseline methods. Specifically, examine how the incorporation of parametric knowledge modeling affects the in- ference time under both discriminative and gener- ative tasks. Compared with the baseline MRAG, model introduces only modest computational overhead, which mainly arises from the additional operations required to encode and integrate para- metric knowledge. Nevertheless, this extra cost remains limited, as the parametric knowledge com- ponent without contextual input is relatively con- cise. Moreover, all compared methods except the baseline also require modeling of both contextual and parametric knowledge. Consequently, the overall inference cost of our model is comparable to that of these methods. We evaluate all models on both the discriminative multi-choice dataset Infoseek (32) and the free-form generative dataset Human [23] using single input sample. The results summarized in Tab. demonstrate that our method achieves similar inference times.
Table 7: Inference time of different methods over one sample with LLaVA-1.5 [14].
Variants
InfoSeek
Human
Regular
14
0.46s (1.00x)
0.50s (1.00x)
CD 51
0.62s (1.35x)
0.72s (1.44x)
CAD 26
0.62s (1.34x)
0.72s (1.44x)
AdaCAD [2
0.62s (1.35x)
0.73s (1 (1.45x)
Entropy 53
0.63s (1.37x)
0.73s (1.46x)
COIECD 54
0.62s (1.35x)
0.73s (1.45x)
VCD 58
0.63s (1.38x)
0.74s (1.47x)
AGLA 57
0.83s (1.81x)
1.02s (2.04x)
ALFAR (ours)
0.62s (1.35x)
0.73s (1.46x)
6.4 Qualitative Examples
Fig. 6 shows qualitative comparison of four decoding approaches. It can be observed that Vanilla LLaVA [4] without context produces false response due the lack of knowledge about the entity in the image. LLaVA with MRAG (27] incorporates the contextual knowledge but still produces the same incorrect response as Vanilla LLaVA [1), primarily due to knowledge conflicts that cause MI.LMs to favor their parametric knowledge. I.LaVA with CAD [26] can mitigate knowledge conflicts effectively by reducing the influence parametric knowledge, However, produces a false response well, mainly due attention bias with excessive focus images and uniform attention toward context. In contrast, ALFAR introduces attention reallocation and adaptive logits fusion, enabling MLLMs to prioritize query-relevant contextual knowledge and produce accurate responses.
6.5
ALFAR on MILMM Scalability
Table 8: Experiments with larger MLLMs.
Tab. presents experimental results of the 13B variants -L. [14]
Decoding
LLaVA-13B
InstructBLIPP-13B
and InstructBLIP [00 over the InfoS-
Parametric
44.40
8.93
eek dataset (32 Notably, ALFAR con-
Regular
55,83
18.70
sistently improves performance across both models, demonstrating supe-
ALFAR (ours)
59.63
27.63
rior scalability with respect to the model size. Interestingly, observe performance decline in

---

image
CONLEXT
Attention Distribution
Attention Distribttin
Attention Distribution ALFAR
Huis Teee Busck is thome park in Sonehoo Nagenaki Supan, which the Netherlumas disphayingg Wie-stied Danck buildings
qizesition
after Thui ten Bosch Hagie...
Whiich vountry does this
Aeshor
Nallum
park from??
Ground-nuth
contect X
LLaVA
Japan
Hulland
Mollond
Nathoriandz
Japan
Figure 6: Illustration of generated responses and corresponding attention distributions across different as the backbone model.
decoding methods, using LLaVA-1.5 [4]
InstructBLIP 13B compared to the 7B variant, which may be attributed to the model's increased reliance on parametric knowledge.
6.6 Effect of Different Decoding Strategies
LLaVA MRAG
LLaVA ALFAR
Beyond the multinomial sampling strategy dis- cussed in this paper, we further investigate the robustness of the proposed ALFAR framework under diverse decoding settings. To this end, we conduct experiments using LLaVA-1.5 [4]
VQA Aucuracy
58
on the multi-choice InfoSeek dataset [32] and evaluate six additional decoding strategies com- monly adopted in MLLM generation. These strategies include Top-P sampling 667 with p 0.7, Top-K sampling (68) with k 50, greedy decoding [69]. temperature sam-
Top
Temperatuee Potemp. Top KeTamp.
Greedy
Decoding Strategy
Figure 5: Experimental results with different decod- ing strategies for response generation.
pling [70] with 0.5, Top-P sampling with
temperature (p = 0.7, t = 0.5), and Top-K sampling with temperature (k = 50, t = 0.5). As illustrated in Fig. 5 ALFAR consistently enhances model performance across all decoding strategies. This consistent gain highlights the generalizability of ALFAR's learning principle and suggests that it effectively complements various decoding schemes. Moreover, the results indicate that ALFAR can serve as robust and plug-and and-pla enhancement to existing MLLMs, ensuring stable performance regardless of decoding configuration.
7 Conclusion
In this paper, we examine representative MLLMs and find that they often struggle to fully utilize retrieved knowledge for Knowledge-intensiee tasks. We attribute this limitation attention bias toward different tokens and knowledge conflicts between parametric and contextual knowledge. To address these challenges, we introduce Adaptive Logits Fusion and Attention Reallo- cation (ALFAR). training-free and plug-and-play approach that enhances MLLM performance by dynamically reallocating attention and harmonizing parametric and contextual knowledge. Specif- ically, ALFAR mitigates attention bias by adaptively shifting focus from visual tokens to context tokens based on query-context relevance. Furthermore, it decouples and balances parametric and contextual knowledge at the output logits, effectively resolving conflicts. Experiments across multi- ple MLLMs and benchmarks show that ALFAR consistently surpasses state-of-the-art methods by substantial margins without requiring additional training or external tools, highlighting its versatility and effectiveness for various knowledge-intensive tasks.
key factors:
Acknowledgments
This work was supported by the National Science and Technology Major Project 00022D01171022 National Natural Science Foundation of China (62177038, 62293551, 62277042, 62377038), Project of China Knowledge Centre for Engineering Science and Technology, "LENOVO-XITI" Intelli- gent Industry Joint Laboratory Project, The Youth Al Talents Fund of the Chinese Association of Automation under Major Program DIBRC-JKYZD-22243111
10

---

References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[2] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. U12: Unifying language learning paradigms. In The Eleventh International Conference Learning Representations, 2022.
[3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233: 2022.
[6] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
[7] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[8] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint 1rXx:22309.16609, 2023.
[9] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 20122288.22900 12966, 2023.
[10] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong. Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpos vision-language models with instruction tuning. arXiv preprint arXiv:2306.043:: 2023.
[11] Tao Gong, Chengqi I Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: vision and language model for dialogue with humans. arXiv preprint arXiv:2305.047900 2023.
[12] Wenbin An, Jiahao Nie, Yaqiang Wu, Feng Tian, Shijian Lu, and Qinghua Zheng. Empowering multimodal I1ms with external tools: A comprehensive survey. arXiv preprint arXiv:2508. 10955, 2025.
[13] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03376, 2023.
[14] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint a884v:2304.088455 2023.
[15] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi. Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang. and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14/78, 2023.
[16] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models, arXiv preprint arXiv:2304.100 10592, 2023.
11

---

[17] Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. In Findings of the Association for Computational Linguistics ACI. 2024, pages 12401-12430, 2024.
[18] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B2ip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 2023.
[19] Ahmad Sami AI- Al-Shamayleh, Omar Adwan, Mohammad A Alsharaiah, Abdelrahman H Hussein, Qasem M Kharma, and Christopher Ifeanyi Eke. A comprchensive literature review on image captioning methods and metrics based deep learning technique. Multimedia Tools and Applications, (3(12):3211934222 2024.
[20] Shilin Sun, Wenbin An, Feng Tian, Fang Nan, Qidong Liu, Jun Liu, Nazaraf Shah, and Ping Chen. A review of multimodal explainable artificial intelligence: Past, present and future. arXiv preprint arXiv:2412. 14056, 2024.
[21] Wenbin An, Feng Tian, Jiahao Nie, Wenkai Shi, Haonan Lin, Yan Chen, Qian Wang, Yaqiang Wu, Guang Dai, and Ping Chen. Knowledge acquisition disentanglement for knowledge- based visual question answering with large language models. arXiv preprint arXiv:2407. 15346, 2024.
[22] Jusung Lee, Sungguk Cha, Younghyun Lee, and Cheoljong Yang. Visual question answering instruction: Unlocking multimodal large language model to domain-specific visual multitasks. arXiv preprint arXiv:2402.0888 003360 2024.
[23] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre-trained vision and language models answer visual information-seckinn questions? In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 19948814968, 2023.
[24] Paul Lemerr Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-basee visual question an- swering about named entities. In Proceedings the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3108-3120, 31200, 2022.
[25] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929-3938. PMLR, 2020.
(26) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 783-791, 2024.
[27] Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical strieval--uugmener generation for multimodal llms. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1818-1826, 2024.
[28] Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Augmenting multimodal lims with self-reflective tokens for knowledge- based visual question answering. arXiv preprint arXiv:2411.16863 2024.
[29] Yibin Yan and Weidi Xie. Echosight: Advancing visual-language models with wiki knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1538-1551, 2024.
[30] Amit Artzy and Roy Schwartz. Attend first, consolidate later: On the importance of attention in different IIm layers. In Proceedings of the 7th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 177-184, 2024.
12

---

[31] Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3113-3124, 2023.
[32] Tinghui Zhu, Qin Liu, Fei Wang. Zhengzhong Tu, and Muhao Chen. Unraveling cross-modality knowledge conflicts in large vision-language models. arXiv preprint arXiv:2410.03659, 2024.
[33] Mehrdad Farahani and Richard Johansson. Deciphering the interplay of parametric and non- parametric memory in cerreval-augmentt language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16966-16977, 2024.
[34] Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan Ö Ank. Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. arXiv preprint arXiv:2410.07176, 2024.
[35] Justin Kruger and David Dunning. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6): 121, 1999.
[36] Aniket Kumar Singh, Suman Devkota, Bishal Lamichhane, Uttam Dhakal, and Chandra Dhakal. The confidence-competence gap in large language models: A cognitive study, arXiv preprint arXiv:2309.16145, 2023.
[37] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byme. Preflmr: Scaling up fine-grad fine-grained iute-interaction multi-modal retrievers. arXiv preprint NiXi::2002.08322,, 2024.
[38] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387-404 Springer, 2025.
[39] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workens for text-annotation tasks. arXiv preprint 2.0.v:::333.5050 2023.
[40] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023.
[41] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[42] Hugo Laurençon, Léo Tronchon, Matthieu Cord. and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.0222 2024.
[43] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[44] Xiaofeng Zhang, Yihao Quan, Chaochen Gu, Chen Shen, Xiaosong Yuan, Shaotian Yan, Hao Cheng, Kaijie Wu, and Jieping Ye. Seeing clearly by layer two: Enhancing attention heads to alleviate hallucination in Ivlms. arXiv preprint arXiv:2411.099% 2024.
[45] Haonan Lin, Mengmeng Wang. Yan Chen, Wenbin An, Yuzhe Yao, Guang Dai, Qianying Wang. Yong Liu, and Jingdong Wang. Dreamsalon: A staged diffusion framework for preserving identity-context in editable face generation. arXiv preprint arXiv:2403.1923: 2024.
[46] Xiaofeng Zhang, Chen Shen, Xiaosong Yuan, Shaotian Yan, Liang Xie, Wenxiao Wang. Chaochen Gu, Hao Tang, and Jieping Ye. From redundancy to relevance: Enhancing plainability in multimodal large language models. arXiv e-prints, pages arXiv-2406, 2024.
[47] Jiahao Nie, Gongjie Zhang, Wenbin An, Yap-Peng Tan, Alex C Kot, and Shijian Lu. Mmrel: A relation understanding dataset and benchmark in the milm era. arXiv preprint 1r111:2406..1121 2024.
13

---

[48] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval -uggeeeted generation for large language models: A survey, arXiv preprint arXiv:2312. 10997, 2023.
[49] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-ungmented generation for knowledge-intensivee nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
[50] Tao Zhang, Ziqi Zhang, Zongyang Ma, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Yuxuan Zhao, Zehua Xie, et al. mra Multimodal satrrvvvlleelecttto-aaagmmme generation for srowledge-basee vqa. arXiv preprint arXiv:241 15041, 2024.
[51] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12286-12318 2023.
[52] Han Wang, Archiki Prasad, Elias Stengel-Este and Mohit Bansal. Adacad: Adaptively decoding to balance conflicts between contextual and parametric knowledge. arXiv preprint WxXv::449.003394 2024.
[53] Zexuan Qiu, Zijing Ou, Bin Wu, Jingjing Li. Aiwei Liu, and Irwin King. Entropy-based decoding for -ugmenel large language models. arXiv preprint arXiv:2406. 17519, 2024.
[54] Xiaowei Yuan, Zhao Yang, Yequan Wang, Shengping Liu, Jun Zhao, and Kang Liu. Discerning and resolving knowledge conflicts through adaptive decoding with contextual information- entropy constraint. arXiv preprint arXiv:2402. 1893, 2024.
[55] Yanwen Huang, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, and Jing Xiao. Dynamic attention-guided context decoding for mitigating context faithfulness hallucinations in large language models. arXiv preprint 2XXv:2500..0055 2025.
[56] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[57] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406. 12718, 2024.
(58) Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision- -anguug models through visual contrastive decoding. arXiv preprint 1xXXI:23111169222 2023.
[59] Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. Accelerating multimodel large language models by searching optimal vision token reduction. arXiv preprint arXiv:2412.00556. 2024.
[60] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782-791, 2021.
[61] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprint 20V.:2404114449 2024.
[62] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2023.
[63] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
14

---

[64] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[65] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottggi.. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146-162. Springer, 2022.
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763 8763..87...66 PMLR, 2021.
[67] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 09751,, 2019.
[68] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv: 1805. 044333 2018.
[69] Ronald A Vore and Vladimir N Temlyakov. Some remarks greedy algorithms, Advances in computational Mathematics, 5(1):173-187, 1996.
[70] David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 91):147-169, 1985.
[71] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rX1v:2303.08774. 2023.
[72] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 1122773-227444 2022.
[73] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 8769-8778, 2018.
[74] Feilong Tang, Zile Huang, Chengzhi Liu, Qiang Sun, Harry Yang, and Ser-Nam Lim. Intervening anchor token: Decoding strategy in alleviating hallucinations for MILMs. In The Thirteenth International Conference on Learning Representations, 2025.
[75] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. URL https://arxiv. org/abs/2300.03117,, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately pinpoint the contributions and scope of this paper in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the Section Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the information needed to reproduce the main experiments for reproducibility.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We will provide open access to the data and code to reproduce the main experiments in our paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp) cc/
publ ic tttp://iieeeeeeeeeoeeee /teeeeeeeeeeeemmmmmommmm ss Loonolics cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ ctc.ps.ccppublicoo
CooeeoodeSSSaa Submisbbbsss ss cessoooPllcyy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and testing details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the mean and variance of results over three runs for the main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources needed to repro- duce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts in Section Broader Impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper thcode com dataaetttett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:///eeri..e..eeee http://rrr...cc//offfeeeeeeeeeee http://cccccccoeeeeeeee////////e 2025/ LLM) for what should should not described.
2025/
22

---

A1 Limitation and Future Work
Despite its strong performance in enhancing knowledge utilization for MILMS through adaptive logits fusion and attention reallocation, our approach has certain limitations. First, our framework requires access to MLLM parameters, making it inapplicable to black-box API-based models such as GPT-4 77. Extending our framework to black-box MLLMs represents a promising direction for future research. Additionally, we observe that MLLMs struggle effectively extract relevant information from long contexts. Addressing this limitation by improving MLLMs' ability to leverage extended contexts will be another focus of future work.
A2 Broader Impacts
The proposed model for enhancing knowledge utilization in MLLMs carries significant broader impacts. First, by addressing the critical issue of MRAG, our method enhances the reliability and trustworthiness of MLLMs. This improvement is essential for deploying these models in sensitive and high-stakes applications such as autonomous driving, medical diagnostics, and surveillance systems. Second, the insights and methods introduced in this paper contribute to the broader field of MLLMs, particularly in understanding and improving the knowledge utilization mechanisms within MILMMs This advancement can spur further research and innovation in integrating visual and textual data, leading to more robust and versatile Al models.
A3 Conflict Rate and Performance Drop
To quantify the conflict between parametric and contextual knowledge and its impact on model performance, we introduce two metries: Conflict Rate and Performance Drop.
Conflict Rate measures the proportion of instances where parametric and contextual knowledge provide different information, and Performance Drop quantifies the decline in model performance due to knowledge conflict. Since parametric knowledge is implicitly embedded in model parameters and is not directly observable, approximate its correctness by evaluating the model's outputs. Specifically, if the model (without external context) produces the correct answer, we assume its parametric knowledge is correct; otherwise, is considered incorrect. Given access to ground-truth contextual knowledge, the Conffict Rate can be defined as the error rate of parametric knowledge, i.c. the proportion of incorrect responses generated by the vanilla model without input context:
Conflict Rate Err(Mo g T), 9)
(A1)
where Err is a function that calculates the error rate of the output, only images and questions inputs, and is the ground-truth answer.
1) is the output with
When correct contextual knowledge is available, the ideal model should achieve 100% accuracy in the absence of knowledge conflicts. However, influenced by knowledge conflicts, the model cannot achieve 100% accuracy, then we can define Performance Drop as the error rate of outputs when both parametric and contextual knowledge are used:
Performance Drop Err T,
(A2)
where Me(vl,, is the output with ground-truth context additional inputs.
A4 Retrieval Recall
To investigate the retrieval recall from different retrieval rankings, we present the recall with Ground- Truth knowledge and knowledge from various retrieval rankings the multi-choice InfoSeck dataset [23 32 in Tab. [T The low recall negatively impacts performance on knowledge-intensing VQA tasks, highlighting the necessity of developing more effective retriever.
A5 Dataset
We present statistics of different datasets and the corresponding knowledge bases in Tab. A2 Specifi- cally, for Validation (23 and InfoSeck (32), we follow previous works [27] and adopt knowledge

---

Table Al: Retrieval recall with Ground-Trun knowledge (GT) and knowledge from different retrieval rankings on the multi-choics InfoSeck dataset [23 32).
Index
GT
2
3
4
Recall 100
58.37
10.57
5.07
3.07
Table A2: Statistics of the datasets and details of the knowledge bases used.
Dataset
# VQA pairs
Knowledge Base
Validation
73,620
Wikipedia 23
Human 23
8,931
Wikipedia 23
InfoSeek 32
3,000
Wikipedia 23
ViQuAE 32
3,000
Wikipedia 23
OK-VQA 64
5,046
GPT-3.5
21
AOK-VQA 66
1,145
GPT-3.5
21
E-VQA 31
700
Encyclopedia 33
base containing 1.7K entities derived from the original Wikipedia knowledge base 23 For Hu-
man (23] and ViQuAE (24] we use the original Wikipedia knowledge base 233, selecting 73.6K
entities accompanied by images for knowledge retrieval. For OK-VQA (64) and AOK-VQA [65],
we utilize the knowledge base provided by (21), which was generated using GPT-3.5 772. For
E-VQA [II, we select templated questions with images from the iNaturalist dataset 73 and use the
corresponding ground-truth knowledge for inference. All evaluations are conducted using the official scripts.
A6 Adaptive Plausibility Constraints
We follow prior studies 26
and adopt adaptive plausibility constraints [5]] for fair com- parisons. Specifically, calibrating the entire output distribution may penalize valid outputs from the original distribution and promote implausible outputs from the modified distribution. To mitigate this issue, we selectively consider tokens with high original probabilities and truncate other tokens as follows:
53 58
Vioken (yes) = (y6 E V : Po (y:) 2 3 max po (w)
(A3)
( = 0, if ys o Vikke (y<s)
where Vinken is the of selected tokens and V is the output vocabulary. We select / 0.7 to retain only high-probabilityy tokens.
A7 Effect of Intervention Layers
We investigate the impact of attention reallocation different MLLM layers on the InfoSeck dataset [23] using LLaVA-1.5 (14), as summarized in Tab. The results show that reallocating attention in shallow layers (layers 1-16) enhances model performance by mitigating attention bias toward image tokens, thereby improving the extraction of low-level features [3) In contrast, applying attention reallocation in middle layers (layers 17-24) yields smaller gains, as these layers primarily handle multimodal alignment and feature aggregation (74), where attention bias is less severe. Notably, reallocating attention late layers (layers 25-32) leads these layers are responsible for reasoning and directly affect output generation [74). Furthermore, leveraging attention reallocation in both shallow (layers 1-8) and deep (layer 32) layers yields the best performance.
most substantial performance gains, as

---

Table A4: An ablation study with different layers for attention reallocation.
Intervention Layers
VQA Accuracy
None
56.70
[1 8]
58.08
[9 16
58.17
[17 24
57.57
[25 32
58.10
8] U [32]
58.67
A8 Effect of different numbers of knowledge
We examine the effect of varying the amount of knowledge provided to MILMs on retrieval
Table A3: Experimental results with different num-
bers of knowledge using LLaVA-115 [14]
recall and model performance on the InfoSeck dataset [23]. As shown in Tab. A3 appending additional knowledge to the prompt improves retrieval recall but has limited impact model performance, as MLLMs often struggle to ef- fectively utilize information from lengthy input contexts (75). Our model addresses this limita- tion by guiding MLLMs based query-context relevance. However, the modest performance gains underscore the need for future research on enhancing MILMMs' ability process and leverage extended contexts.
#Knowledge
Recall
LLaVA [1]
Ours
58.37
51.97
58.35
2
49.90
58.70
3
74.00
50.13
58.57
4
76.77
50.23
58.20


Reviews:
[{"review_id": "uTOZ4lvhsp", "reviewer": "Reviewer_NFNU", "summary": "This work introduces a training-free, plug-and-play framework that tackles attention bias and knowledge conflicts in multimodal large language models through adaptive attention reallocation and logits fusion, yielding consistent performance gains of 6–15% across multiple VQA benchmarks and backbone models without additional training.", "questions": "My main concerns are in the method section, please see the weakness above.", "limitations": "The limitation discussions are sound and clear to me.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The method is simple and effective with training-free and plug-and-play manners, requiring no additional model training or external tools for integration.\n\n2. The preliminary findings on attention bias and knowledge conflicts, which hinder the effective utilization of retrieved context, are convincing.\n\n3. Extensive evaluation across multiple MLLMs and diverse VQA benchmarks demonstrates consistent performance gains (6–15% improvements). Ablation studies and tests with different retrievers confirm the effectiveness and robustness of each component and the overall approach.\n\nWeaknesses:\n\n1. The proposed method appears incremental, as the techniques involving attention weighting and logits fusion are already well-known and mainly borrowed from previous works.\n\n2. A comparative inference cost between the proposed method and the existing approach (MRAG) is not provided.", "was_revised": true, "final_justification": "Strengths:\n\n1. The method is simple and effective with training-free and plug-and-play manners, requiring no additional model training or external tools for integration.\n\n2. The preliminary findings on attention bias and knowledge conflicts, which hinder the effective utilization of retrieved context, are convincing.\n\n3. Extensive evaluation across multiple MLLMs and diverse VQA benchmarks demonstrates consistent performance gains (6–15% improvements). Ablation studies and tests with different retrievers confirm the effectiveness and robustness of each component and the overall approach.\n\nAfter rebuttal: I was very impressed and satisfied with how authors addressed my concerns regarding experiments on inference costs and comparison with other existing approaches. Thus I decided to raise my score.", "rebuttal": "Thanks for the constructive comments and suggestions. Below please find our clarification regarding your concerns.\n\n---\n\n**[W1]** Attention weighting and logits fusion are borrowed from previous works.\n\n**[R1]** We acknowledge that attention weighting and logits fusion have been explored in prior studies, represented by [1] and [2]. However, our approach differs significantly in motivation, method design, and empirical performance, as outlined below:\n\n**1. Motivation:**\nPrior works [1][2] focus on the multimodal hallucination task, [1] by enhancing visual attention, and [2] by subtracting visual logits to counteract language priors. In contrast, our motivation is orthogonal: we aim to reduce image interference via attention reallocation and fuse parametric and retrieved knowledge via logits addition to enhance knowledge utilization. This distinct objective is supported by thorough analysis and empirical validation, indicating that our method is not a simple reuse of existing techniques.\n\n**2. Method Design:**\nUnlike [1][2], which apply fixed factors for attention reweighting or logits subtraction, our method employs retrieval similarity and question-context similarity to dynamically adjust attention scores for image and context tokens. Additionally, we leverage these attention scores for adaptive knowledge fusion, enabling the model to selectively utilize different knowledge sources. This design is particularly beneficial when retrieved knowledge is noisy or partially incorrect.\n\n**3. Performance:**\nTo further demonstrate the uniqueness and effectiveness of our approach, we conducted ablation studies replacing our components with those from [1][2]. As shown in the table below, these substitutions consistently led to performance drops, validating the advantage of our design over prior methods.\n\n\n| Model                      | InfoSeek | ViQuAE |\n|---------------------------|----------|--------|\n| Ours                      | 58.35    | 55.91  |\n| Ours replaced with [1]    | 54.13    | 53.25  |\n| Ours replaced with [2]    | 55.24    | 54.78  |\n\nWe appreciate the reviewer’s comment and will include a more detailed discussion and comparison with related work in the revised version.\n\n---\n\n**[W2]** Inference cost.\n\n**[R2]** We appreciate the reviewer’s concern regarding the inference cost. Compared with the baseline MRAG, our model introduces a modest overhead for parametric knowledge modelling. However, **the extra cost remains acceptable as the input for modeling parametric knowledge without context is relatively short**. Besides, **all compared methods except the baseline need to model both contextual and parametric knowledge.** The inference cost of our method is comparable to that of the compared methods in a broader context. We evaluate all compared methods over one sample on both discriminative and generative tasks. The tabulated results below confirm that our method takes comparable inference time. In fact, our method can achieve competitive inference efficiency as the baseline MRAG by processing the two knowledge sources in parallel. We will elaborate on this issue with the new experiments in the revised manuscript.\n\n| Model   | InfoSeek          | Human            |\n|---------|-------------------|------------------|\n| MRAG    | 0.46s (1.00x)     | 0.50s (1.00x)    |\n| CD      | 0.62s (1.35x)     | 0.72s (1.44x)    |\n| CAD     | 0.62s (1.34x)     | 0.72s (1.44x)    |\n| AdaCAD  | 0.62s (1.35x)     | 0.73s (1.45x)    |\n| Entropy | 0.63s (1.37x)     | 0.73s (1.46x)    |\n| COIECD  | 0.62s (1.35x)     | 0.73s (1.45x)    |\n| VCD     | 0.63s (1.38x)     | 0.74s (1.47x)    |\n| AGLA    | 0.83s (1.81x)     | 1.02s (2.04x)    |\n| Ours    | 0.62s (1.35x)     | 0.73s (1.46x)    |\n\n---\n\nReferences\n\n[1] Paying More Attention to Image: A Training-Free  Method for Alleviating Hallucination in LVLMs.\n\n[2] Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding."}, {"review_id": "svLbQp7Dd3", "reviewer": "Reviewer_4mPJ", "summary": "This paper addresses a key limitation of multimodal large language models (MLLMs): their inability to effectively leverage retrieved external knowledge. To overcome this, the authors propose ALFAR, a training-free and plug-and-play method designed to mitigate attention bias and resolve knowledge conflicts in MLLMs when tackling knowledge-intensive tasks. Through its two core components—Attention Reallocation and Adaptive Logits Fusion—ALFAR significantly enhances knowledge utilization performance. Experiments conducted on four MLLMs across six knowledge-centric datasets demonstrate that ALFAR consistently and substantially outperforms prior state-of-the-art decoding methods.", "questions": "1. The analysis of different intervention layers is interesting but might be better placed in the main paper rather than as supplementary material.\n2. The definitions of certain variables (e.g., β, γ, λ) should be clarified and explained more thoroughly.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. The authors accurately identify two critical issues that limit knowledge utilization in MLLMs, attention bias and knowledge conflicts, and provide detailed analyses and validation of their causes and impacts.\n2. The proposed two modules, attention reallocation and adaptive logits fusion, offer a novel and effective perspective to enhance MLLM performance in knowledge-intensive tasks. \n3. Experiments across multiple representative MLLMs and diverse datasets validate the effectiveness of the proposed framework.\n4. As a training-free and plug-and-play module, ALFAR can be directly applied to many MLLMs in MRAG pipelines.\n\n# Weakness\n1. The evaluated LVLMs are somewhat outdated; the framework’s effectiveness should be validated on newer LVLMs (e.g., Qwen2.5-vl, LLaVA-Next).\n2. Additional analysis and validation of the framework’s robustness to varying retrieval quality would be beneficial.\n3. The details of the compared methods are not clearly described.", "was_revised": true, "final_justification": "The author's response effectively addressed my concerns, and I believe this article is worth accepting. Therefore, I decided to improve my rating.", "rebuttal": "Thank you for the constructive comments and suggestions. Below please find our clarification regarding your concerns.\n\n---\n\n**[W1]** Validation on newer LVLMs.\n\n**[R1]** We appreciate the reviewer’s suggestion. In response, we have conducted **additional experiments on two newer LVLMs, LLaVA-Next and Qwen2.5-VL, using the Infoseek dataset. The tabulated results below show that our method improves with both models consistently, validating its effectiveness and robustness on more recent architectures.\n\n| Method   | LLaVA-Next | Qwen2.5-VL |\n|----------|------------|-------------|\n| Parameter | 45.96     | 42.50       |\n| Regular   | 53.00     | 53.90       |\n| CD        | 56.23     | 51.43       |\n| CAD       | 58.67     | 53.83       |\n| ADCAD     | 57.87     | 55.60       |\n| Entropy   | 58.67     | 54.43       |\n| COIECD    | 59.23     | 53.93       |\n| VCD       | 58.56     | 53.27       |\n| AGLA      | 59.13     | 54.03       |\n| Ours      | 61.57     | 58.47       |\n\n---\n\n**[W2]** Framework’s robustness.\n\n**[R2]** We appreciate the reviewer’s suggestion. We would share that the robustness of our framework has been validated from four perspectives:\n\n1. **Robustness to knowledge quality**: As shown in Figure 1 of the submitted paper, our model achieves consistent improvements when the quality of retrieved knowledge varies. This indicates that the model can adaptively determine which knowledge to rely on, reflecting its robustness.\n\n2. **Robustness across MLLMs**: As shown in Tables 2, 3, and 4, our model integrates effectively with four different MLLMs, demonstrating its compatibility and generalizability across diverse architectures.\n\n3. **Robustness to retrievers**: As shown in Table 6, our model consistently improves MRAG performance across different retrievers, highlighting its tolerance to retrieval noise.\n\n4. **Robustness to model scale**: As shown in Table 7, our method yields consistent gains on 13B-scale MLLMs, confirming its scalability with respect to model size.\n\n---\n\n**[W3]** Details of the compared methods.\n\n**[R3]** We appreciate the reviewer’s suggestion. For all compared methods, we follow the implementation and model settings as described in their original papers to ensure a fair comparison. We will include more detailed descriptions of these methods and their configurations in the revised version for better clarity and reproducibility.\n\n---\n\n**[Q1]** Analysis of different intervention layers might be better placed in the main paper.\n\n**[A1]** Thank you for the helpful suggestion. We agree that the analysis of different intervention layers is an important component of our study. Accordingly, we will revise the paper to include this experiment in the main paper as suggested, ensuring that readers can better appreciate its contribution.\n\n---\n\n**[Q2]** Definitions of certain variables.\n\n**[A2]** We apologize for the confusion. The definitions of the variables are as follows:\n\n1. As described in Line 176, $\\beta$ is a factor that controls the strength of adjustment for image tokens.\n\n2. As described in Line 190, $\\gamma$ controls the strength of adjustment for each context token.\n\n3. As defined in Equation 10, $\\lambda$ represents the summed attention weights over image and context tokens, used for weighted fusion.\n\nWe ensure that all these definitions will be stated clearly and prominently in the revised paper to improve readability and understanding."}, {"review_id": "5agpTyPlES", "reviewer": "Reviewer_aGW7", "summary": "This paper addresses the limitations of Multimodal Large Language Models (MLLMs) in knowledge-intensive tasks, especially their inability to fully utilize retrieved contextual knowledge. The authors propose ALFAR, a training-free and plug-and-play method that mitigates attention bias and resolves conflicts between parametric and retrieved knowledge. Experiments show that ALFAR significantly improves LVLM performance across various datasets, surpassing existing state-of-the-art approaches.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. The paper offers novel insights by identifying attention bias and knowledge conflicts as critical obstacles to effective knowledge utilization in Multimodal Large Language Models (MLLMs).\n\n2. It proposes ALFAR, a training-free and plug-and-play method that effectively reallocates attention and balances parametric and contextual knowledge.\n\n3. Extensive experiments demonstrate ALFAR’s superior performance and wide applicability across diverse generative and discriminative multimodal tasks.\n\nCons:\n\n1. While your method has shown strong performance on a range of VQA datasets, it would further strengthen the contribution to demonstrate its applicability across a broader spectrum of vision-language tasks. In particular, evaluating on image captioning tasks could highlight the versatility of your approach in generating context-aware and knowledge-grounded descriptions.\n\n2. It would be helpful to cite a recent survey on Multimodal Large Language Models to provide readers with a comprehensive background and better contextualize your contribution within the broader MLLM research landscape [1].\n\n[1] \"MM-LLMs: Recent Advances in MultiModal Large Language Models.\" Findings of the Association for Computational Linguistics ACL 2024. 2024.", "was_revised": true, "final_justification": "Author has conducted additional experiments to validate applicability across a broader spectrum of vision-language tasks.", "rebuttal": "Thank you for the constructive comments and suggestions. Below please find our clarification regarding your concerns.\n\n---\n\n**[W1]** Applicability across a broader spectrum of vision-language tasks.\n\n**[R1]** We appreciate the reviewer’s valuable suggestion. As suggested, we conducted **additional experiments on two vision-language tasks using LLaVA-1.5 and InstructBLIP: image captioning and multimodal hallucination mitigation**.\n\nFor image captioning, we selected 500 images from the COCO dataset and evaluated caption reliability using the CHAIR metric (lower is better) and caption detailedness using Recall (higher is better). For multimodal hallucination mitigation, we conducted experiments on the POPE dataset, with accuracy and F1 score measuring the model’s ability to reduce hallucinations.\n\n**Evaluation on image captioning**\n| Model              | $CHAIR_i$ ↓ | $CHAIR_s$ ↓ | Recall ↑ |\n|--------------------|---------------------|----------------------|----------|\n| LLaVA              | 51.0                | 15.2                 | 75.2     |\n| LLaVA + Ours       | 48.3                | 14.5                 | 78.3     |\n| InstructBLIP       | 54.0                | 18.1                 | 71.1     |\n| InstructBLIP + Ours| 50.2                | 13.7                 | 72.6     |\n\n**Evaluation on multimodal hallucination mitigation**\n| Model               | Accuracy | F1   |\n|---------------------|----------|------|\n| LLaVA               | 83.5     | 82.3 |\n| LLaVA + Ours        | 85.8     | 85.5 |\n| InstructBLIP        | 80.4     | 80.9 |\n| InstructBLIP + Ours | 83.2     | 83.1 |\n\nAs shown in the tables, **our approach consistently improves performance across all metrics and tasks** by effectively retrieving similar images and relevant descriptions. These new experiments further highlight the versatility and robustness of our method in enhancing vision-language models to generate more reliable and detailed outputs.\n\n---\n\n**[W2]** Missing references.\n\n**[R2]** We appreciate the reviewer’s valuable suggestion. We will **include the recommended paper along with other recent relevant works** to ensure a comprehensive and up-to-date review of the MLLM research landscape in the revised manuscript."}, {"review_id": "Yoi5dz1f5b", "reviewer": "Reviewer_hLen", "summary": "This paper proposes a method (named ALFAR) to address two claimed limits of existing Multimodal Large Langeuage Models (MLLMs) 1. attention bias towards image tokens over text tokens and 2. knowledge conflicts between parametric and contextual knowledge.\nTo do this, ALFAR modifies and reallocates attentions on tokens of different modalities, adaptively learns logit representations for both parametric and contextual knowledge, and fuse two knowledge for answer prediction.", "questions": "Questions\n1. How is parametric and contextual knowledge formally defined? Is parametric knowledge limited to the knowledge acuired during training, and contextual knowledge limited to external data? as claimed in line 27-30 for MRAG? Is this contradictory from Fig. 4, where both logits for parameteric and contextual knowledge are derived from external data obeservations?\n2. Can Eq. 11 be explained with more details? What does each term represent?\n3. In Fig. 6, which part of the framework makes the attention be allocated more to Japan for LLaVa with ALFAR? Why doesn't the method promote other words such as Holland or Netherlands like LLaVA with MRAG or with CAD? Is the issue then really the bias of image tokens over text tokens? It seems to be accuracy identify which text tokens to pay more attention? It is not convincing that the bias attention towards image tokens is a problem or the root cause of incorrect answers.\n\nOther minor questions\n1. Should the y-axis of Fig. 2 be Attention Values (Sum) or alike instead of Distribution? What is the distribution in Fig. 2?", "limitations": "No.\nThe authors claim to have a Limitation section but is missing from the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is fair in quality and calrity, good in originality and significance.\n- Clarity: The paper clearly motivates the problem, describes the workflow and experiments, and explains the results, although some contents need more articulation such as the definitions of the parametric and contextual knowledge\n- Quality: The paper correctly formulates the problem, coherently develops the approach, and conducts experiments to support its claims, although some decisions should be examined and inspected, and some parts should be further explained. Please see Questions.\n- Originality: The paper proposes a novel approach to reallocate attentions and fuse knowledge, although similar ideas for attention reallocations have been introduced by earlier work.\n- Significance: The paper targets an important problem, but more validations are needed to enhance the significance of the work.", "was_revised": true, "final_justification": "The response from authors has addressed some of my concern. I therefore choose to adjust my rating.", "rebuttal": "Thanks for the constructive comments and suggestions. Below please find our clarification regarding your concerns.\n\n---\n\n**[Q1]** Definition of parametric and contextual knowledge.\n\n**[A1]** Your understanding of parametric and contextual knowledge is correct, and this distinction **aligns with our definition** and is widely adopted in the LLM literature [1] [2].\n\nRegarding Figure 4, we would like to clarify **a possible misunderstanding**: in the upper part of the figure, the logits represent parametric knowledge, which **does not have access to the context (depicted by the absence of blue squares)**. In contrast, the lower part illustrates contextual knowledge, where the context (blue squares) is visible. This setup is consistent with the definitions of the two types of knowledge as used throughout our paper.\n\nWe will revise the figure caption and description in the paper to make this distinction clearer.\n\n---\n\n**[Q2]** Explanation about Equation 11.\n\n**[A2]** Thank you for your question. We will add more details in the revised manuscript to clarify the intent and components of Equation 11. Specifically:\n1. $logit_c$ and $logit_p$ represent the logits derived from contextual knowledge and parametric knowledge, respectively.\n\n2. $\\lambda_{v}^{t}$ and $\\lambda_{c}^{t}$ denote the summed attention weights over image tokens and context tokens at decoding step $t$. These values reflect the relative importance of the image and context at that time step.\nWe will revise the explanation in the revised paper to improve the clarity and interpretability of this equation.\n\n---\n\n**[Q3]** Question about Figure 6.\n\n**[A3]** Your statement is correct and **aligns well with the key contribution of our paper**: accurately identifying which context tokens to attend to is essential, and this is precisely what our model is designed to achieve.\n\nAs shown in Equations (6) and (7), we **compute the relevance between the question and each context token** and use it as a guiding signal for attention reallocation. Such a design allows the model to **assign higher attention weights to context tokens that are more semantically aligned with the question**.\n\nIn the case illustrated in Figure 6, our model correctly focuses on the token \"Japan\", as it appears near the word \"Park\", which is also a keyword in the question. This localized relevance enables the model to ground the answer more accurately. In contrast, LLaVA with MRAG or CAD, which lacks such fine-grained context control, tends to over-rely on visual cues (e.g., the European-style building in the image), leading to incorrect predictions like \"Holland\" or \"Netherlands\".\n\nIn summary, our approach enables fine-grained control over context-level attention with question-context similarity, which dynamically reweights attention and reduces interference from less relevant image features. This leads to better knowledge utilization and more accurate predictions, which is aligned with your observations.\nWe will revise the text to better highlight this point and clarify this contribution.\n\n---\n\n**[Q4]** Y-axis of Figure 2.\n\n**[A4]** Thank you for the insightful question. \nIn this figure, we first sum the attention weights assigned to image tokens and context tokens separately, and then **normalize these two values so that their sum equals one**. The normalization allows us to reflect the relative proportion of attention allocated to each type of token, rather than their absolute magnitudes.\nWe referred to the Y-axis of Figure 2 as a “distribution” because it represents a normalized split of attention across the two modalities. To avoid confusion, we will revise the y-axis label to a more precise term like 'Normalized Attention Weights', and clarify the computation process in the main text.\n\n---\n\n**[Q5]** Missing Limitation section.\n\n**[A5]** We apologize for the confusion. Due to space constraints, **the Limitation section was placed in the Appendix**. In the revised version, we will move it into the main paper to ensure better visibility and completeness of the discussion.\n\n---\n\nReference\n\n[1] Knowledge Conflicts for LLMs: A Survey.\n\n[2] Mitigating Knowledge Conflicts in Language Model-Driven Question Answering."}]

Author rebuttal:

Meta-review: This work introduces ALFAR, a training-free and plug-and-play method to enhance Multimodal Large Language Models (MLLMs) in knowledge-intensive tasks by better utilizing retrieved contextual knowledge. The approach addresses two key challenges: attention bias toward different tokens and conflicts between parametric and contextual knowledge. ALFAR improves model reasoning by (1) reallocating attention from visual to context tokens based on query relevance, and (2) adaptively fusing logits to balance parametric and retrieved knowledge. Extensive experiments across multiple MLLMs and benchmarks show that ALFAR consistently outperforms prior methods, achieving state-of-the-art performance without additional training or external tools. During the rebuttal, the authors successfully addressed all reviewer concerns, and we have decided to accept this work. We encourage the authors to incorporate the promised changes and refinements in the camera-ready version.
Decision: Accept (oral)