Title: Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention Reallocation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Models

Multimodal Retrieval Augmented Generation
Abstract: Despite their recent progress, Multimodal Large Language Models (MLLMs) often struggle in knowledge-intensive tasks due to the limited and outdated parametric knowledge acquired during training. Multimodal Retrieval Augmented Generation addresses this issue by retrieving contextual knowledge from external databases, thereby enhancing MLLMs with expanded knowledge sources. 
However, existing MLLMs often fail to fully leverage the retrieved contextual knowledge for response generation. We examine representative MLLMs and identify two major causes, namely, attention bias toward different tokens and knowledge conflicts between parametric and contextual knowledge. To this end, we design Adaptive Logits Fusion and Attention Reallocation (ALFAR), a training-free and plug-and-play approach that improves MLLM responses by maximizing the utility of the retrieved knowledge. Specifically, ALFAR tackles the challenges from two perspectives. First, it alleviates attention bias by adaptively shifting attention from visual tokens to relevant context tokens according to query-context relevance. Second, it decouples and weights parametric and contextual knowledge at output logits, mitigating conflicts between the two types of knowledge. As a plug-and-play method, ALFAR achieves superior performance across diverse datasets without requiring additional training or external tools. Extensive experiments over multiple MLLMs and benchmarks show that ALFAR consistently outperforms the state-of-the-art by large margins. Our code and data are available at https://github.com/Lackel/ALFAR.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method (named ALFAR) to address two claimed limits of existing Multimodal Large Langeuage Models (MLLMs) 1. attention bias towards image tokens over text tokens and 2. knowledge conflicts between parametric and contextual knowledge.\nTo do this, ALFAR modifies and reallocates attentions on tokens of different modalities, adaptively learns logit representations for both parametric and contextual knowledge, and fuse two knowledge for answer prediction.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is fair in quality and calrity, good in originality and significance.\n- Clarity: The paper clearly motivates the problem, describes the workflow and experiments, and explains the results, although some contents need more articulation such as the definitions of the parametric and contextual knowledge\n- Quality: The paper correctly formulates the problem, coherently develops the approach, and conducts experiments to support its claims, although some decisions should be examined and inspected, and some parts should be further explained. Please see Questions.\n- Originality: The paper proposes a novel approach to reallocate attentions and fuse knowledge, although similar ideas for attention reallocations have been introduced by earlier work.\n- Significance: The paper targets an important problem, but more validations are needed to enhance the significance of the work.\n\nQUESTIONS:\nQuestions\n1. How is parametric and contextual knowledge formally defined? Is parametric knowledge limited to the knowledge acuired during training, and contextual knowledge limited to external data? as claimed in line 27-30 for MRAG? Is this contradictory from Fig. 4, where both logits for parameteric and contextual knowledge are derived from external data obeservations?\n2. Can Eq. 11 be explained with more details? What does each term represent?\n3. In Fig. 6, which part of the framework makes the attention be allocated more to Japan for LLaVa with ALFAR? Why doesn't the method promote other words such as Holland or Netherlands like LLaVA with MRAG or with CAD? Is the issue then really the bias of image tokens over text tokens? It seems to be accuracy identify which text tokens to pay more attention? It is not convincing that the bias attention towards image tokens is a problem or the root cause of incorrect answers.\n\nOther minor questions\n1. Should the y-axis of Fig. 2 be Attention Values (Sum) or alike instead of Distribution? What is the distribution in Fig. 2?\n\nLIMITATIONS:\nNo.\nThe authors claim to have a Limitation section but is missing from the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the limitations of Multimodal Large Language Models (MLLMs) in knowledge-intensive tasks, especially their inability to fully utilize retrieved contextual knowledge. The authors propose ALFAR, a training-free and plug-and-play method that mitigates attention bias and resolves conflicts between parametric and retrieved knowledge. Experiments show that ALFAR significantly improves LVLM performance across various datasets, surpassing existing state-of-the-art approaches.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The paper offers novel insights by identifying attention bias and knowledge conflicts as critical obstacles to effective knowledge utilization in Multimodal Large Language Models (MLLMs).\n\n2. It proposes ALFAR, a training-free and plug-and-play method that effectively reallocates attention and balances parametric and contextual knowledge.\n\n3. Extensive experiments demonstrate ALFAR’s superior performance and wide applicability across diverse generative and discriminative multimodal tasks.\n\nCons:\n\n1. While your method has shown strong performance on a range of VQA datasets, it would further strengthen the contribution to demonstrate its applicability across a broader spectrum of vision-language tasks. In particular, evaluating on image captioning tasks could highlight the versatility of your approach in generating context-aware and knowledge-grounded descriptions.\n\n2. It would be helpful to cite a recent survey on Multimodal Large Language Models to provide readers with a comprehensive background and better contextualize your contribution within the broader MLLM research landscape [1].\n\n[1] \"MM-LLMs: Recent Advances in MultiModal Large Language Models.\" Findings of the Association for Computational Linguistics ACL 2024. 2024.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a key limitation of multimodal large language models (MLLMs): their inability to effectively leverage retrieved external knowledge. To overcome this, the authors propose ALFAR, a training-free and plug-and-play method designed to mitigate attention bias and resolve knowledge conflicts in MLLMs when tackling knowledge-intensive tasks. Through its two core components—Attention Reallocation and Adaptive Logits Fusion—ALFAR significantly enhances knowledge utilization performance. Experiments conducted on four MLLMs across six knowledge-centric datasets demonstrate that ALFAR consistently and substantially outperforms prior state-of-the-art decoding methods.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n1. The authors accurately identify two critical issues that limit knowledge utilization in MLLMs, attention bias and knowledge conflicts, and provide detailed analyses and validation of their causes and impacts.\n2. The proposed two modules, attention reallocation and adaptive logits fusion, offer a novel and effective perspective to enhance MLLM performance in knowledge-intensive tasks. \n3. Experiments across multiple representative MLLMs and diverse datasets validate the effectiveness of the proposed framework.\n4. As a training-free and plug-and-play module, ALFAR can be directly applied to many MLLMs in MRAG pipelines.\n\n# Weakness\n1. The evaluated LVLMs are somewhat outdated; the framework’s effectiveness should be validated on newer LVLMs (e.g., Qwen2.5-vl, LLaVA-Next).\n2. Additional analysis and validation of the framework’s robustness to varying retrieval quality would be beneficial.\n3. The details of the compared methods are not clearly described.\n\nQUESTIONS:\n1. The analysis of different intervention layers is interesting but might be better placed in the main paper rather than as supplementary material.\n2. The definitions of certain variables (e.g., β, γ, λ) should be clarified and explained more thoroughly.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

comments: SUMMARY:
This work introduces a training-free, plug-and-play framework that tackles attention bias and knowledge conflicts in multimodal large language models through adaptive attention reallocation and logits fusion, yielding consistent performance gains of 6–15% across multiple VQA benchmarks and backbone models without additional training.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The method is simple and effective with training-free and plug-and-play manners, requiring no additional model training or external tools for integration.

2. The preliminary findings on attention bias and knowledge conflicts, which hinder the effective utilization of retrieved context, are convincing.

3. Extensive evaluation across multiple MLLMs and diverse VQA benchmarks demonstrates consistent performance gains (6–15% improvements). Ablation studies and tests with different retrievers confirm the effectiveness and robustness of each component and the overall approach.

Weaknesses:

1. The proposed method appears incremental, as the techniques involving attention weighting and logits fusion are already well-known and mainly borrowed from previous works.

2. A comparative inference cost between the proposed method and the existing approach (MRAG) is not provided.

QUESTIONS:
My main concerns are in the method section, please see the weakness above.

LIMITATIONS:
The limitation discussions are sound and clear to me.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03