Towards General Continuous Memory for Vision-Language Models
Authors: Wenyi WU
Zixuan Song
Kun Zhou
Yifei Shao
Zhiting Hu
Biwei Huang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Multimodal Large Language Models
Multimodal Reasoning
Retrieval-Augmented Generation
Memory-Augmented Models
Primary area: deep_learning
TL;DR: We enable vision-language models to efficiently encode and utilize multimodal multilingual knowledge through a lightweight, continuous memory mechanism.

Abstract: Language models (LMs) and their extension, vision-language models (VLMs), have achieved remarkable performance across various tasks. However, they still struggle with complex reasoning tasks that require multimodal or multilingual real world knowledge. To support such capabilities, an external memory system that can efficiently provide relevant multimodal information is essential. Existing approaches generally concatenate image and text tokens into a long sequence as memory, which, however, may drastically increase context length and even degrade performance. In contrast, we propose using continuous memory-a compact set of dense embeddings-to more effectively and efficiently represent multimodal and multilingual knowledge. Our key insight is that a VLM can serve as its own continuous memory encoder. We empirically show that this design improves performance on complex multimodal reasoning tasks. Building on this, we introduce a data-efficient and parameter-efficient method to fine-tune the VLM into a memory encoder, requiring only 1.2\% of the model’s parameters and a small corpus of 15.6K self-synthesized samples. Our approach CoMEM utilizes VLM's original capabilities to encode arbitrary multimodal and multilingual knowledge into just 8 continuous embeddings. Since the inference-time VLM remains frozen, our memory module is plug-and-play and can be flexibly integrated as needed. Extensive experiments across eight multimodal reasoning benchmarks demonstrate the effectiveness of our approach. Code and data is publicly released here https://github.com/WenyiWU0111/CoMEM.

Towards General Continuous Memory for Vision-Language Models
Wenyi Wu", Zixuan Song', Kun Zhou', Yifei Shao, Zhiting Hu, Biwei Huang University of California, San Diego
kuzhou@ucsd.co edu
Abstract
Language models (LMs) and their extension, vision-language models (VLMs), have achieved remarkable performance across various tasks. However, they still struggle with complex reasoning tasks that require multimodal or multilingual real-world knowledge. To support such capabilities, an external memory system that can efficiently provide relevant multimodal information is essential. Existing approaches generally concatenate image and text tokens into a long sequence as memory, which, however, may drastically increase context length and even de- grade performance. In contrast, we propose using continuous memory-a compact set of dense embeddings-to more effectively and efficiently represent multimodal and multilingual knowledge. Our key insight is that a VLM can serve as its own continuous memory encoder. We empirically show that this design improves per- formance on complex multimodal reasoning tasks. Building on this, we introduce a data-efficient and parameter-effice efficient method to fine-tune the VLM into a memory encoder, requiring only 1.2% of the model's parameters and small corpus of 15.6K self-yytteesized samples. Our approach CoMEM utilizes VLM's origi- nal capabilities to encode arbitrary multimodal and multilingual knowledge into just continuous embeddings. Since the inference-time VLM remains frozen, memory module is plug-and-play and can be flexibly integrated needed. Extensive experiments across eight multimodal reasoning benchmarks demon- strate the effectiveness of approach. Code and data is publicly released here /tte:////ttttt... http:/gtaaa.cc///eyy////1///////
Introduction
Through large-scale training, language mod- els (LMs) 2 have demonstrated remarkable performance across diverse real-world tasks. LMs even surpass human capabilities in language reasoning tasks (3] such mathematical problem solving [4) commonsense reasoning [3, and code synthesis 66) However, when confronted with complex reasoning tasks that demand multimodal or multilingual world knowledge, both LMs and their vision vision-language model (VLM) extensions continue to insufficient world knowledge representation.
1. COMEN
tyfremm buldive offcol rooddly Datr
Encodar
Continsous Enbedding
2. Traditional RAG
Figure 1: CoMEM architecture in comparison to the traditional RAG method.
face significant challenges 7 8], primarily due
'Equal Contribution
CCorresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inspired by how humans offload facts, plans, and ideas to external repositories like notebooks or databases for on-demand access, it is promising to develop a general external memory that contains useful world knowledge for augmenting VLMSs 2 I0 Farly approaches directly concatenate the collected useful information into a long token sequence, and feeds it into VL.Ms p 11 e.g., parrieval-augennte generation (RAG) methods. However, multimodal representations demand significantly more input tokens (e.g., 8 to 14227 tokens per image in Qwen2.5-VLL (12)). Thus, simple concatenation would greatly increase the input length, making it difficult for the memory content to be used [3] (see the degradation in performance shown in Table after using RAG). To solve the token overload issue, token pruning methods have been proposed remove unimportant in-context tokens 15). However, token pruning generally leads to incomplete contextual contents, which impedes the VLM's ability to accurately understand and utilize the compressed information [8].
Compared to discrete tokens, continuous embeddings naturally have stronger representation capability for complex data [16] 17 [8. This advantage makes them particularly promising for memory encoder architectures designed condense multimodal information into continuous representations. However, training such encoders faces two key challenges: (I) achieving generalizable compression ability across diverse multimodal inputs, and (2) maintaining semantic alignment with the VLM [19]. While large-scale training can improve performance, it greatly increases the training cost and becomes heavily sensitive to the training data distribution. For example, when dominated by simple cases or a single domain, the encoder tends to overfit and degrade generalization performance (20 21].
In this paper. we focus on efficiently training a general continuous memory encoder to effectively supply multimodal knowledge for VIMMs. To avoid costly training for semantic alignment, it is essential to minimize the representation gap between the memory encoder and downstream VL.Ms before training. Therefore, a natural way is to use the VLM itself as the memory encoder. Our empirical study confirms that the VLM can serve as a memory encoder for itself without any additional training. Benefiting from the stacked self-attention mechanism, its generated continuous embeddings in each layer have already aggregated rich semantic information [2 233 As shown in Fig. 2 even a simple rule-based embedding selection strategy for constructing the memory can greatly boost the performance of VLMs in complex multimodal multilingual reasoning tasks, compared to RAGs.
Based on our empirical findings, we propose a data-efficient and parameter-efficient training recipe to further improve the compressor rate and adaptation performance of the VI.M-based general continuous memory encoder. Concretely, we only need to fine-tune the low-rank adaptation matri- ces (LoRA) (44 in the VLM-based memory encoder, and lightweight Q-Former (25) for further compressing the VLM representations into only eight embeddings, 1.2% parameters in total. In terms of data, we only need the VLM itself to synthesize 15.6k samples for training. This efficient training strategy enables our continuous memory to reuse the original ability of the VLM, to effectively encode multimodal and multilingual knowledge. Since we do not need to train the inference VLM, memory is souu-and--lay and can be flexibly integrated with the VLM when necessary.
To demonstrate the effectiveness of our approach, we apply our method state-of-the-ar VLMs, and evaluate the performance across eight visual reasoning benchmarks. For six English visual reasoning benchmarks, our method achieves an average improvement of +8.0% on Divenn-Inttruct-i VL and +7.7% on Swen2.55IInsrrue VL. On two multilingual multimodal benchmarks, our approach further improves performance by +5. 5..1% 1% and +4.3% on Owen2-IIntruceh VL and Swee2.55IInstruee VL. respectively. Furthermore, our adaptation study results also indicate the transferability of our VLM- based memory encoder to improve LMs visual reasoning tasks. The long context understanding study also exhibits the stable and superior performance of our method.
Empirical Analysis with VLM as Memory Encoder
In this section, we conduct an empirical study to examine (1) whether VLM can serve as a continuous memory encoder to compress multimodal information into compatible embeddings, and (2) whether a few embeddings from the VLM can preserve key information to improve multimodal reasoning tasks.
'In this paper, external memory denotes any detachable module or function that supplies the knowledge with- out changing LM parameters, in contrast to internal memory that embeds knowledge by modifying parameters.

---

Table 1: Comparison between our approach and other representative line of work.
Properties
Scenarios
Training Cost
Category
Method
Continuous Pulg-and-Play Multimodal
Multilingual
Data
Parameters
EchoSight
26
XText
X
900K
300M
Multimodal-R.e
ReflectiVA
X
/Image+Text
X
6,82M
8B
RoRA-VLM
28
X
Immage+Text
1M
7B
XRAG
29
Xlext
3M
40M
Comtext-Compression
Kv-Distill
30
XText
500K+
150M
VoCo-LLaMA
31
X
VImage/Video
665K
7B
LONGMEM 10
Xlext
114M
558M
MA-LMM 32
Vedeo+Texxt
NA
200M
LM Memory
M+ 33
/
X
XText
5M
NA
MemGPT
XText
NA
NA
CoMEM
Image+Text
15K
2.1 Analysis Setup
For the empirical study, we conduct experiments on two state-of-the-art VLMs, i.e. Qwen2-VL-7B and Qwen2.5-VL-7B, and test the performance three multimodal reasoning benchmarks.
Evaluation Settings. To compare the effectiveness of different memory and context compression methods, select three benchmarks: InfoSeek [35], OK-VQA (36,, and A-OKVQA 37 These benchmarks contain complex visual questions that require both accurate visual entity identification and multi-step reasoning to derive the correct answer. Following existing work [38 288 for each question, we utilize CLIP-based retriever (39] to collect relevant top- 10 multimodal knowledge items
from Wikipedia- wikseedda-bbsedd source dataset WiT 40) construct the input data for the memory.
Table Comparison of training-free memory methods, Bold indicates the best performance. For "LM-as-Memon here, we use the cache KV from a VLM without fine-tuning, which differs from the main method described in Section and is intended for preliminary exploration.
InfoSeck
OKVQA
AOKVQA
Backbone Model
Method
Query
Entity
All
22.5
22.4
22.5
35.0
39.8
+RAG +FastV
17.7
18.8
18.2
31.3
34,9
Owen2.5-VL-Instruee
26.2
22.6
24.2
31.5
34,9
-VLM-as-Memone
29.3
28.0
28.6
37.3
44.4
VLMM-as MeMM------
30.0
25.3
27.5
37.9
41.8
17.9
17.8
17.9
36.3
41.8
22.7
19.0
20.5
45.3
Qwen2-VL--Intruuct
+FastV
23.6
23.8
23.7
42.0
45.4
WLM-as-Memanl
28.8
29.7
29.3
37.7
38.9
MeMoossMemory+AA
31.7
28.8
30.2
34.3
36.4
Memory Methods. We test the effectiveness of our VI.M-as-memory method by comparing with RAG, token pruning, and our variations using different embedding selection strategies.
Vanilla RAG; simply concatenates all the multimodal knowledge items into long sequence, and then feeds it with the visual question as the input of VLM.
FastV [4] it adopts token pruning strategy that discard the image tokens with lower attention scores from the multimodal knowledge items. Then, the pruned token sequence is fed into the VLM.
ViMM-a-Memory: utilize the VLM itself to encode the knowledge items, and extract the hidden states all layers as the memory. These are concatenated at corresponding layers of the VLM. For efficiency, only add the memory in 17-19 layers.

---

VLM-as-Mee -MMMmooryooo we utilize the average attention scores across all layers within the VLM to select the top-25% key continuous embeddings to compose the memory.
2.2 Results and Findings
In this part, present the results and discuss the findings analyze whether VLM can be memory encoder, assessing both their compression efficiency and semantic alignment capabilities.
Effectiveness Study VLM-as-Memory Methods. As shown in Table 2 the vanilla RAG method causes perfor- mance degradation in several tasks, underscoring its lim- itations for memory integration. FastV mitigates this is- sue by pruning redundant tokens in the memory, result- ing in measurable improvements. Notably, the VLM-as- Memory method outperforms both approaches across most tasks, suggesting that the VLM's continuous embeddings are inherently more compatible with its own processing than token-based input. Furthermore, with the addition of simple attention-based compression mechanism, the VLM- as-Memory method achieves even greater performance gains. Thus, we conclude that:
Infrook
GK-VD4
ADE VQA
RAG
RAG
BAG
Acoumna (%)
30
25-
20-
25%
75%
90%
Compression Rate
Figure 2: Qwen2.5 accuracy with vary- ing attention- based compression rates.
(I) VLMs effectively serve their own memory coders for external multimodal knowledge. The continuous embeddings they generate can be directly reused by the same model without requiring additional training.
(2) The continuous embeddings produced by VLMs effectively preserve knowledge content. They remain robust under simple compression strategies and reliably enhance performance.
Compressibility Study of VLM-as-MMmmon Methods. To study the compressibility, another key feature of an effective memory mechanism, investigate how performance varies under different compression rates using the (LM-as-Memory approach. As shown in Fig. 2 despite employing simple token selection strategy, our method outperforms the baseline even high compression rate of 5This suggests that a small number of continuous embeddings already encapsulate most of the essential contextual information from the input. Therefore, we can conclude that:
(3) The continuous embeddings produced by VLMs support high compression rates. This highlights the potential for achieving even greater compression through more advanced compression methods.
Approach
gelte de Curerr 2o
Memory Encoder
Answer: Greece
Renfeeee Information
Q-Former
What
Age (iresl Ayps cerieaakly fitees
VLM Layer
Albention Layer
Share Params
Memory
Whrt
Layer
LM
Input Embedding
Figure 3: Overview of the CoMEM architecture. Given vision-language query, the system retrieves relevant multimodal knowledge via visual features. Retrieved image-text pairs are processed by Memory
consists of a VLM and Q-Former-a to generate a dense continuous memory. This memory and the original query are fed into a frozen LM to produce accurate, grounded answers.
According to our empirical study, the VLM can an effective memory encoder for itself, owing to the satisfactory semantic alignment and compressibility of its produced continuous embeddings.

---

Building on this insight, we aim to efficiently train the VLM into a continuous memory encoder, to supply supplementary multimodal knowledge during inference. Concretely, we add trainable lightweight Q-Former to control the compression rate, synthesize small training dataset using the VLM itself, and perform data-efficient and parameter-efficient training.
3.1
Task Definition.
We aim to train a general-purpont continuous memory encoder capable of mapping arbitrary multi- modal and multilingual data into continuous embeddings that augment the knowledge of a VLM. To ensure alug-nn--pla compatibility, we keep the VLM's parameters frozen during inference, while continuous memory embeddings are directly used for downstream tasks. To achieve this, the memory encoder should (1) efficiently condense diverse multimodal and multilingual data and (2) produce embeddings that are both readable and functional for the VLM.
In this paper, we focus on using general continuous memory to enhance VLMs in complex multimodal reasoning tasks. Formally, given an instance comprising an image and a natural language question the task is to predict an accurate answer a. Following prior work [28], we assume access to relevant multimodal knowledge items (from external knowledge source), each consisting of an image i and natural language description d. Our memory encoder learns to transform each knowledge item into continuous vector, formulated as V. = f(i,, d.). These vectors are aggregated into unified memory. which the VLM then utilizes for answer prediction: p(ali, q, (V.)..))
3.2 VLM-Based Continuous Memory
For our continuous memory, the core idea is to leverage the VLM with a Q-Former as the encoder, and adopt simple plug-and-play mechanism that enables the VLM to use the memory information.
Continuous Encoder. Given each multimodal knowledge item (ir, de). we first use the VLM to encode it and collect the continuous representations E, in the last layers. Then, we employ a query Transformer (Q-Former) the compressor to condense E, into k continuous embeddings V.. The Q-Former consists of l: query embeddings q and L. Transformer layers. In the first layer, the query embeddings attend to all the continuous representations from the VLM through the cross-attention mechanism. The output representations are then used as query embeddings for the next layer, and the final layer outputs serve as the memory vector V,. The whole process is formulated as:
H (0) q.
H TrrasffrmerLayerr (2) (H((,1), E,), V, V (L)
(1)
To reduce the parameter scale of the Q-Former, share parameters across all Transformer layers, and set & = 8. In this way, only a few parameters are added, and any multimodal knowledge item will be compressed into 8 continuous embeddings. This design ensures lower training cost and higher compression rate which is helpful to handle large-scale knowledge items and save the storage cost.
Plug-and-PHe Mechanism. After obtaining the continuous embedding set [Ve) for all multi- modal knowledge items, we adopt simple plug-and-play mechanism to equip the VLM the memory. Concretely, we simply concatenate the embeddings into a sequence as the memory, which is prepended to the input embedding E, of the VLM during the inference time, formulated as [Vi;..- Vn, Ey]. In this way, the VLM can naturally perform autoregressive generation to predict the answer, using its originally learned knowledge and capabilities.
x n continuous vectors
3.3 Efficient Training Recipe
Since we introduce the Q-Former, we need train its parameters achieve full alignment between the continuous memory and the VLM. Thanks to our design that employs the VLM as the memory encoder, this alignment can be efficiently accomplished through parameter-efficicet training using only small amount of self-synthetic multimodal and multilingual data.
'Notably, the average token number of knowledge items in this work is 643.7, and few extremely long ones contain more than 2000 tokens. By compressing into 8 tokens, we achieve more than 80 compression rate.
5

---

Training Data Self-synthesis. To ensure training efficiency, we construct our training dataset by synthesizing responses using the VLM itself, based on multilingual and multimodal questions from existing benchmarks. Specifically, we begin by selecting questions from the training sets of InfoSeek [35], Encyclopedie-VO0 (EVQA) [41], and OK-VQA (36] to ensure coverage of diverse multimodal reasoning tasks. For each question, we retrieve three relevant image-text pairs from the WIT [40] knowledge base using CLIP, following the retrieval setup in prior work (28). These pairs serve supplementary multimodal knowledge items. We concatenate the question with knowledge items and input the sequence into Owen2.5-VVLIInsruue to simulate a vanilla RAG setting. Only outputs yielding correct answers retained, resulting in 13.8k high-quality training instances. To extend our dataset beyond English, we randomly select 200 training samples and employ GPT-4o- mini to translate the text part into nine languages: Bulgarian, Chinese. Egyptian Arabic, Filipino, French, Japanese, Portuguese, Russian, and Spanish. This results in an additional 1.8k multimodal multilingual training samples, which aims at activating our model's ' cross-lingual capabílities. In total, our final fine-tuning corpus for continuous memory includes 15.6K curated samples, covering a variety of multimodal tasks and languages.
Parameter-effice effficiett Fine-tuning. Given the above training data, we perform parameter-efffcient fine-tuning on the Q-Former and LoRA layers in the VLM encoder. For efficiency, we apply LoRA with a rank of 16 and share parameters across all layers of the Q-Former. Therefore, only 1.2% of total parameters are trainable. The above parameter and data efficient designs guarantee that our entire training process can be completed on single NVIDIA H100 GPU in 20 hours. We also empirically find the training converges fast, and a single epoch is sufficient to achieve strong performance.
3.4 Discussion
In T we compare our method CoMEM with ten closely related works: i.e., multimodal RAG (EchoSight [26] ReflectiVA (271, and RoRA-VLLY (28)), context compression (xRAG [29], KV-
Distill 30 and YoMM BI), and LLM memory methods (LONGMEM [TO], MA-LMM 32)
M+ 33 and MemGPT 34)) The comparison spans three dimensions: Properties, where we
examine whether the method is continuous and plug-and-plaa Scenarios, evaluating support for multimodal and multilingual inputs; and Training Cost, which includes the amount of training data required and trainable parameters.
While some existing methods also adopt continuous embeddings and support plug-and-play usage, they often require substantial training resources-typicee
involving millions of training samples and extensive parameter updates. In contrast, our method achieves comparable functionality with significantly reduced cost: it utilizes only 15.6k self- celf-synthesized training samples and fine-tunes just 200M parameters, amounting to only 1.2% of the full model. Morcover, a key advantage of our method is our method can handle both multimodal (text and image) and multilingual data, which is very helpful for potential applications in low-resource language settings.
In summary, our proposed method, COMEM, provides generalizable, scalable, and compute-efficien solution for augmenting VLMs with a continuous memory mechanism. By leveraging the VLM itself as the memory encoder, CoMEM ensures strong semantic alignment between the memory and the model, while supporting seamless plug-and-play integration for diverse downstream tasks. This design enables effective reasoning over complex multimodal and multilingual inputs, offering a unified and efficient alternative to existing approaches that often rely on discrete context inputs, heavy fine-tuning. or multi-stage retrieval pipelines.
4 Experiments
4.1 Experimental Setup
Evaluation Settings We use WIT 40 (Wikipedia- based Image Text Dataset) as our retrieval knowledge base. Building upon this, we conduct experiments across eight multimodal and multilin- gual reasoning benchmarks, including six multimodal reasoning benchmarks: InfoSeek (35), OVEN [42] MRAG-Bench 43 OK-VQA 36 A-OKVQA (37, and ViQuAE 44]. and two multilingual
benchmarks: CVQA 45 and multilingual InfoSeek. Here we use GPT--4-mini 46 to translate the
InfoSeck from English into five different languages to match the language settings of CVQA.

---

Note (1) InfoSeek and OVEN are constructed from Wikipedia and consist of challenging factual questions. (2) BeRAG,,ennc, OK- VQA, A-OKVQA, and ViQuAE focus on multimodal real-world, inowledge-intensive tasks. (3) CVQA and multilingual InfoSeek evaluate model's s ability to reason diverse linguistic and cultural contexts. Further details about benchmarks are Appendix A
Baseline Methods We compare our method against three types of baselines: (1) VLMs. (2) VLMs with vanilla RAG, and (3) advanced RAG methods, covering a total of 18 different models.
For VLMs, we evaluate their original capabilities on multimodal reasoning tasks without access to external knowledge, including: LLa [47) LLaVA-v1.6 47), LLaVA-NeXT-LLaMA (denoted as LLaMA3 in tables) 488)
CComposer22....mmcmm 5vl ((nternL.2..5vl) [49], mPLUG- Owl3 [50], Qwen2-VL VL-Instruct (Qwen2-VL) [1T] and Qwen2.5- ----nttuut (Qwen2.5-VLL [2]
For VLMs with vanilla RAG, we directly insert retrieved image-text pairs into the input prompts of models, without making any architectural modifications applying additional fine-tuning. This setup evaluates the effectiveness of naive retrieval- -aasee augmentation.
Wiki-LLaVA and RORA-VLM use two-stage retrieval to improve knowledge relevance, while ReflectiVA adds reflective tokens for self-filtering. All three fine-tune the inference-time model. In contrast, EchoSight trains a separate Q-Former for retrieval without training the inference model. However, they all rely on discrete context inputs, which limits their ability to handle long contexts.
Implementation Details Our experimental pipeline comprises three phases: Knowledge Retrieval, Knowledge Compression, and Answer Generation. To ensure fairesss, we consistently use the top-10 retrieved image-text pairs across all experimental settings. We evaluate our method on Qwen2- Instruct-VI VL and Qwen2.5 Sween.5-Instruet-VL. demonstrating its strong generalization capability across different VLMs and question types. More implementation details can be found in Appendix B
4.2 Main Results
Evaluation on Multimodal Reasoning Task Table presents the performance comparison across six multimodal reasoning benchmarks, categorized into Base Models, Retrieval-Augmennte Baselines, and our Continuous Memory approach. Among base models, Qwen2- VL and Qwen2.5-VL achieve the highest performance across most benchmarks, which is likely due to their extensive multimodal alignment. However, standard RAG integration often
training corpus and strong leads to inefficiencies in processing longer multimodal inputs, resulting in unstable performance that sometimes underperforms base models. To address this issue, advanced RAG models incorporate mechanisms that retrieve and use relevant content more effectively, resulting in improved performance on reasoning tasks. However, as shown in Section 3.4 existing methods still face limitations, such as difficulty in adapting across modalities
lack of generalizability across diverse task settings.
In comparison, our approach shows significant gains across multimodal reasoning benchmarks, with particularly strong improvements (over 15%) on OKVQA and A-OKVQA versus baselines. These advancements originate from our VLM-based continuous memory architecture, which exhibits both strong adaptability to different VLMs and excellent generalization across diverse tasks. Remarkably. this level of performance requires minimal fine-tuning (1.2% of parameters on 15.6k samples from InfoSeek, OK and EVQA subsets). yet still achieves remarkable improvements on unseen benchmarks such as OVEN and A-OKVQA. This suggests that our method can effectively fuse multimodal long-context knowledge, and generalize effectively to a wide range of downstream tasks.
Evaluation on Multimodal Multilingual Reasoning Task We further evaluate our model's mul- tilingual reasoning capabilities on the multilingual InfoSeek and CVQA benchmarks. As shown in Table 4. standard RAG methods demonstrate reduced effectiveness for non-English questions, potentially due to misalignment between retrieved multilingual content and input queries. In contrast, our memory mechanism encodes and stores transferable semantic representations that preserve core cross-modal and cross-lingual knowledge. This design translates into consistent accuracy improve- ments across all evaluated languages, achieving absolute gains of 6-12 points on InfoSeek-All scores while simultaneously showing enhanced performance on CVQA metrics. Notably, the model achieves particularly strong performance gains for Bulgarian (18%) and Russian (10%), underscoring the value of our manguage-agnoste memory mechanism for lower-resource settings where high-qualit,
7

---

Table 3: Performance comparison with three types of baselines on knowledge-intes intensive VQA benchmarks. Bold indicates the best performance, and underscore denotes the second-bes best.
InfoSeek
OVEN
MRAG
OKVQA
AOKVQA
ViQuAE
Avg.
Model
Q
R
Q
LLaVA-v1.5
20.0
11.1
15.1
LLaVA-v1.6
17.9
33,4
31.7
18.7
19.3
LLaMA3
16.8
25.3
17.2
17.1
IntemLLM-2.5vl
10.8
14.5
29.7
19.5
mPLUG-Ow13
20.7
23.1
21.4
Qwen2-VL
25.5
41.8
34.5
27.8
Qwen2.5-VL
22.5
22.4
29.3
16.3
39.8
39.0
30.8
LLaVA-v1.5 RAG
14.6
11.4
11.7
7.6
34.7
9.8
8.7
7.6
13.3
I.AVV-v1.6 RAG
6.7
5.8
9.7
1,2
32.6
25.6
22.6
17.0
15.2
[LaMA3 RAG
12.1
10.8
24.7
21.5
36.4
20.7
22.1
18.1
20.8
InternLM-2.5vll RAG
10.5
9.5
15.2
13.6
34.3
25.9
27.8
29.6
20.8
mPLUG-Owl3 RAG
12.6
7.2
18.0
12.0
41.9
24.7
26.4
22.5
20.7
Qwen2-VL- RAG
22.7
19.0
24.7
21.5
40.4
41.9
45.3
33.6
31.1
Qwen2.5-VL+ RAG
17.7
18.8
23.0
19.7
42.1
31.3
34.9
33.5
27.6
Wiki-LLaVA
28.6
25.7
27.2
RORA
27.3
26.2
22.9
EchoSight
18.0
19.8
41.3
20.0
16.9
25.2
23.5
ReflectiVA
28.6
28.1
39.7
47.5
47.6
29.8
36.9
CoMEM + Qwen2VL
32.6
33.1
30.5
23.6
35.1
57.7
60.6
36.3
38.7
CoMEM + Qwen2.5VL
32.8
28.5
26.0
20.8
38.1
47.6
55.0
34.7
35.4
retrieval is hardest obtain. Overall, the results show that method enables robust grounding of multilingual queries and enhances reasoning capabilities across diverse tasks.
Table 4: Performance comparison on Multilingual knowledge-intensive VQA benchmarks.
Owen2.55-nstrructvv VL
Swee2-Inntructee
Language
Method
Multilingual InfoSeek
Multilingual InfoSeek
CVQA
CVQA
Unseen-Q
Unseen-F
All
Unseen-Q
Unseen-F
All
17.4
15.4
82.32
15.1
10.9
74.60
Chinese
RAG
14.8
9.8
74.60
11.5
10.1
72.35
CoMEM
22.5
21.5
22.0
78.46
23.1
19.5
21.1
73.31
15.6
14.7
15.1
66.50
13.0
13.8
13.4
71.00
Russian
RAG
10.6
8.9
9.7
66.00
13.6
10.9
12.1
62.50
+ CoMEM
21.8
21.3
21.5
70.00
19.3
20.4
19.8
71.00
17.3
16.5
16.9
75.79
16.7
16.7
16.7
72.64
Spanish
RAG
12.3
11.4
11.8
79.25
9.5
8.1
8.7
76.10
CoMEM
24.0
23.3
23.6
79.87
23.0
21.8
24.3
75.47
18.7
18.1
18.4
66.55
18.4
19.3
18.8
66.90
Portuguese
RAG
15.8
13.8
14.7
62.32
13.7
13.5
13.6
70.07
CoMEM
27.1
27.2
27.2
66.90
24.1
26.1
25.1
67.96
12.5
12.0
12.2
46.09
8.0
7.9
7.9
45.55
Bulgarian
RAG
9.8
7.0
8.2
46.63
8.5
7.1
7.7
39.89
CoMEM
19.3
17.4
18.3
47.44
15.9
18.3
17.0
50.13
17.3
16.2
16.7
67.45
14.8
14.4
14.6
66.14
Overall
RAG
13.8
11.4
12.5
65.76
13.3
11.3
12.1
64.18
CoMEM
24.9
23.6
24.2
68.53
23.0
23.2
23.4
67.57
4.3 Further Analysis
Long Context Understanding Study
To evaluate the ability of models to handle long-context inputs, compare method against vanilla RAG under varying numbers of retrieved image-text

---

knowledge pairs. Specifically, we evaluate Qwen2- VL- "L-Inntruct and In--ssruttt on Infoseek, using both vanilla RAG and our method across different top-k retrieval settings (from 3 to 50).
As shown in Figure 4 the results reveal clear trend: RAG-based performance begins to degrade when more than 30 re- trieved pairs are added, but our method remains stable and performs consistently well across all retrieval sizes. These find- ings show that discrete token-based meth- ods struggle with long context, while con- tinuous memory enables scalable and re- liable long-context reasoning. This ro- bust performance as context length in- creases underscores the advantage of our approach in processing long, information- dense inputs.
QwarZ-RGG
Quen22-Durs
30
Accurio
20-
15-
10
40
Top-k Retrieved Knowledge Pairs
Figure 4: Comparison Long Context Ability of RAG and Ours Infoseek.
Table 5: Transferability Study of vision- language memory encoded by CoMEM on LLMs
InfoSeek(
OVEN(%)
LLM
Unseen-Q
Unseen-E
Query
Entity
Avg.
Qwen2.5- I-sstruct
4.8
2.4
0.1
1.3
Qwen2.5-Instruce RAG
13.4
10.3
11.9
1.8
2.7
2.2
7.0
Qwen2.5-Instrice CoMEM (using VI.M)
29.3
27.4
28.3
6.8
7.7
7.2
17.8
To investigate whether the multimodal and multilingual continu- ous memory generated by a VLM can be effectively transferred and leveraged by pure Large Language Model (LLM), we conduct a transferability study. Specifically, we use Qwen2.5- VL- Instruct to encode visual and textual knowledge into dense continuous memory, and appended the input embeddings of -.wetn.--Insrruct, language-only LLM without vision capabilities.
Transterability Study to LLMs.
We evaluate our approach on InfoSeek and OVEN. As shown in Table our approach significantly outperforms both the vanilla LLM and the LLM augmented with text RAG, achieving an average accuracy of 17.8%, compared to 7.0% (RAG) and 3.1% % (baseline). These results demonstrate that LLMs can effectively leverage VLM-generater memory, even without vision modules. This highlights a promising direction for cross-modal knowledge transfer, enabling LLMs to gain visual understanding through shared continuous memory without any architectural modifications.
5 Related Work
Vision-Language Models. LLMs have seen significant advancements, with models like GPT- 4 [46] and Qwen-2.5 [52) demonstrating emergent capabilities such as in-context learning and complex reasoning. Building upon these advancements, VLMs have emerged to integrate visual and textual modalities, enabling models to process and understand multimodal data. To effectively extend language understanding into the visual domain, VLMs combine specialized neural network architectures for vision processing (such as Vision Transformers) with language models, enabling joint reasoning over visual and textual inputs. These models are typically trained on large-scale datasets that pair images with descriptive text to learn joint representations, using techniques like contrastive learning [53 39). multimodal pretraining [54 55 and instruction-aware tuning [47 [66] 57].
Context Compression. The constrained context windows of language models limit their infor- mation processing capacity, prompting the development of context compression methods to enable longer-sequence handling. One of the approach towards context compression in LLMs is through token pruning. FastV [1] distills vision-language knowledge into compact key-value memory slots, while Sparse [1] selects sparse subset of visual tokens via top-down routing. In contrast, Cisting[58] compress long prompts into small set of reusable "gist tokens" by modifying Trans- former attention masks. Another approach involves soft prompts, which introduce trainable vector

---

embeddings to input sequences, enabling efficient task adaptation. IC-Former [59] compresses long input sequences into compact digest vectors, while 60 combines natural language sum- marization with trainable soft prompts. Both methods condense lengthy input sequences into shorter representations, enhance the efficiency of LLMSs and preserve over 90% of the original performance.
Memory for Language Models. As LMs face limitations in context length and long-term informa- tion retention, memory mechanisms have emerged to enhance their capacity for information-intent reasoning and knowledge storage. Early retrieval-basee approaches such as RAG [ and REALM [[]] retrieve external documents and inject them as long token sequences during inference time. However, these methods are constrained by context length limits and the inefficiency of discrete token represen- tations, especially for supporting multimodal information. Recent advances shift toward continuous memory, representing knowledge as dense vectors rather than raw text. Approaches like VoCo-
LLaMA [31] and MA-LMM [32] compress visual content into compact embeddings. Concurrently, strategies for memory storage have evolved. Persistent memory systems such as LONGMEM [TO] store compressed knowledge in cache key-value (K formats, while retrieval-based methods like WikiLLaVA [38], RORA-VLM (88), and EchoSight (26 treat external knowledge bases as memory
banks, using dedicated retrieval frameworks to support VQA tasks.
Conclusion
In this paper, we empirically demonstrate that VLM can effectively serve its memory encoder, capable of converting multimodal knowledge into compact continuous embeddings. Building on this insight, we develop data- and parameter-efficient method fine-tune the VLM as continuous memory encoder. Specifically, by updating only 1.2% of the model's parameters using just 15.6k self- synthesized samples, the resulting memory module can encode diverse multimodal and multilingual knowledge into merely 8 continuous embeddings. Importantly, since the VLM remains unchanged during inference, our memory module can be seamlessly integrated or detached as needed. Extensive evaluations across six English and two multilingual vision-reast resionnreasonig benchmarks demonstrate the effectiveness and versatility of our approach.
In future work, we plan to extend our approach to a wider range of complex reasoning and planning tasks. Additionally, we aim to integrate the continuous memory mechanism into multimodal agents and evaluate its effectiveness in facilitating knowledge transfer across multiple language and vision- language models.
References
[1] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 1(2), 2023.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:17771190,, 2020.
[3] Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effects on reasoning tasks. arXiv preprint arXiv:2207.0705). 2022.
[4] Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint 200.007402. 2024.
[5] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
[6] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. survey on large language models for code generation. arXiv preprint 1.XX::24060005155. 2024.
10

---

[7] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large lasouala models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292-305, 2023.
[8] Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem? arXiv preprint arXiv:2502. 1501, 2025.
[9] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Angela Fan, Vishrav Chaudhary, Matthias Gallé, Veselin Stoyanov, and Wen-tau Yih. Retrieval-augmented generation for mhowledge-intensiin nlp tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[10] Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07774, 2023.
[11] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wa Wei Chang. Realm: Retrieval-augmented language model pre-training. In International Conference Machine Learning (ICML), 2020.
[12] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 1392:33300..39233 2025.
[13] Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based eetrieall-augmeee generation multi-modality documents. arXiv preprint arXiv:2410.10594, 2024.
[14] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large lisiun-language models. European Conference on Computer Vision, pages 19-35. Springer, 2024.
[15] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, al. Sparsevim: Visual token sparsification for efficient vision-language model inference. arXiv preprint urXiv:241000441 2024.
[16] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hattield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
https://transforrmm circuits
index..ddxxe///// html
[17] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dode Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. arXiv preprint arXiv:2209. 2022.
[18] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
[19] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.147933 2021.
11

---

[20] Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICMI.), 2017.
[21] David Grangier and Dan Iter. The trade-off's of domain adaptation for neural language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 3802- 3813, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 03762, 2017.
[23] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04344 2019.
[24] Edward J Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0040 106.09685, 2021.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 2023.
[26] Yibin Yan and Weidi Xie. Echosight: Advancing visual-language models with wiki knowledge. arXiv preprint 1rXiv:2407.1273:: 2024. Accepted at EMNLP 2024 Findings.
[27] Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Augmenting multimodal Iims with self-reflective tokens for knowledge-basse based visual question answering. arXiv preprint arXiv:2411.66800 2024. Accepted CVPR 2025.
[28] Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang. and Lifu Huang. RORA-VLM: Robust retrieval augmentation for vision language models. arXiv preprint 24X1::2400.08876, 2024.
[29] Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao, xrag: Extreme context compression for serieval-augmented generation with one token. arXiv preprint arXiv: 13792, 2024.
[30] Vivek Chari, Guanghui Qin, and Benjamin Van Durme. Kv-distill: Nearly lossless learnable context compression for llms. ArXiv, 0tts/2503.103377 2025.
[31] Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, and Yansong Tang. Voco-llama: Towards vision compression with large language models. arXiv preprint arXiv:2406. 12275, 2024.
[32] Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. MA-LMM: Memory-augmented large multimodal model for long-term video understanding. arXiv preprint arxiv:2400..05726, 2024.
[33] Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, and Zexue He. M+: Extending memorylim with scalable long-term memory. arXiv preprint arXiv:2502.00592 2025.
[34] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G Patil, lon Stoica, and Joseph E Gonzalez. Memgpt: Towards Ilms as operating systems. arXiv preprint 10.0002300.08560, 2024.
[35] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? arXiv preprint arXiv:2302. / 2023.
[36] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEFFleyf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
12

---

[37] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146-162. Springer, 2022.
[38] Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Comnaa, Lorenzo Baraldi, and Rita Cucchiara. Wiki-LLaVA: Hierarchical strieval-augmeene generation for multimodal llms. arXiv preprint arXiv:2404.15400 2024.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PmLR, 2021.
[40] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-basee image text dataset for multimodal multilingual machine learning. In Pro- ceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 2443-2449, 2021.
[41] Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3113-3124, 2023.
[42] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wos Wei Chang. Open-domain visual entity recogmition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12065-12075, 2023.
[43] Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng. Mrag-bench: Vision-centric evaluation for retrieval-augmenttd multimodal models. arXiv preprint arXiv:2410.08182, 2024.
[44] Paul Lemerr Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-based visual question answering about named entities. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 3108-3120, 2022.
[45] David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. CVQA: Culvurally-diverse multilingual visual question answering benchmark. arXiv preprini arXiv:2406.00967, 2024.
[46] OpenAl. Gpt-4o technical report, 2024. Accessed: 2025-05-10.
[47] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 230.::2300408485, 2023.
[48] LLaVA-VL Team. Llava-next: Open large multimodal models. https://githunn com/
LLVVA-VLLLLaVA-
2024. Accessed: 2025-05-10.
[49] Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang. Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang. Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternIm- xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.00 0333000 2024.
[50] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. oplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04488 2024.
[51] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing language model's perception of the world at any resolution. arXiv preprint 2mXiv:2409.12191 2024.
13

---

[52] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:24/2. 15115, 2024.
[53] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PmLR, 2020.
[54] Brandon McKinzie, Zhe Gan, Jean- Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Anton Belyi, et al. Mml: methods, analysis and insights from multimodal IIm pre-training. In European Conference on Computer Vision, pages 304-323. Springer, 2024.
[55] Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, Roberto Martín- Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 27091-27101, 2024.
[56] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296 26306, 2024.
[57] Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, and Ji-Rong Wen. Less is more: Data value estimation for visual instruction tuning. CoRR, 2024.
[58] Jesse Mu, Xiang Li, and Noah Goodman. Learning compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36: 19327-195,,,,, 2023.
[59] Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, and Enhong Chen. In- context former: Lightning-fast compressing context for large language model. arXiv preprint arXiv:2406. 13618, 2024.
[60] Cangqing Wang. Yutian Yang. Ruisi Li, Dan Sun, Ruicong Cai, Yuzhu Zhang, and Chengqian Fu. Adapting Iims for efficient context processing through soft prompt compression. In Proceedings of the International Conference Modeling, Natural Language Processing and Machine Learning, pages 91-97, 2024.
[61] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-389 2009.
[62] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre- trainrd vision and language models answer visual information-seekknn questions? arXiv preprint arXiv:2302.11713 2023.
[63] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects context. arXiv preprint arXiv: 0312 2014.
[64] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wa Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint 2xXiv:23100022 10,02255, 2023.
[65] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi-discipiie multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.165000 2023.
14

---

A Benchmark Details
InfoSeek InfoSeek is a visual question answering (VQA) dataset tailored for cntermatiion-eekiiie questions that cannot be answered with only common sense knowledge. It combines human-annotated and automatically collected data from visual entity recognition datasets and Wikidata, providing over one million examples for model fine-tuning and validation [35]. For InfoSeek, the ground truth answers for test sets are not publicly available, so we follow prior work [38] 22, [7] and report results
on the validation sets. These sets include questions not seen during training and those associated with unseen entities.
OVEN OVEN (Open-domain Visual Entity Recognition) challenges models select among six million possible Wikipedia entities, making it a general visual recognition benchmark with the largest number of labels. is constructed by re-purposing existing datasets with all labels grounded onto one single label space: Wikipedia entities [42]. Similar with Infoseek, the ground truth ans unswers for the test sets of OVEN are not publicly available, so we also report results on the validation sets.
MRAG-Bench MRAG-Bench is multimodal retrieval- -aggmeete generation benchmark designed to evaluate the performance large vision-language models (LVLMs) in scenarios where visual knowledge retrieval is more beneficial than textual information. It consists of 16,130 images and 1,353
multiple-choice questions across nine distinct scenarios 43
OK-VQA OK-VQA includes more than 14,000 open-ended questions that require external knowl- edge to answer. The dataset is manually filtered to ensure all questions necessitate information beyond the image content, such as from Wikipedia [36]
A-OKVQA A-OKVQA is a crowdsourced visual question answering dataset composed of approxi- mately 25,000 questions requiring a broad base of commonsense and world knowledge to answer. Unlike existing knowledge-based VQA datasets, the questions generally cannot be answered by simply querying a knowledge base and instead require some form of commonsense reasoning about the scene depicted in the image [37.
ViQuAE ViQuAE a dataset focusing on nholledge-based visual question answering about named entities. It covers wide range of entity types, such persons, landmarks, and products, and evaluates models' abilities to ground visual content with knowledge base information [44].
CVQA CVQA (Culturally-diverse Multilingual Visual Question Answering) dataset a benchmark that offers a broad, inclusive representation by incorporating culturally images and questions from a wide range of countries and languages|459 45). In this study, we evaluate five of the most widely used languages in CVQA: Chinese, Russian, Spanish, Portuguese, and Bulgarian.
For all benchmarks, we follow the official evaluation protocols compute the accuracy of the model's responses. Specifically: (1) For InfoSeek, OK-VQA, A-OK and ViQuAE, we use exact match evaluation verify whether the model's 's response exactly matches the ground-truth answers. (2) For OVEN, we adopt the official evaluation script, which uses BM25 [6 to match the model's 's answer with relevant Wikipedia entities. (3) For MRAG-Bench and CVQA, which are in multiple-choice format, we evaluate accuracy by checking whether the model selects the correct option.
B Implementation Details
Knowledge Retrieval Our knowledge base is constructed using the Wikipedia-basee Image- Text (WIT) 4ot which consists of 37.5 million curated image-text pairs from Wikipedia articles across 108 languages. Based on WIT knowledge base, we implement Case-based image-to-ines to-image retrieval system identify the most relevant external knowledge. Following the stage- 1 retrieval methodology of RoRA 28), we first encode all images in WIT using a frozen CLIP image encoder 39 to build dense vector-search database. Given query image I, its CLIP embedding CLI is compared against all vectors in the knowledge base via cosine similarity, followed by softmax normalization over the similarity scores. The image retriever then returns the top-k highest-scoring images along with their associated textual descriptions.
15

---

Memory Encoding Given the retrieved image-text pairs, we employ memory encoder, consisting of VLM and Q-Former to compress multimodal information. For Each image-text pair compressed into an 8-token vector. These token vectors are then concatenated and passed into the inference-time model. For Qwen2t5-Instruct VI., we uses Qwen2.5- VI. as both the inference-time model and the memory encoder, and for Qwen2-Instruct VI., we uses Qwen2-Instruct VL as both the inference-time model and the memory encoder.
Answer Generation The concatenated compressed tokens are plug into the inference-time model to generate answers. We should note that our compression module model model-aggostee allowing the memory encoder to be plugged into other LMs. This flexibility is further demonstrated in Section 4.3
C Ablation Study on Embedding Size
Table 6: Performance on Infoseek of Different Embedding Sizes
Metric
4-Emb
8-Emb
16-Emb
24-Emb
Unseen Question Score
29.32%
32.80 %
31.95% 30.03%
31.37%
Unseen Entity Score
29.64%
32.33% %
30.83%
Final Score
29.48%
32.74%
30.96%
31.10%
To identify the optimal memory size for CoMEM, we conduct an ablation on the number continuous embeddings (4, 8, 16, 24) using the InfoSeek [62] benchmark. As shown in Table 6 using 8 embeddings yields the best overall performance (32.74%), outperforming both smaller and larger configurations. Fewer embeddings underrepresent complex multimodal knowledge, while more embeddings introduce redundancy and noise. This result confirms that embeddings offer an effective trade-off between expressiveness and efficiency, supporting design choice in CoMEM
D Method Generalizability
D.1 Evaluation Image Captioning
we evaluate the generalizability of CoMEM on a caption generation task using the COCO 2014 dataset (63). We randomly sampled 100 image-caption pairs from the test set and used CLIP to retrieve relevant image-caption pairs from the training set. We then compared three setups: (1) Original model (no retrieval); (2) RAG-style retrieval (retrieved image-caption pairs prepended in prompt); (3) Our method (retrieved image-caption pairs encoded into continuous memory). We tested both Qwen2.5 5 and Qwen2, and report standard captioning metrics below in Table 7 We obtain the following conclusions:
Better precision and fluency: BLEU-1 BLEU-4 measure n-gram precision; METEOR balances precision and recall with synonym matching; higher scores indicate better word accuracy. CoMEM consistently improves BLEU-1, BLEU-4, and METEOR across both models, indicating more fluent and accurate captions.
Substantial gain in content relevance: CIDEr scores use tf-idf weighted n-grams to measure relevance to ground-truth captions. It's increased significantly in Qwen2.5-VI. with CoMEM (from 0.24 + 0.64), showing better content relevance to ground- -ruth captions.
Improved semantic similarity: ROUGE-L captures the longest common subsequence with the ref- erence, reflecting surface-level overlap. BERTScore-F computes semantic similarity using contextual embeddings. Memory-augmentt models generate captions with high ROUGE-L and BERTS scores, showing stronger lexical overlap and semantic similarity.
These results confirm that CoMEM generalizes well beyond QA tasks, offering effective and lightweight memory mechanism for open-ended generation tasks like image captioning.
D.2 Evaluation Reasoning Tasks
16

---

Table Evaluation Metrics for Qwen2.5-VL and Qwen2- VL on COCO Captioning task
Model
Method
BLEU-1
BLEU4
METEOR
CIDEr
ROUGE-I
BERTScore-F
Trigram Diversity
Original
0.26
0.04
0.18
0.24
0.26
0,81
0.92
qwen2.5-VL.
+RAG
0.05
+CoMEM
0.34
0.07
0.21
0.64
0.32
0.81
0.86
Original
0.34
0.08
0.21
0.79
0.34
0.81
0.84
qwen2 VI.
+RAG
0.34
0.07
0.64
0.32
0.81
0.83
+oMEM
0.36
0.08
0.23
0.76
0.32
0.82
0.88
Table 8: Performance Comparison Reasoning Task MMMU
Metriiss
Method
Accounting
Architecture
Clinical
Computer
Ecunomics
Electronics
Manugement
Materials
Pharmacy
Q.cn2.5-VL
Accuncy
-RAG
23.3
40.0
33.3
13.3
33.3
+CoMEM
40.0
26.7
43.3
40.0
46.7
30.0
36.7
33.3
40.0
Qwen2.5 VL
26.7
26.7
43.3
26.7
46.7
26.7
40.0
30.0
30.0
Similarit
+RAG
23.3
26.7
40.0
33.3
33.3
16.7
36.7
33.3
40.0
.CoMEM
40.0
33.3
43.3
40.1
50.0
30,0
43.3
33.3
43.3
In recent LLM/MLLM reasoning studies, various reasoning-heavy benchmarks such as Math Vista [64] and MMMU [65] have been widely adopted. It would be interest- ing to see whether the proposed method can effectively handle reasoning tasks on these benchmarks.
Table 9: Performance Comparison Reasoning Task Math Vista
Metrics
Method
Math
Textbook
Visual QA
Figure QA
QwenZ.5-VI.
Accuracy
+RAG
63
53
37
54
+CoMEM
65
46
39
56
Qwen2.5.VL.
Concretely, we split the dataset into evalu- ation set and retrieval set, and retrieve rel- evant image-text pairs using CLIP in RAG and Memory settings. We calculate accuracy with exact matching and similarity with overlapping ratio between predicted and ground truth tokens.
Similarity
+RAG
57
37
56
+CoMEM
66
19
41
57
Our results show that CoMEM consistently improves performance across diverse domains:
On MMMU (Table B our method consistently outperforms both the Original and RAG baselines across most domains, especially in domains like Accounting, Computer Science, Pharmacy, Electron- ics, etc. On Math Vista (Table we also observe improvements in different types of reasoning- heavy tasks. These results demonstrate that our proposed method effectively enhances accuracy complex reasoning scenarios across domains, showing the robustness and generalizability of our method.
E
Method Cost
E.1
Inference Latency Test
To test the latency of baselines and our method, we evaluated the throughput (tokens per second) for each model on 100 random sampled instances. The higher throughput indicates faster inference speed. All experiments were conducted on a single NVIDIA H100 and the results are summarized in Table I0
The findings show that CoMEM maintains competitive inference speed the original model, because it does not significantly increase input length. For RAG, as it expands the input by up to 15x and causes a proportional increase in attention cost, the inference speed decreases lot. Since our CoMEM adds less than 100 continuous memory tokens, this keeps inference cost close to that of the base model while improving accuracy.
F.2 Token Cost Comparison
Continuous embeddings better support high compression, continuous embeddings can densely encode the information, which prevents the great increase of context length (see Table is rather helpful for VLMs to read and understand massive multimodal data. For the example of RAG scenario,
17

---

Table 10: Inference Latency Evaluation accross different benchmarks
Token per Second
InfoSeek
OVEN
MRAG
OKVQA
AOKVQA
viQuAE
Qwen2 models
Qwen2-V VI.
55.93
100.91
39.54
45.81
53.38
99.53
Qwen2- -
40.71
49.27
8.38
34.66
23.60
37.50
48.93
53.29
18.61
40.42
51.60
41.50
Qwen2.5 models
Qwen2.5-VL
53.44
72.71
42.13
45.64
42.87
53,49
Owen2.5-VL+RAG
41.17
42.04
15.89
32.22
27.93
39.61
Owen2.5-VL+CoMENM
51.05
54.39
19.79
42.86
61.41
48.81
Table 11: Token Number and Accuracy Across Datasets
Token Number (Accuracy %)
Infoseek
Oven
MRAG
OKVQA
AOKVQA
VIQuAE
Qwen2-VL VL
357.6 (17.8)
408 (25.5)
(39.3)
404.8 (36.3)
418.4 (41.8)
424.7 (34.5)
Swen2-VI.+RANe
6163.8 (19.0)
6481.5 (24.7)
5882.4 (40.4)
6803.8 (41.9)
(715.4445.33
6943.2 (33.6)
Qwen2-VL 10-22-LL
416.8 (33.1)
492.9 (30.5)
176.9 (35.1)
453.8 (57.7)
478.3 (60.6)
489.2 (36.3)
here we list the context length in three settings: Original model(no retrieval) Qwen2-VL+RAG and our CoMEM method. We can clearly see that although we only retrieve 10 examples in the context, the context length increases by approximately 15x when using RAG. It will cause the VLM suffer from the long context understanding problem. In contrast, our method converts the retrieved examples into continuous embeddings, which leads to fewer than 100 additional embeddings to the input sequence. It avoids the long context understanding problem and also reduces the cost of encoding such long input sequences.
F Training Efficiency
To evaluate the training efficiency of our method, we assess the performance
Infoseek
of CoMEM on Qwen2.5-VL using the Infoseck benchmark under varying amounts of training data and trainable parameters. In the original setting, we use only 15.6k training samples and fine- tune 1.2% of the total parameters. For the data variation setting, we scale the training data by factors of 0.25x, 0.5x, 2x, and 4x. For the parameter variation setting, we adjust the LoRA rank and the number of Q-Former layers by the same scaling factors to control the num- ber of trainable parameters.
Training Settings
Unseen-Q
Unseen-E
All
Original
32.8
28.5
30.7
4x
34.8
28.4
31.3
2x
32.2
29.8
30.9
Data
0.5x
26.5
24.4
25.4
0.25x
17.8
17.5
17.6
4x
26.4
22.1
24.1
2x
28.6
24.8
26.3
Parameters
0.5x
27.8
24.7
26.1
0.25x
23.1
20.3
21.6
Table 12: Performance of CoMEM on Qwen2.5-VV -L under different training data and parameter settings.
As shown in Table 12 increasing the training data by 2x even 4x results only marginal performance gains, suggesting that the original data size is already adequate for effective training. Similarly, increasing the number of trainable parameters does not yield improvements, while reducing them below the original configuration leads notable drop in performance. These findings highlight that our training recipe both data- and parameter-efficien, achieving strong results with minimal resource expenditure.
18

---

G Limitations
Evaluation Benchmarks While we evaluate our method on multimodal and 2 multilingual reasoning tasks, most of benchmarks are static and synthetic. Real-world applications with dynamic or noisy inputs (e.g., web data, live video) may introduce challenges.
Multi-Agent Settings Our current framework is designed and evaluated a single-model setting, where inference language model uses the continuous memory module for enhanced reasoning. However, many real-world applications involve multiple collaborating agents or a combination of LMs and VLMSs. Whether continuous memory can effectively transmit and share knowledge across multiple models remains unexplored and will be investigated in future work.
H
Case Study
Question
Retrieved Information
Answer
Q: Whom was this building officiallly opened by?
Ragers Stirk Horbour Pertners: Esto lista contiere los proyectos del estudlo desde su fundoción en 1977 hasta la actualiddd. hond les trabajes prewes de Richard Rogers, Team Atchard and Regers y Pone Rogers, vlose articulo Richord Ropers Birhord Ropers Portnership Edificie Llayd's. Lendres. Reino Unide (1978-1984) Fabrica Fleetguard, Qumper, Francis (1979-1981)
Qwen2.5:
Nigel Lawson X
Qwen2,5+RAG:
No information available X
Owea2.5:CoME.M:
Reasoning Process:
Richard Rogersi RiEardas Diardlas Rediirsss (ongl. George Regars, g. 1933 liepss Floreriijjje) joje). passulints carso britu architokres
The building in the image is
Llayd's Building, which
bidbiaaa Britanija, makts Anchitokttres jss molykloje...
tevels
was officially opened by Queen Elizabeth II on 1986. Final answer:
L'edificle Llayd's eretho rethongolore ouattordici plem struttura in cemento, correponnde lunghezza...
consiste
langhezza
boxed (Queen Ellizabeth II]
Question
Retrieved Information
Answer
Q: what was the date this
into
The Beeing 780s proved to success airline the tronsstaantic routes, To supplement these. Aer Lingus
On 4pril 1958, Acrlinte Eiream opercted its first tronsctlantic
Sharnon New York...
Qwen2.5:
1959 X
Qwen2.S+RAG:
delivery of first longer Beeiing
Qwen2.5+CoMEME
Scab 90 Scandia: Soab 90 Scandia oli reisllernkk mida toot Seab, Kuna 1944, assital hakits selgkks sasmo, sojet eyev (Teine masilnssidd) Eurospes haikab Ejpens, siis leids Stub, kes al seri pohiliseit tootnud militcarvanv vorustust, et poob oma tootevelikur lslendama. Juhstus otsustus tocto...
Reasoming Process:
siscraft the image
Boeing 707. The Boeing 707 entered into service on February 1958, Final answer: (boxod(1958)
Hisstoyy of United Airlines: United sought
the 1960s. but the Transpocific Route Case (1969) deried them did
until 1983, when they flights to Tokyo from Portlond and Seattle..
Figure 5: Case studies comparing CoMEM with baseline model and model with RAG.
In this appendíx, present qualitative case study demonstrate the effectiveness of our proposed model. Given a question and a corresponding query image, our pipeline first retrieves the top 10 relevant image-text pairs from the WIT knowledge base to provide rich contextual information. Due to space constraints, we only display three representative retrieved pairs for each example in this appendix. We then compare the performance of our CoMEM model against standalone Qwen2.5-VL. and a baseline retrieval-augmented generation (RAG) model. CoMEM can effectively capture key information from retrieved supporting texts, even when the exact answer is not explicitly provided, and perform reasoning to derive the correct answer.
baselines: the
These case studies demonstrate that CoMEM is able to generate accurate answers in challenging scenarios where baseline models either fail or return incomplete information. This highlights CoMEM's ability effectively encode and leverage complex multimodal and multilingual knowledge, leading to stronger performance in advanced reasoning tasks.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly and accurately reflect the paper's main contributions and scope. The proposed method uses VLM itself as continuous memory encoder with minimal fine-tuning effort, which addresses the issues exist in existing RAG- based and experiment, which is shown in
models. Our approach is validated by the result of massive
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not include theoretical results this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all of our experiment details in 4.1 and Appendix B The information provided in these two sections is enough for reproducing the results. All of our code and data will be released.
Guidelines:
The answer NA means that the paper does not include experiments. the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
21

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will release code, data, and model checkpoints for review, and open- source them upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We disclose all of our experiment details and Appendix B The information provided in these two sections is enough for reproducing the results. of our code and data will be released.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the
the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not include statistical significance results, we have achieved apparent and great improvements, and there no need for significance
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We clearly disclose our experiment settings and implementation details in Section 3.3 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:(Yes
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that our research complies with outlined principles.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: This work proposes an advanced method for improving LLM's performace,
and is positive for the improvement for the whole society. We discuss the impact of our work in the Introduction | Discussion 3.4 and Further Analysis part 4.3
23

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our work does not involve the release of new models or datasets with high risk of misuse. All models and datasets used publicly available, widely adopted in the community, and considered safe under their respective usage policies.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models used in this work, such as InfoSeek, OK-VQA EVQA, WIT, CLIP, Qwen2-VI. and Qwen2.5-VI. are publicly available and used in accordance with their respective licenses. Appropriate citations are provided for each asset, and no proprietary restricted data or models were used beyond their permitted scope.
Guidelines:
The answer NA means that the paper does not existing assets.

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All of the new assets introduced in the paper, including the code and data, is well documented and will be published.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
This work introduces novel approach that uses Vision- Vision-Language Models (VLMs), specifically Qwen2.5-VL and Qwen2-VL, as continuous memory encoders for multimodal reasoning. The VLM itself is used to synthesize training data and generate memory embeddings. For more details, We disclose all of our experiment settings 4.1 and Appendix B
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve L.LMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
26


Reviews:
[{"review_id": "vMn4QSykvC", "reviewer": "Reviewer_hjMA", "summary": "The paper proposes CoMEM, a continuous memory mechanism for vision-language models (VLMs), using the VLM itself to encode multimodal and multilingual knowledge into compact embeddings. It outperforms traditional RAG and token pruning in tasks like InfoSeek and OKVQA, enabling efficient, plug-and-play integration with strong performance in long-context and low-resource language settings.", "questions": "1. Why do continuous embeddings work better than discrete tokens?\nThe results show clear advantages but lack explanation. Is it because of higher compression efficiency or better compatibility with the VLM's attention mechanism? A simple theoretical analysis or comparison of different compression rates (e.g., 8-dim vs 16-dim embeddings) would make this more convincing.\n\n1. Does this work only for QA tasks?\nAll current experiments are on question answering. Could you test it on generation tasks like image captioning? Even one small experiment on COCO captioning would prove the method's generalizability.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n- **Strong Empirical Results**: Demonstrates consistent improvements across eight multimodal benchmarks, with significant gains over baselines.  \n- **Efficient Design**: Uses only 1.2% additional parameters and a small self-synthesized dataset, making it practical for deployment.  \n- **Novel Approach**: Introduces the idea of a VLM serving as its own memory encoder, avoiding costly alignment training and enabling plug-and-play integration.  \n- **Scalability**: Handles long-context inputs robustly, unlike RAG methods that degrade with more retrieved items.  \n\n## Weaknesses\n- **Limited Theoretical Justification**: Lacks analysis of why continuous embeddings outperform discrete tokens or how compression affects performance.  \n- **Narrow Benchmarking**: Focuses heavily on VQA tasks; testing on broader tasks (e.g., captioning) would strengthen claims of generalizability.", "was_revised": true, "final_justification": "The rebuttal effectively addressed my concerns. The authors provided both intuitive explanations and ablation studies (e.g., varying memory size) to clarify the benefits of continuous embeddings over discrete tokens. They also extended their evaluation beyond QA tasks with a caption generation experiment on COCO, showing consistent improvements across several standard metrics.\n\nThese additions strengthen the paper's clarity, empirical support, and generalizability. Given the overall contribution, the practical value of the proposed mechanism, and the quality of the response, I have raised my rating from 4 to 5.", "rebuttal": "## Rebuttal\n\nWe sincerely thank the reviewer for the insightful comments and appreciate the positive feedback! We reply to all the concerns raised in the weaknesses and questions part.\n### **Weakness-1: Limited Theoretical Justification**\n> Lacks analysis of why continuous embeddings outperform discrete tokens or how compression affects performance. \n\n**Reply-to-Weakness1-A**: \n\nFor continuous embeddings and discrete tokens, it is hard to propose a proof that conclusively establishes the superiority of either approach. Instead, we demonstrate the intuitive and practical advantages of continuous embeddings from several perspectives.\n- **Conceptual alignment with human cognition**: Cognitive‑science research shows that humans store and manipulate knowledge at the level of abstract concepts, not at the level of exact pixel patterns or phoneme sequences. When you think of “the Great Wall” or “the Eiffel Tower,” you effortlessly summon a coarse, high‑level mental image while discarding countless low‑level details such as the precise arrangement of every brick or rivet. Therefore, the continuous embedding based memory better mirrors the way our brains compress and retrieve multimodal information.\n- **Continuous embeddings better support high compression**: as continuous embeddings can densely encode the information, which prevents the great increase of context length (see Table 2). It is rather helpful for VLMs to read and understand massive multimodal data. For the example of RAG scenario, here we list the context length in three settings: Original model(no retrieval) , Qwen2-VL+RAG and our CoMEM method.  We can clearly see that although we only retrieve 10 examples in the context, the context length increases by approximately **15×** when using RAG. It will cause the VLM to suffer from the long context understanding problem. In contrast, our method converts the retrieved examples into continuous embeddings, which leads to fewer than 100 additional embeddings to the input sequence. It avoids the long context understanding problem and also reduces the cost of encoding such long input sequences.\n- **Better empirical results of continuous embedding**: since continuous embeddings generally have higher compression rate, their rather short context length allows the model to process multimodal information more effectively. In our method, we jointly compress both image and text knowledge into a unified continuous latent space, which enables better cross‑modal integration and achieves superior performance on most benchmarks compared to state‑of‑the‑art methods. We also add results comparison with up-to-date RAG methods. We also added comparisons with up‑to‑date RAG baselines, and CoMEM consistently delivers the best results across all benchmarks:\n\n**Table 1: Performance Comparison Across Datasets**\n\n| Model              | infoseek(Q) | infoseek(E) | OVEN  | MRAG  | OKVQA | AOKVQA | ViQUAE |\n|--------------------|-------------|-------------|--------|--------|--------|---------|--------|\n| Wiki-LLaVA         | 28.6        | 25.7        | –      | –      | –      | –       | –      |\n| RORA-VLM           | 27.3        | 25.1        | 15.5   | –      | –      | –       | –      |\n| REVEAL             | –           | –           | –      | –      | 59.1   | 52.2    | –      |\n| Echosight          | 18.0        | 19.8        | 27.0   | 41.3   | 20.0   | 16.9    | 25.2   |\n| ReflectiVA         | 28.6        | 28.1        | 20.6   | 39.7   | 47.5   | 47.6    | 29.8   |\n| Qwen2+CoMEM        | 32.6        | 33.1        | 23.6   | 35.1   | 57.7   | 60.6    | 36.3   |\n| Qwen2.5+CoMEM      | 32.8        | 28.5        | 20.8   | 38.1   | 47.6   | 55.0    | 34.7   |\n\n\n**Table 2: Token Number and Accuracy Across Datasets**\n| **Token Number (Accuracy)** | **Infoseek**       | **Oven**          | **MRAG**          | **OKVQA**         | **AOKVQA**        | **ViQuAE**        |\n|-----------------------------|--------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n| **Qwen2-VL**                | 357.6 (17.8)      | 408.6 (25.5)     | 84.5 (39.3)      | 404.8 (36.3)      | 418.4 (41.8)      | 424.7 (34.5)      |\n| **Qwen2-VL+RAG**            | 6163.8 (19.0)     | 6481.5 (24.7)    | 5882.4 (40.4)    | 6803.8 (41.9)     | 6715.4 (45.3)     | 6943.2 (33.6)     |\n| **Qwen2-VL+COMEM**          | 416.8 (33.1)       | 492.9 (30.5)      | 176.9 (35.1)      | 453.8 (57.7)      | 478.3 (60.6)      | 489.2 (36.3)     |\n\n**Reply-to-Weakness1-B**: \n\nComparison of different compression rates: We conducted an ablation study by varying the number of memory embeddings (4, 8, 16, 24) on the InfoSeek benchmark, to study the effectiveness of different compression rates. As shown in Table 3 below, 8 embeddings consistently achieve the best overall performance across both unseen question and unseen entity settings. \n  - 4 embeddings do not perform well due to limited capacity—it is difficult to encode complex multimodal and multilingual knowledge with such a compact representation.\n  - 16 and 24 embeddings perform slightly worse than 8. While they offer more representational capacity, they also introduce redundancy and noise, making them harder to train effectively. These longer embeddings require more data to properly align with the frozen inference model, reducing training efficiency.\n  - 8 embeddings strike an optimal balance between compactness and expressiveness. They are sufficient to capture relevant knowledge while remaining lightweight, easy to integrate, and efficient to train.\n\nThese results suggest that 8 embeddings represent an effective inflection point, supporting our design choice in CoMEM and emphasizing the importance of tuning memory size for both performance and training efficiency.\n\n**Table 3: Performance Across Different Embedding Sizes on Infoseek**\n|                         | 4-Embedding | 8-Embedding | 16-Embedding | 24-Embedding |\n|-------------------------|-------------|-------------|--------------|--------------|\n| Unseen Question Score   | 29.32       | 32.80        | 31.95        | 31.37        |\n| Unseen Entity Score     | 29.64       | 32.33       | 30.03        | 30.83        |\n| Final Score             | 29.48       | 32.74       | 30.96        | 31.10         |\n\n### **Weakness-2: Narrow Benchmarking**\n> Focuses heavily on VQA tasks; testing on broader tasks (e.g., captioning) would strengthen claims of generalizability.\n\n**Reply-to-Weakness2**: \n\nFollowing the suggestion of the reviewer, we evaluate the generalizability of CoMEM on a caption generation task using the COCO 2014 dataset. We randomly sampled **100 image-caption pairs** from the test set and used CLIP to retrieve relevant image-caption pairs from the training set. We then compared three setups: (1) Original model (no retrieval); (2) RAG-style retrieval (retrieved image-caption pairs prepended in prompt); (3) Our method (retrieved image-caption pairs encoded into continuous memory). We tested both Qwen2.5 and Qwen2, and report standard captioning metrics below in Table 4. We obtain the following conclusions:\n  - **Better precision and fluency**: BLEU-1 / BLEU-4 measure n-gram precision; METEOR balances precision and recall with synonym matching; higher scores indicate better word accuracy. CoMEM consistently improves BLEU-1, BLEU-4, and METEOR across both models, indicating more fluent and accurate captions.\n  - **Substantial gain in content relevance**: CIDEr scores use tf-idf weighted n-grams to measure relevance to ground-truth captions. It's increased significantly in Qwen2.5-VL with CoMEM (from 0.24 → 0.64), showing better content relevance to ground-truth captions.\n  - **Improved semantic similarity**: ROUGE-L captures the longest common subsequence with the reference, reflecting surface-level overlap. BERTScore-F computes semantic similarity using contextual embeddings. Memory-augmented models generate captions with high ROUGE-L and BERTScore-F scores, showing stronger lexical overlap and semantic similarity.\n\nThese results confirm that CoMEM generalizes well beyond QA tasks, offering an effective and lightweight memory mechanism for open-ended generation tasks like image captioning.\n\n**Table 4: Evaluation Metrics for Qwen2.5-VL and Qwen2-VL on COCO Captioning task**\n| Model | Method  | BLEU-1 | BLEU-4 | METEOR | CIDEr | ROUGE-L | BERTScore-F | Trigram Diversity |\n|------|-------|------|------|--------|--------|----------|--------------|--------------------|\n| qwen2.5-VL    | Original| 0.26   | 0.04   | 0.18   | 0.24   | 0.26     | 0.81         | 0.92               |\n|            | +RAG     | 0.30   | 0.05   | 0.20   | 0.45   | 0.27     | 0.81         | **0.94**           |\n|            | +CoMEM  | **0.34** | **0.07** | **0.21** | **0.64** | **0.32**   | 0.81         | 0.86               |\n| qwen2-VL      | Original| 0.34   | 0.08   | 0.21   | 0.79   | 0.34     | 0.81         | 0.84               |\n|            | +RAG     | 0.34   | 0.07   | 0.20   | 0.64   | 0.32     | 0.81         | 0.83               |\n|            | +CoMEM  | **0.36** | 0.08   | **0.23** | 0.76   | 0.32     | **0.82**     | **0.88**           |\n\n### **Question-1**\n> Why do continuous embeddings work better than discrete tokens? The results show clear advantages but lack explanation. Is it because of higher compression efficiency or better compatibility with the VLM's attention mechanism? A simple theoretical analysis or comparison of different compression rates (e.g., 8-dim vs 16-dim embeddings)  would make this more convincing.\n\n**Reply-to-Question-1**: \n\nPlease refer to Reply-to-Weakness1-A and Reply-to-Weakness1-B\n### **Question-2**\n> Does this work only for QA tasks? All current experiments are on question answering. Could you test it on generation tasks like image captioning? Even one small experiment on COCO captioning would prove the method's generalizability.\n\n**Reply-to-Question-2**: \n\nPlease refer to Reply-to-Weakness2"}, {"review_id": "GjXe05MoMt", "reviewer": "Reviewer_WDJh", "summary": "This paper introduces CoMEM, a plug-and-play continuous memory mechanism for frozen vision-language models that encodes retrieved multimodal context into compact embeddings to bypass token overload. The core method utilizes a lightweight Q-Former to compress image–text retrievals into just eight dense vectors that are prepended to the model’s input. The proposed approach is highly efficient, requiring only 15.6k self-synthesized QA samples and LoRA fine-tuning of 1.2% of parameters while 6–12% improvements on multilingual VQA benchmarks.", "questions": "My main concerns are in the method section. I hope the author could provide more method insights and motivation. See the weakness above.", "limitations": "The limitation discussions are sound and clear to me.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The idea of using the frozen VLM itself as a continuous memory encoder is simple and effective.\n2. It achieves performance improvements with only 1.2% of parameters fine-tuned and 15.6k training data.\n3. The evaluation is broad, spanning six English and two multilingual VQA benchmarks.\n\nWeaknesses:\n1. The paper lacks clarity in its writing, with many terms left unexplained. As a result, it is difficult to follow and understand. For example, Figure 1 does not clearly explain what the attention-based compression rates refer to.\n2. The methodological novelty over prior Q-Former and RAG frameworks is limited and incremental.\n3. The comparison to RAG seems unfair, as many recent studies have significantly improved its performance without substantial cost. I believe the evaluation should include the latest works in RAG, rather than relying solely on a vanilla baseline.\n4. The trade-off between RAG and continuous memory is that RAG can be used in a training-free paradigm, whereas continuous memory typically requires training to adapt to the inference-time VLM which is not properly mentioned in the paper. \n5. The paper lacks an ablation study to justify why using 8 embeddings is sufficient for the memory.\n6. Inference latency of the whole framework is not evaluated.", "was_revised": true, "final_justification": "Thanks for authors' detailed and timely rebuttal. All of my questions have been answered with good answers and extensive qualitative results, and I am happy to raise my score.", "rebuttal": "## Rebuttal - Part 1\n### **Weakness-1: Lacks Clarity in its Writing**\n> The paper lacks clarity in its writing, with many terms left unexplained. For example, Figure 1 does not clearly explain what the attention-based compression rates refer to.\n\n**Reply-to-Weakness1**: \n\nWe appreciate the reviewer’s feedback and will add further clarifications to improve the readability of the paper in all figures and tables. \n\nFirst of all, we define the compression rate as: \n$$\n\\\\mathbf{Compression\\\\ Rate} = \\\\frac{\\\\#\\\\ \\\\text{tokens after compression}} {\\\\#\\\\ \\\\text{tokens before compression}} \\\\times 100\\\\% \n$$\nHere, we provide details for the experiment in Figure 1.\n  - The continuous memory module first encodes each retrieved item into continuous embeddings. We then compute the average attention scores across all layers within the VLM. Embeddings with the highest attention scores are retained with a compression rate, and the rest are discarded (same as described in Section 2.1, Memory Methods - VLM-as-Memory+Attn). \n  - Figure 1 demonstrates that, even with this simple token‑selection strategy, VLM‑as‑Memory methods outperform the baseline at high compression rates. This figure provides the intuition and preliminary evidence for our proposed method; therefore, we kept the explanation brief in the original submission.\n  \n  To further clarify and prevent any possible ambiguities, we provide the following additional explanations:\n  - **Table 1** (in paper): This table compares various training‑free memory methods across three benchmarks (InfoSeek, OK‑VQA, A‑OKVQA). Methods include vanilla RAG, token pruning (FastV), and our VLM‑as‑Memory variants. The table shows that naive RAG can degrade performance due to context length overload, while our VLM‑as‑Memory methods consistently outperform token‑based methods because continuous embeddings align more naturally with the VLM.\n  - **Figure 3** (in paper): This figure analyzes our model’s long‑context robustness. We vary the number of retrieved image‑text pairs (x‑axis) and plot accuracy (y‑axis) for RAG and our method on InfoSeek. RAG performance drops when more than 30 items are retrieved because the increased context length introduces attention dilution and ambiguity. In contrast, our method remains stable across retrieval sizes since the compressed continuous memory does not excessively increase input length, demonstrating better scalability for long‑context scenarios.\n\n### **Weakness-2: Novelty of the framework**\n> The methodological novelty over prior Q-Former and RAG frameworks is limited and incremental.\n\n**Reply-to-Weakness2**: \n\nIn this paper, our target is to validate the effectiveness of using continuous memory for augmenting VLMs. Concretely, Q-Former is the technical way for achieving this goal, while RAG is a typical scenario for evaluating our approach. Follows are more detailed discussion about our contribution.\n  - **Our key contribution is to construct an information-rich and plug-and-play continuous memory with minimal cost.** To achieve this goal, we try to use simple and easy-to-follow techniques to verify whether the continuous memory is effective for improving VLMs. According to our extensive experiments, we validate that we can compress each multimodal item into just 8 continuous embeddings via the VLM itself, and the compressed embeddings are more helpful than SOTA RAG methods to better improve the performance.\n  - **Our design is meaningful and opens a new direction with many possibilities:** \n    - Firstly, it turns a VLM into its own memory encoder and achieves strong semantic alignment without modifying the base model.\n    - Secondly, our approach supports both multimodal and multilingual reasoning, and has shown strong performance across a broad range of tasks—including knowledge-intensive VQA, image captioning, and reasoning tasks—demonstrating its generality. \n| Method      | Knowledge-Intensive VQA (Average)  | Image Captioning (COCO) | Reasoning (MMMU) |\n|-------------|-------|------------------|-----------|\n| Qwen2.5-VL  | 30.8  | 0.39             | 32.22      | \n| +RAG        | 27.6  | 0.43            | 31.48      | \n| +CoMEM      | **35.4** | **0.46**      | **37.41**  | \n    - Thirdly, our memory also acts as a bridge between VLMs and LLMs, enabling the transfer of visual knowledge to pure language models. \n| LLM                        | InfoSeek: Unseen-Q | InfoSeek: Unseen-E | InfoSeek: All | OVEN: Query | OVEN: Entity | OVEN: All | Avg. |\n|---------------------------|--------------------|---------------------|----------------|--------------|---------------|-------------|--------------|\n| Qwen2.5-Instruct          | 5.0                | 4.8                 | 4.9            | 2.4          | 0.1           | 1.3         | 3.1          |\n| Qwen2.5-Instruct + RAG    | 13.4               | 10.3                | 11.9           | 1.8          | 2.7           | 2.2         | 7.0          |\n| Qwen2.5-Instruct + CoMEM  | **29.3**           | **27.4**            | **28.3**       | **6.8**      | **7.7**       | **7.2**     | **17.8**     |\n    - Importantly, our method opens up new possibilities such as grounding LLMs with vision, storing and accessing large-scale memory efficiently, and supporting long-horizon reasoning and planning in agent-based scenarios.\n  - **Our simple and effective designs well prove our concept, and also reduce the cost**: It has low training and inference cost with small inference latency, and the encoded continuous memory can be reused across tasks. We achieve strong performance using only 15.6k self-synthesized training samples and fine-tuning just 1.2% of parameters, making it both data and compute efficient, especially suitable for low-resource or cross-lingual settings. Across benchmarks and scenarios (knowledge-intensive VQA, image captioning, and reasoning tasks), our method consistently outperforms original vanilla RAG, up-to-date RAG baselines, confirming the effectiveness, robustness, and generalizability of our method.\n\n### **Weakness-3 Unfair Comparison to RAG**\n> The evaluation should include the latest works in RAG, rather than relying solely on a vanilla baseline.\n\n**Reply-to-Weakness3**: \n\nThank you for your suggestion. We agree that stronger and more diverse baselines are important for a fair evaluation, and we have selected SOTA ones in our paper for comparison.\n  - **Competitive RAG methods**: In Table 3 (in paper), we have compared our method with four highly relevant and latest multimodal RAG-based methods: Wiki-LLaVA, RORA-VLM, EchoSight, and ReflectiVA. \n  - **Adding More Experiments on RAG Baselines**: To address your concern, we have: (1) Re-implemented EchoSight and ReflectiVA (the only open-sourced methods among the four) and tested them on all benchmarks (MRAG, OKVQA, AOKVQA, ViQUAE) to provide a comprehensive evaluation. (2) Included more relevant RAG-based methods for comparison:\n    - **Wiki-LLaVA**[1]: Uses hierarchical retrieval from Wikipedia to add external knowledge for visual question answering.\n    - **RORA-VLM**[2]: Enhances robustness with two-stage retrieval and adversarial training to filter irrelevant content.\n    - **EchoSight**[3]: Applies a visual-first retrieval followed by text-image reranking for better encyclopedic knowledge integration.\n    - **ReflectiVA**[4]: Uses reflective tokens to decide when and what external knowledge to retrieve.\n    - **REVEAL**[5]: Pre-trains a retrieval-augmented VLM with a large multimodal memory for knowledge-intensive tasks.\n\n    The results are shown in the following Table 1. This extended evaluation demonstrates that CoMEM consistently outperforms open-source multimodal RAG methods (Echosight, ReflectiVA) across tasks, and is competitive with or superior to closed-source systems like RORA-VLM, REVEAL, etc. These results further validate the generality, scalability, and effectiveness of our proposed memory-based method.\n\n**Table 1: Performance Comparison Across Datasets**\n\n| Model              | infoseek(Q) | infoseek(E) | OVEN  | MRAG  | OKVQA | AOKVQA | ViQUAE |\n|--------------------|-------------|-------------|--------|--------|--------|---------|--------|\n| Wiki-LLaVA         | 28.6        | 25.7        | –      | –      | –      | –       | –      |\n| RORA-VLM           | 27.3        | 25.1        | 15.5   | –      | –      | –       | –      |\n| REVEAL             | –           | –           | –      | –      | 59.1   | 52.2    | –      |\n| Echosight          | 18.0        | 19.8        | 27.0   | 41.3   | 20.0   | 16.9    | 25.2   |\n| ReflectiVA         | 28.6        | 28.1        | 20.6   | 39.7   | 47.5   | 47.6    | 29.8   |\n| Qwen2-VL+CoMEM        | 32.6        | 33.1        | 23.6   | 35.1   | 57.7   | 60.6    | 36.3   |\n| Qwen2.5-VL+CoMEM      | 32.8        | 28.5        | 20.8   | 38.1   | 47.6   | 55.0    | 34.7   |\n\n[1] Davide Caffagni et al. Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In CVPR 2024.\n\n[2] Jingyuan Qi et al. RoRA-VLM: Robust Retrieval-Augmented Vision Language Models. In 2024.\n\n[3] Yibin Yan . EchoSight: Advancing Visual-Language Models with Wiki Knowledge. In EMNLP 2024.\n\n[4] Federico Cocchi et al. Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering. In CVPR 2025.\n\n[5] Ziniu Hu et al. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory. In CVPR 2023."}, {"review_id": "DLy1Nruxkw", "reviewer": "Reviewer_W84i", "summary": "This work aims to enhance the capability of vision-language models (VLMs) in acquiring multimodal or multilingual real-world knowledge. To efficiently leverage an external memory system, the paper proposes using continuous memory, i.e., a compact set of dense embeddings, to represent multimodal or multilingual knowledge. Specifically, the authors show that a VLM can serve as its own continuous memory encoder. Based on this observation, the paper introduces a data-efficient and parameter-efficient method to fine-tune the VLM into a memory encoder. Extensive experiments show that the proposed method achieves strong performance with minimal additional parameters.", "questions": "Please see the weakness.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nThe paper provides an empirical study demonstrating that a VLM can serve as a continuous memory encoder to compress multimodal information, and that a few embeddings from the VLM can preserve key information to improve performance on multimodal reasoning tasks.\n\n**Weakness**\n\nI am quite confused about the memory mechanism introduced in this work. From a high-level technical perspective, the proposed framework seems to be a LoRA-based fine-tuning framework with retrieved images/texts. A typical memory system should involve key processes such as information storage, updating, and retrieval. These aspects are not clearly illustrated in the paper, and the current presentation makes the proposed method appear more like a LoRA-based fine-tuning approach rather than a complete memory mechanism.\n\nIn the inference stage, are only the memory embeddings used for making predictions, while the rest of the VLM’s parameters are unused? The claim that “we keep the VLM’s parameters frozen during inference, while continuous memory embeddings are directly used for downstream tasks” is unclear. The authors should provide more details about the inference stage and clarify how the memory contributes to prediction.\n\nThe compared methods are somewhat limited. In the SOTA table, only vanilla RAG is compared. However, various multimodal RAG methods have been studied [a]. The authors should conduct a more comprehensive comparison with other RAG-based approaches to better validate the effectiveness of their method.\n\n[a] Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation.\n\nIn recent LLM/MLLM reasoning studies, various reasoning-heavy benchmarks such as MathVista and MMMU have been widely adopted. It would be interesting to see whether the proposed method can effectively handle reasoning tasks on these benchmarks.\n\nMinor: The authors could consider moving Figure 1 and Table 1 closer to the sections where the relevant concepts are first discussed (e.g., the Introduction), to improve readability.", "was_revised": true, "final_justification": "Thanks for the authors' detailed response. My concerns have been addressed, so I have increased the score to 4.", "rebuttal": "## **Rebuttal - Part 1**\n\nWe thank the reviewer for the detailed and constructive feedback. We respectfully reply to all the concerns raised in the weaknesses and questions section below.\n\n### **Question-1: Unclear Memory Mechanism**\n> The proposed method is presented as a memory system, but key aspects typically associated with memory—such as information storage, updating, and retrieval—are not clearly defined or illustrated. From a high-level perspective, the method appears more like a LoRA-based fine-tuning framework with retrieved image/text inputs, rather than a complete or standalone memory architecture.\n\n**Reply-to-Question1**: \n\nWe appreciate the reviewer’s thoughtful question and would like to clarify our design and contributions as follows:\n  - **Memory Encoder, Not a Full Memory System**: Our work proposes a memory encoder module, rather than a complete memory system. Its primary goal is to encode external knowledge into compact continuous embeddings that can be flexibly integrated into the VLM to enhance its capabilities.\n\n  - **Plug-and-play and Generalizable Memory**: As demonstrated in Table 3 in the paper, the proposed memory encoder is effective across different backbones, such as Qwen2-VL and Qwen2.5-VL. Furthermore, Table 1 below shows that our continuous memory can successfully transfer visual and textual knowledge to text-only LLMs. This indicates that the continuous memory representations are broadly compatible and can effectively preserve transferable multimodal knowledge for augmenting any LLM.\n\n  - **Simple Yet Effective Retrieval**: While our current implementation adopts a CLIP-based retrieval mechanism, consistent with prior work, our method already demonstrates strong performance. We believe that incorporating more advanced retrieval strategies could further enhance the results.\n\n  - **Storage Efficiency**: Our approach offers a significant storage advantage over traditional designs. The following Table 2 shows the average context length for each benchmark with or without our method. It clearly shows that the discrete token format is approximately 15x times larger than the continuous memory format. By storing compact continuous embeddings instead of original images or discrete token sequences, we can substantially reduce memory and storage requirements.\n\n**Table 1: Transferability Study of Vision-Language Memory Encoded by CoMEM on LLMs**\n\n| LLM                        | InfoSeek: Unseen-Q | InfoSeek: Unseen-E | InfoSeek: All | OVEN: Query | OVEN: Entity | OVEN: All | Avg. |\n|---------------------------|--------------------|---------------------|----------------|--------------|---------------|-------------|--------------|\n| Qwen2.5-Instruct          | 5.0                | 4.8                 | 4.9            | 2.4          | 0.1           | 1.3         | 3.1          |\n| Qwen2.5-Instruct + RAG    | 13.4               | 10.3                | 11.9           | 1.8          | 2.7           | 2.2         | 7.0          |\n| Qwen2.5-Instruct + CoMEM  | **29.3**           | **27.4**            | **28.3**       | **6.8**      | **7.7**       | **7.2**     | **17.8**     |\n\n\n**Table 2: Token Number and Accuracy Across Datasets**\n\n| **Token Number (Accuracy)** | **Infoseek**       | **Oven**          | **MRAG**          | **OKVQA**         | **AOKVQA**        | **ViQuAE**        |\n|-----------------------------|--------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n| Qwen2-VL                | 357.6 (17.8)      | 408.6 (25.5)     | 84.5 (39.3)      | 404.8 (36.3)      | 418.4 (41.8)      | 424.7 (34.5)      |\n| +Multimodal Knowledge in Discrete Tokens            | 6163.8 (19.0)     | 6481.5 (24.7)    | 5882.4 (**40.4**)    | 6803.8 (41.9)     | 6715.4 (45.3)     | 6943.2 (33.6)     |\n| +Multimodal Knowledge in Continuous Memory          | 416.8 (**33.1**)       | 492.9 (**30.5**)      | 176.9 (35.1)      | 453.8 (**57.7**)      | 478.3 (**60.6**)      | 489.2 (**36.3**)      |\n\n### **Question-2: Inference Process Ambiguity**\n> It is unclear how the memory embeddings interact with the VLM during inference. The claim that “we keep the VLM’s parameters frozen during inference, while continuous memory embeddings are directly used for downstream tasks” lacks clarity. More detailed explanation of the inference workflow and the contribution of memory in prediction is needed.\n\n**Reply-to-Question2**: \n\nWe apologize for the confusion; there was a typo in the original statement. The correct description is:  “We keep the VLM’s parameters frozen during training, and only train the LoRA adapters and the Q‑Former.” Below we clarify both the training and inference processes:\n  - **Training stage**:\n    - **Trainable Components**: (i) LoRA adapters (low‑rank update matrices) in selected layers of the VLM encoder and (ii) the Q‑Former, which compresses multimodal knowledge items into 8 continuous memory embeddings each.\n    - **Frozen Components**: All original VLM parameters are frozen.\n      \n    This design updates only 1.2% of parameters, following a parameter‑efficient fine‑tuning strategy (Section 3.3, “Parameter-efficient Fine-tuning”). \n  - **Inference stage**: \n    - The trained Q‑Former generates continuous memory embeddings for retrieved multimodal knowledge items. These memory embeddings are then prepended to the input embedding sequence of the frozen VLM(Section 3.2, “Plug‑and‑Play Mechanism”). The VLM then performs normal autoregressive generation, attending to the memory embeddings and the query jointly. \n    - Therefore, the continuous memory is consumed by the VLM’s self‑attention layers during decoding, which provide retrieved knowledge to guide prediction. This design allows the memory module to be plug‑and‑play without altering the base VLM’s parameters. As shown in Table 2 in the paper, this paradigm achieves the lowest training cost among most state‑of‑the‑art RAG models while still delivering superior performance.\n\n### **Question-3: Limited Comparison Baselines**\n> In the SOTA table, only vanilla RAG is compared. The authors should conduct a more comprehensive comparison with other RAG-based approaches to better validate the effectiveness of their method.\n\n**Reply-to-Question3**: \n\nThank you for your suggestion. We agree that stronger and more diverse baselines are important for a fair evaluation, and we have selected SOTA ones in our paper for comparison.\n  - **Competitive RAG methods**: In Table 3 (in paper), we have compared four highly relevant and latest multimodal RAG-based methods: Wiki-LLaVA, RORA-VLM, EchoSight, and ReflectiVA. \n  - **Adding More Experiments on RAG Baselines**: To address your concern, we have: (1) Re-implemented EchoSight and ReflectiVA (the only open-sourced methods among the four) and tested it on all benchmarks (MRAG, OKVQA, AOKVQA, ViQUAE) to provide a comprehensive evaluation. (2) Included more relevant RAG-based methods for comparison:\n    - **Wiki-LLaVA**[1]: Uses hierarchical retrieval from Wikipedia to add external knowledge for visual question answering.\n    - **RORA-VLM**[2]: Enhances robustness with two-stage retrieval and adversarial training to filter irrelevant content.\n    - **EchoSight**[3]: Applies a visual-first retrieval followed by text-image reranking for better encyclopedic knowledge integration.\n    - **ReflectiVA**[4]: Uses reflective tokens to decide when and what external knowledge to retrieve.\n    - **REVEAL**[5]: Pre-trains a retrieval-augmented VLM with a large multimodal memory for knowledge-intensive tasks.\n  \n    The results are shown in the following Table 3. This extended evaluation demonstrates that CoMEM consistently outperforms open-source multimodal RAG methods (Echosight, ReflectiVA) across tasks, and is competitive with or superior to closed-source systems like RORA-VLM, REVEAL, etc. These results further validate the generality, scalability, and effectiveness of our proposed memory-based method.\n\n\n**Table 3: Performance Comparison Across Datasets**\n\n| Model              | infoseek(Q) | infoseek(E) | OVEN  | MRAG  | OKVQA | AOKVQA | ViQUAE |\n|--------------------|-------------|-------------|--------|--------|--------|---------|--------|\n| Wiki-LLaVA         | 28.6        | 25.7        | –      | –      | –      | –       | –      |\n| RORA-VLM           | 27.3        | 25.1        | 15.5   | –      | –      | –       | –      |\n| REVEAL             | –           | –           | –      | –      | 59.1   | 52.2    | –      |\n| Echosight          | 18.0        | 19.8        | 27.0   | 41.3   | 20.0   | 16.9    | 25.2   |\n| ReflectiVA         | 28.6        | 28.1        | 20.6   | 39.7   | 47.5   | 47.6    | 29.8   |\n| Qwen2-VL+CoMEM        | 32.6        | 33.1        | 23.6   | 35.1   | 57.7   | 60.6    | 36.3   |\n| Qwen2.5-VL+CoMEM      | 32.8        | 28.5        | 20.8   | 38.1   | 47.6   | 55.0    | 34.7   |\n\n[1] Davide Caffagni et al. Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In CVPR 2024.\n\n[2] Jingyuan Qi et al. RoRA-VLM: Robust Retrieval-Augmented Vision Language Models. In 2024.\n\n[3] Yibin Yan . EchoSight: Advancing Visual-Language Models with Wiki Knowledge. In EMNLP 2024.\n\n[4] Federico Cocchi et al. Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering. In CVPR 2025.\n\n[5] Ziniu Hu et al. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory. In CVPR 2023."}, {"review_id": "Us5F8ubWRg", "reviewer": "Reviewer_u2aH", "summary": "This paper proposes an innovative continuous memory mechanism to enhance the performance of Vision-Language Models (VLMs) in complex multimodal and cross-lingual reasoning tasks. While RAG concatenates multimodal information as long-sequence inputs, it often leads to excessive context length and performance degradation.\nThis work is the first to leverage VLMs themselves as memory encoders, eliminating the need for costly semantic alignment training.\nExperimental results demonstrate an average accuracy improvement of 5.1%–8.0% across eight multimodal reasoning benchmarks (e.g., OK-VQA, A-OKVQA), with stable performance in long-context scenarios.", "questions": "See weakness. I am open to discuss. If the author can clarify my questions, I am happy to improve my score.", "limitations": "There are no potential moral and ethical issues.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. Proposes VLM-based auto-encoding continuous memory.\n2. Introduces CoMEM, an efficient training framework:\n   \n    (a) Parameter-efficient: Tunes only 1.2% of parameters (via LoRA and lightweight Q-Former).\n\n    (b) Data-efficient: Requires only 15.6K self-generated samples (synthesized by VLM).\n\n    (c) Modular design: Keeps VLM frozen during inference, with plug-and-play memory modules.\n\n3. Extensive experiments demonstrate effectiveness, particularly excelling in low-resource languages (e.g., Bulgarian, Russian).\n\n**Weakness:**\n1. Is the method proposed in this paper still effective for models other than Qwen2?\n\n2. The paper's motivation stems from the concern that RAG \"may drastically increase context length and even degrade performance.\" However, Table 3 shows non-robust performance changes after applying RAG. Perhaps highlighting which data are long-context in Table 3 could better align the results with the motivation.", "was_revised": true, "final_justification": "I appreciate the authors' clarification, which has addressed my concerns.", "rebuttal": "## Rebuttal\n\nWe sincerely appreciate the reviewer’s thoughtful feedback and are grateful for the encouraging remarks. Below, we reply to all the concerns mentioned in the weaknesses and questions section.\n\n### **Weakness-1: Unclear Model Generalization**\n> The effectiveness of the proposed method is primarily demonstrated on Qwen2, leaving it unclear how well the approach generalizes to other models.\n\n**Reply-to-Weakness1**:\n\nOur method is effective for equipping models other than Qwen2-VL to improve the performance, even helpful for augmenting Qwen2-LLM with multimodal knowledge. \n- **Improved Performance on Qwen2-VL and Qwen2.5-VL**: In our paper we evaluate effectiveness of CoMEM on Qwen2-VL and Qwen2.5-VL (Tables 3, 4 in paper). Here we add result table1 by comparing with up-to-date RAG methods. The result shows that CoMEM significantly improves performance over both the original models and strong RAG baselines, demonstrating consistent gains across different models.\n- **Multimodal Knowledge Transfer to Text-only LLM**: In Section 4 (Transferability Study to LLMs), we demonstrate that a text-only LLM (Qwen2.5-Instruct) can leverage the multimodal knowledge provided by our CoMEM using Qwen2.5-VL as the encoder. The result is in Table2. It shows that CoMEM is a powerful and general bridge between VLMs and LLMs, empowering the visual reasoning ability of LLMs, and validating the generalizability of our approach.\n\n**Table1: Performance Comparison Across Datasets (Accuracy %)**\n\n| Model              | infoseek(Q) | infoseek(E) | OVEN    | MRAG    | OKVQA    | AOKVQA     | ViQUAE   |\n|--------------------|-------------|-------------|---------|---------|----------|------------|----------|\n| Wiki-LLaVA         | 28.6        | 25.7        | –       | –       | –        | –          | –        |\n| RORA-VLM           | 27.3        | 25.1        | 15.5    | –       | –        | –          | –        |\n| REVEAL             | –           | –           | –       | –       | **59.1** | *52.2*     | –        |\n| Echosight          | 18.0        | 19.8        | **27.0**| **41.3**| 20.0     | 16.9       | 25.2     |\n| ReflectiVA         | 28.6        | *28.1*      | 20.6    | *39.7*  | *47.5*   | 47.6       | *29.8*   |\n| Qwen2+CoMEM        | *32.6*      | **33.1**    | *23.6*  | 35.1    | *57.7*   | **60.6**   | **36.3** |\n| Qwen2.5+CoMEM      | **32.8**    | 28.5        | 20.8    | 38.1    | 47.6     | *55.0*     | 34.7     |\n\n**Table2: Transferability Study of Vision-Language Memory Encoded by CoMEM on LLMs (Accuracy %)**\n\n| LLM                         | InfoSeek: Query | InfoSeek: Entity | InfoSeek: All | OVEN: Query | OVEN: Entity | OVEN: All | Avg. |\n|----------------------------|--------------------|---------------------|----------------|--------------|---------------|------------|-------|\n| Qwen2.5-Instruct           | 5.0                | 4.8                 | 4.9            | 2.4          | 0.1           | 1.3        | 3.1   |\n| Qwen2.5-Instruct + RAG     | 13.4               | 10.3                | 11.9           | 1.8          | 2.7           | 2.2        | 7.0   |\n| Qwen2.5-Instruct + CoMEM   | **29.3**           | **27.4**            | **28.3**       | **6.8**      | **7.7**       | **7.2**    | **17.8** |\n\n### **Weakness-2: Motivation is not shown in Result clearly**\n> The paper motivates the method by stating that RAG may increase context length and degrade performance, but Table 3 shows only modest or inconsistent performance changes with RAG. Highlighting which samples involve long contexts in Table 3 could help better connect the empirical results to the stated motivation.\n\n**Reply-to-Weakness2**:\n\nWe sincerely thank the reviewer for this insightful suggestion and will report the average context length for each benchmark with or without our method. These clearly show that RAG makes all the inputs of these benchmarks become long-context, approximately **15×** times (see Table 3 below). As long context understanding is a hard problem for LLMs, the performance is also affected:\n- **RAG performance is unstable due to the long context**: While RAG occasionally improves performance (e.g., Qwen2-VL on OKVQA: 36.3 → 41.9), it fails to deliver consistent gains across benchmarks, and even degrades the performance (e.g., Qwen2.5-VL on AOKVQA: 39.8 → 34.9). It indicates that although the additional long context contains useful reference data, the lengthy context makes it difficult for VLM to read and use this information.\n- **CoMEM performs better across different benchmarks, benefiting from its compression ability**: CoMEM provides consistent gains on most of the benchmarks, while using much fewer tokens. This reinforces our claim that our unified memory is not only more efficient but also reliable across diverse tasks.\n\n**Table3: Token Number and Accuracy Across Datasets**\n\n| **Token Number (Accuracy %)** | **Infoseek**       | **OVEN**          | **MRAG**          | **OKVQA**         | **AOKVQA**        | **ViQuAE**        |\n|-----------------------------|--------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n| **Qwen2-VL**                | 357.6 (17.8)      | 408.6 (25.5)     | 84.5 (39.3)      | 404.8 (36.3)      | 418.4 (41.8)      | 424.7 (34.5)      |\n| **Qwen2-VL+RAG**            | 6163.8 (19.0)     | 6481.5 (24.7)    | 5882.4 (40.4)    | 6803.8 (41.9)     | 6715.4 (45.3)     | 6943.2 (33.6)     |\n| **Qwen2-VL+COMEM**          | 416.8 (**33.1**)       | 492.9 (**30.5**)      | 176.9 (**35.1**)      | 453.8 (**57.7**)      | 478.3 (**60.6**)      | 489.2 (**36.3**)      |"}]

Author rebuttal:

Meta-review: The paper received all positive reviews, leading to a final acceptance recommendation.
Decision: Accept (poster)