NeurIPS 2025

Towards General Continuous Memory for
Vision-Language Models
Wenyi Wu∗, Zixuan Song∗, Kun Zhou†, Yifei Shao, Zhiting Hu, Biwei Huang
University of California, San Diego.
kuzhou@ucsd.edu
Abstract
Language models (LMs) and their extension, vision-language models (VLMs),
have achieved remarkable performance across various tasks. However, they still
struggle with complex reasoning tasks that require multimodal or multilingual
real-world knowledge. To support such capabilities, an external memory system
that can efficiently provide relevant multimodal information is essential. Existing
approaches generally concatenate image and text tokens into a long sequence as
memory, which, however, may drastically increase context length and even de-
grade performance. In contrast, we propose using continuous memory-a compact
set of dense embeddings-to more effectively and efficiently represent multimodal
and multilingual knowledge. Our key insight is that a VLM can serve as its own
continuous memory encoder. We empirically show that this design improves per-
formance on complex multimodal reasoning tasks. Building on this, we introduce a
data-efficient and parameter-efficient method to fine-tune the VLM into a memory
encoder, requiring only 1.2% of the model’s parameters and a small corpus of
15.6K self-synthesized samples. Our approach CoMEM utilizes VLM’s origi-
nal capabilities to encode arbitrary multimodal and multilingual knowledge into
just 8 continuous embeddings. Since the inference-time VLM remains frozen,
our memory module is plug-and-play and can be flexibly integrated as needed.
Extensive experiments across eight multimodal reasoning benchmarks demon-
strate the effectiveness of our approach. Code and data is publicly released here
https://github.com/WenyiWU0111/CoMEM.
1
Introduction
Figure 1: CoMEM architecture in comparison
to the traditional RAG method.
Through
large-scale
training,
language
mod-
els (LMs) [1, 2] have demonstrated remarkable
performance across diver...