Memory-Enhanced MLLM for Long-Context Understanding: Addressing Non-Semantic Retrieval

Authors: Chuanyu Zhang, Bo Yang, Chenghua Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: multi-modal MLLM long-context LLM memory

Abstract:
Recent studies have expanded the training data for multi-modal large models, enabling them to handle tasks involving multiple images and videos. However, these models still exhibit suboptimal performance in tasks requiring longer contextual understanding due to limitations in model architecture and training data. Furthermore, differences in model structures often result in incompatible approaches to handling long-context multi-modal tasks. In this paper, we propose a lightweight multi-modal memory component to enhance the long-context processing capabilities of existing multi-modal large models. Our memory component is model-agnostic, allowing it to be applied across different multi-modal architectures. Specifically, we adopt a memory construction approach similar to RAG, where multi-modal input is divided into segments, each encoded with distinct features. Relevant memories are retrieved based on the current input to guide the final generation. During our research, we observed that purely semantic retrieval is often insufficient to provide all the necessary information for multi-modal generation tasks. To address this, we introduce the concept of non-semantic retrieval, which encompasses retrieval tasks that cannot rely solely on semantic information in long-context multi-modal inputs. We compile a variety of common non-semantic retrieval scenarios and establish a corresponding dataset. Based on this, we design and train a model capable of performing both semantic and non-semantic retrieval. Our model leverages attention mechanisms to capture non-semantic information and employs gating mechanisms to balance semantic and non-semantic retrieval results, generating fused feature vectors. This allows our retrieval model to remain compatible with FAISS for high-speed retrieval. We evaluate our approach on three benchmarks focused on long-context multi-modal tasks, demonstrating the effectiveness of the memory module and non-semantic retrieval in enhancing the performance of multi-modal large models, particularly in the challenging tasks of cross-modal image-text interaction and long video understanding. To the best of our knowledge, this is the first work to explore multi-modal memory in large models. We hope our contributions will inspire further research on multi-modal large language models and multi-modal retrieval.

Decision: reject

Reviews and discussion:

Metareview: