CoMemo: LVLMs Need Image Context with Image Memory
Authors: Shi Liu
Weijie Su
Xizhou Zhu
Wenhai Wang
Jifeng Dai
Venue: ICML 2025 poster (2025)
Keywords: Deep Learning
Large Language Models
Large Vision Language Models
Primary area: deep_learning->large_language_models

Abstract: Recent advancements in Large Vision-Language Models built upon Large Language Models have established aligning visual features with LLM representations as the dominant paradigm. 
However, inherited LLM architectural designs introduce suboptimal characteristics for multimodal processing. 
First, LVLMs exhibit a bimodal distribution in attention allocation, leading to the progressive neglect of middle visual content as context expands. 
Second, conventional positional encoding schemes fail to preserve vital 2D structural relationships when processing dynamic high-resolution images. 
To address these limitations, we propose **CoMemo** - a dual-path architecture that combines a **Co**ntext image path with an image **Memo**ry path for visual processing, effectively alleviating visual information neglect. 
Additionally, we introduce RoPE-DHR, a novel positional encoding mechanism that employs thumbnail-based positional aggregation to maintain 2D spatial awareness while mitigating remote decay in extended sequences. 
Evaluations across seven benchmarks,including long-context comprehension, multi-image reasoning, and visual question answering, demonstrate CoMemo's superior performance compared to conventional LVLM architectures.
Project page is available at [https://lalbj.github.io/projects/CoMemo/](https://lalbj.github.io/projects/CoMemo/).

CoMemo: LVLMs Need Image Context with Image Memory
Shi Liu Weijie Su
1
Xizhou Zhu
Wenhai Wang 31 Jifeng Dai
Abstract
parameters. Building upon this foundation, large vision- language models (LVLMs) have emerged powerful mul- timodal systems that align visual representations with LLM embedding spaces to enable cross-modal reasoning (Liu et al., 2024b; Alayrac et al., 2022). Current methodologies for visual information injection predominantly follow two architectural paradigms.
Recent advancements in Large Vision-Language Models built upon Large Language Models have established aligning visual features with LLM representations as the dominant paradigm. How- ever, inherited LLM architectural designs intro- duce suboptimal characteristies for multimodal processing. First, LVLMs exhibit bimodal dis- tribution in attention allocation, leading to the progressive neglect of middle visual content as context expands. Second, conventional posi- tional encoding schemes fail to preserve vital 2D structural relationships when processing dynamic high-resolution images. To address these limi- tations, propose CoMemo dual-path ar- chitecture that combines Context image path with an image Memory path for visual process- ing, effectively alleviating visual information ne- glect. Additionally, we introduce RoPE-DHR, novel positional encoding mechanism that em- ploys thumbnail-based positional aggregation maintain 2D spatial awareness while mitigating remote decay in extended sequences. Evaluations across seven crochmarrk,inclddnng long-context comprehension, multi-image reasoning, and vi- sual question answering, demonstrate CoMemo's superior performance compared to conventional LVLM architectures. Project page is available http://aabb..itttu..o/poopcc//ee
LVLM-X, e.g., Flamingo (Alayrac et al., 2022)) employs cross- coss-attention mechanisms to integrate visual features into textual repre- sentations. While this approach offers flexible modality interaction, recent studies (Laurençon et al., 2024) have re- vealed its suboptimal performance compared to alternative approaches when using identical L.LM backbones and train- ing data finding corroborated by our studies in Figure I.
The first paradigm (referred
The second paradigm (referred to as LVLM-S, e.g., LLaVA (Liu et al., 2024b)) aligns visual tokens into text token embeddings space and then performs autoregressive processing. This paradigm is more compatible with LLM architectures; however, the preservation intrinsic mecha- nisms such as attention bimodality (Xiao et al., 2023; Liu et al., 2025a) and the linearly increasing position encoding leads to critical limitations: (1) the "lost in the middle" (Liu et al., 2024c; Song et al., 2024) phenomenon degrades per- formance with increasing context length, and (2) positional encoding sparsity induces remote decay and 2d-dimensional lost in high- image processing.
In this paper, we propose novel framework for LVLM, named CoMemo. Our key idea is to introduce addi- tional
mechanism that is unaffeccted by context, without modifying the internal mechanisms of the LLM. Specifically, we concatenate image tokens with text tokens as the input sequence for fully autoregressive pro- cessing, while simultaneously feeding the image tokens into mixin layer for cross-attention computation. Cross- attention retrieves image information based on text, avoiding the issue of image neglect that can occur in causal self- attention. However, simply combining these two structures, as in NVLM-H (Dai et al.. 2024), does not work well. To address this, we first investigated a balanced scheme for visual representation inputs. Then, we introduced a three- stage training technique to prevent overreliance on the cross- attention path, effectively transforming it into "memory
1. Introduction
Recent advances in large language models (LLMs) have demonstrated unprecedented generative capabili- ties (Ouyang et al., 2022; Touvron et al., 2023), primarily driven by the exponential scaling of training data and model
Equual contribution PPojectt lead 'Shanghai Artificial Intel- ligence Laboratory "Tsinghua University "The Chinese Univer- sity Hong Kong. Correspondence to: Weijie Su <suwei- jie pjlab.org.cn>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CoMemo: LVLMs Need Image Context with Image Memory
Lang Content
LLM
Mixin LLM
MixinI
LLM
45-
Input Sequence
Input Sequence
Input Sequence
Memory
etton 15
Path
Panlfesl
Tokenizer
Encoder
Tokenizer
Encoder
Tokenizer
Encuder
Nac
DUM
Colfemo
75.4
LVLM-S.
(b) LVLM-X.
(c) Ours.
General VOA
Figure I: Evaluation results of three architectures with same train- ing data and model size (2B). Please refer to Tables 2 to for details.
Figure 2: Comparison three types of architectures for LVLMs. Method (a) use image
encoder to align visual features with the LLM's continuous token representation space. Method (b) employs mixin layer with cross-attention to update LLM's hidden states based on visual features. And Method (c) contrust a dual-path structure to enable the model to focus more on visual content during generation.
path." Meanwhile, the fully autoregressive process serves as the primary path for introducing image context, referred to as the "context path."
Gradiont
Atti, Weight Mean
0.10
0.04
The positional encoding scheme in LVLMs typically adopts RoPE from LLMss, treating each image patch token as an in- dividual token for encoding. However, this approach results in highly sparse positional encodings for dynamic high- resolution image patch tokens. Such sparse encoding can lead to remote decay issues in positional encoding, and the one-dimensional incremental encoding scheme also loses the wwo-dimensional information of the image. To address this, we propose a novel positional encoding scheme based on the dynamic high-resolution method. Specifically, in the dynamic high-resolution approach, the image is divided into multiple image tiles and single image thumbnail. We treat the image thumbnail as part of the input sequence for standard sequential positional encoding, while mapping the image tiles to the image thumbnail index based on their two-dimensional positional relationships.
0.68
Weight
0.02
Gradtent
0.00
0.04
0.02
20
90
100
Tixen Ratio
Figure 3: Average gradients and attention weights assigned to tokens at corresponding positions. We computed the average over 1,000 samples.
2024c; Song et al., 2024), as illustrated in Figure 8a. This refers to their struggle to capture key information placed in the middle of the input. However, this phenomenon arises from the causal self-attention and in-context mechanisms in- herent in these models. Both LLMs and LVLMs are trained using the next-token prediction paradigm, which heavily relies contextual tokens during prediction.
To fully evaluate the impact of the CoMemo architecture, we collected a set of multimodal benchmarks and categorized them into seven evaluation tasks: Caption, Long-generation, Multi-image, Long-conterr Math, General VQA, and OCR- related tasks. The results demonstrates CoMemo's 's superior- ity over LVLM-X/S baselines under same data and model settings. Our framework achieves 17.2%, 7.0% and 5.6% rel- ative improvement on Caption, Long-Generation Generation and Long- Context tasks respectively, with consistent gains across vari- ous benchmarks.
As shown in Figure 3, token during training and the attention weights assigned to each token during inference for the Intern VL2 model. To handle sequences of varying lengths, we map each sequence into 100 bins based on the position of each token. A signifi- cant portion of the gradient for the current predicted token is backpropagated to nearby tokens. As a result, models trained in this way tend to allocate most of their attention to adjacent tokens during inference, while the initial token, which has a larger gradient and attention, acts as an attention sink to release redundant attention (Xiao et al., 2023). This results in the effective capture of key information at the be-
we plot the gradients of each input
2. Design Thinking for CoMemo
2.1. Why LVLMs Tend to "lose in the middle"?
Previous studies have shown that both LLMs and LVLMS
exhibit the "Lost in the middle" phenomenon (Liu et al.,
2

---

CoMemo: LVLMs Need Image Context with Image Memory
Score
log Seerayy
30
+2.
Reletive 5tm
Zeatge SCOrR
20
10
JAM
100
med
Dre..al
200 600 600
1000
Figure 5: Balancing experiments. Experiment settings described in Section 2.3. "Ik", "2k" and "4k" means pre- train steps. All scores are evaluated after fine-tuning the pretrained checkpoint corresponding to the x-axis.
Figure 4: Remote decay estimation for Intern VL2-2B. The relative distance refers to the difference between absolute position IDs. RoPE, the position ID each input token increments by with the input sequence.
creases (Su, 2021), shown in Figure 4. While standard InternVI.2 processes 256 image tokens per image, activating DHR with dynamic number of 6 expands this to 1,792 tokens-a 7x increase that further reduces the influence of image tokens during generation.
ginning of the sequence, as well as nearby tokens benefiting from contextual attention.
However, key information placed in the middle of the quence is more likely to be lost. As the context length increases, the middle segment, which is at a higher risk of being lost, also extends.
The context expansion from DHR fun- damentally aggravates image neglect.
Finding 2:
The attention distribution mechanism in causal self-attention is the cause of the "Lost in the middle" phenomenon. This mechanism also limits the performance of LVLMs in long-context scenarios.
Finding 1:
2.3. The Balance Between tow Pathways
CoMemo faces a critical challenge in balancing two visual processing pathways: the cross-attention path and the fully autoregressive path. Through systematic experiments with different high-resolution allocation strategies and training strategies, we identify two key balancing principles. In the mixin layers, we incorporate a gating mechanism that utilizes gate values to reflect the influence of the cross- attention path during the decoding process. Therefore, we choose to evaluate the dependency of LVLM on these two paths from this perspective. Our analysis averages the gates values of mixin layers to quantify pathway preference, with performance evaluated across captioning, general VQA, and OCR tasks.
2.2. Remote Decay in LVLMs with DHR
Dynamic high resolution significantly enhances the per- formance of visual tasks by introducing more visual con- text, especially in tasks that require high resolution, such as OCR-related evaluations. However, this benefit introduces a critical trade-off: excessive visual context exacerbates the remote decay issue in Rotary Position Embedding (RoPE), ultimately limiting model effectiveness long-context sce- narios.
RoPE implements relative position encoding through abso- lute encoding mechanisms, formally expressed as:
Balance in DHR Allocation. We compare three distinct DHR allocation strategies: (1) allocating DHR information exclusively to the fully autoregressive path, termed DHR-S, (2) assigning it solely to the cross-attention path, termed DHR-X, which corresponds to the allocation mechanism of NVLM-H (Dai et al., 2024), and (3) distributing it to both pathways, termed DHR-B. unilateral allocation sce- narios, the counterpart pathway receives only thumbnail resolution information. As shown in Figure 5, DHR alloca- tion significantly influences pathway specialization. When DHR is exclusively allocated to one pathway, the model shows pronounced bias toward that pathway. contrast, dual-pathway allocation results in more stable and balanced
d/2-1 Σ
( = Re
qp-241 Kpa2r41 M
n)o
(1)
where m,n denote token positions, d the embedding di- mension, and 0; follows the sinusoidal position encoding scheme. This formulation inherently inherits the remote decay characteristics of sinusoidal encodings.
RoPE exhibits the remote decay property where the relative size between tokens decreases as the relative distance in-
3

---

CoMemo: LVLMs Need Image Context with Image Memory
model behavior.
represents the start position biases for the tile in the width and height dimensions.Thhe mapping preserves 2D spatial relationships while maintaining compatibility with existing RoPE implementations. In Figure 6, we visualize the raw image,the thumnail and their corresponding patch token position IDs using a color bar.
Balance in training steps. Our findings reveal that pretrain- ing steps profoundly affects the equilibrium between the two mechanisms. Although fine-tuning involves extended training (9,000 steps), the visual processing paradigm is largely established during pretraining. In Figure 5, the DHR-B configuration demonstrates that insufficient pre- training steps result in suboptimal alignment, leading to compromised fine-tuning performance. Conversely, exces- sive pretraining induces over-reliance on specific pathways. While fine-tuning stage attempts to mitigate this imbalance, ultimately fails to fully rectify the entrenched dependency in DDHR-B44K" in Figure 5.
Crucially, our method decouples positional encoding from absolute sequence positions through: (1) length reduction: prevent the sparse position encoding for DHR by compres- sion the position length in global perspective. (2) geometry reserve: tile patches inherit positional context from their thumbnail anchors.
3.2. Architecture of CoMemo
This phenomenon arises because during pretraining, only the memory branch and projector parameters are trainable. The projector's 's limited function of mapping image repre- sentations into the text space provides minimal gains in visual comprehension. Consequently, prolonged pretraining naturally reinforces reliance on the cross attention branch.
While LLaVA-like architectures demonstrate effective visual-language alignment, they exhibit a tendency to disre- gard visual information when processing lengthy contextual inputs or generating extended responses. To address this limitation, we propose the CoMemo architecture, which is based on three key structures:
Finding 3: CoMemo exhibits a fundamental bal- ancing challenge between dual visual processing pathways.
Dual-stream Structure. CoMemo maintains tow visual processing streams: (1) context path serves as the primary processing stream where image representations are treated as special tokens, concatenated with text tokens to form the input sequence for autoregressive modeling. (2) memory path establishes an auxiliary processing stream where im- age representations interact with the input sequence through cross-attention mechanisms. The two paths maintain iden- tical image representations, ensuring feature consistency while enabling complementary processing and the balance between in tow path as we disccussed in Section 2.3.
CoMemo's dual-path visual process- ing paradigm establishes during pretraining.
Finding 4:
3. Method
3.1. RoPE-DHR
Standard RoPE implementations in LLMs employ a con- tinuous incremental positional encoding scheme, which is inherited by LVLMs. While effective for sequential text data, this approach faces sigmificant limitations when pro- cessing high-resolution visual inputs: (1) remote decay due to extended context lengths, and (2) dimensional collapse of 2D spatial relationships. To address these limitations, we propose RoPE-DHR, novel position encoding method that employs hierarchical strategy.
Position-aware Cross-attention. Existing LVLM-X typi- cally employ absolute positional encoding for image patch tokens during encoding (Dubey al., 2024), with some variants introducing additional positional semantics through specialized tokens (Dai et al., 2024). However, these ap- proaches provide only unidirectional positional awareness. To address this limitation, we implement RoPE in cross- modal attention, establishing bidirectional positional aware- ness: query positions (pos,) correspond to input sequence token ordering and key positions (pos;) align with visual token indices in the input sequence. The attention mask em- ploys bidirectional visibility constraints, similar to mPlug- ow13 (Ye et al., 2024), where image tokens are visible to their corresponding sequence positions while maintaining bidirectional attention.
We first processes the thumbnail patch tokens using con- ventional RoPE to generate base position IDs. For high- resolution tiles, we establish geometric correspondence by mapping each tile patch's 's coordinates (Ieaee Mtate) to its cor- responding thumbnail patch index éhhmb- This mapping is defined as:
Waite W Worig
wbbite Lyeate X Horig
Memory Mixin Strategy As shown in Figure 7, CoMemo mixin layers are interleaved with standard transformer blocks at a 1:4 ratio. Each memory layer performs: (1) Gated Cross-attention: Modulates visual influence through learnable attention gates (attn_gate). (2) Adaptive Feed-
dehumb (LetaeX x
+hbrae)
Hukumb
(2)
where (Worig: Horig) and Wister Haste) denote the original and tile dimensions respectively. The terms whiste and hbbate
4

---

CoMemo: LVLMs Need Image Context with Image Memory
Image Tokens
Mixin LLM
Layer n
Algorithm 1 Mixin Layers
(I. (2,
1)
Require:
ht (sequence hidden states).
Mixin Layer
Projector
thumnail
h; (image hidden states),
(1,2) (2, 2)
Layer
attn-gate ffw-gaate pos, (sequence position IDs),
Pixel Shuffle
Input Sequence
pos; (image position IDs)
Encoder
Ensure: Updated hgg
1: h2 f- his + Lanh(atln (aahe) O cross attn(q h1s kv
(1, 1)
Dynamic High Resolution
Tokenizer
hi pos, pos,: pos; = pos:)
thumnail
User Message
2: hg
h. + tanh(ffw gate) @ ffw(hs)
Figure 6:
The computation
Figure 7: Framework of CoMemo. Both paths share the same encoder and projector. The pixel shuffle technique is adopted from InternVL series (Team, 2024; Chen et al., 2024c).
3: Return: h4
process of Rope-DHR. The colors are assigned based on mapping of position IDs in RoPE.
forward: Enhances feature transformation via gated nonlin- earity (ffw-gate).
of projector and mixin layers. In the next stage, we freeze the gate parameters. This design aims to enable the model to learn representation alignment and balance the dual-path structure in the first stage, After a certain number of training steps to prevent over-reliance on the cross-attention path, we freeze the corresponding gate control parameters and continue training until the alignment structure is sufficiently learned.
During autoregressive decoding, CoMemo requires only single-step computation between the current decoding token and the cached visual memory states, eliminating the need for key-value caches. This approach circumvents the issue of increasing key-value cache size as the sequence lengthens. The orthogonal design of the architecture ensures compatibility with existing LLaVA variants.
The subsequent fine-tuning stage adopts full-parameter train- ing paradigm. During this stage, all model parameters be- come trainable with the training objective shifted towards instruction-following
3.3. Training Stages
Traditional LVLMs training consists of two phases: pre- training and fine-tuning. During the pretraining phase, we selectively update the projector module and memory archi- tecture parameters while keeping other components frozen. This phase prioritizes cross-modal representation alignment and dynamic equilibrium between dual visual processing pathways.
4. Experiments
4.1. Setup
To ensure a fair comparison of the capabilities across various architectures, all three of our models utilize the same pre- training and fine-tuning datasets. Each architecture adopts InternI MM-18B as the LLM backbone and Intern ViT-300M as the image encoder. The hidden dimensions of each ar- chitecture's 's network, along with hyperparameters such as learning rate and weight decay during training, are also kept consistent. We use the same training data as Intern Therefore, LVLM-S in our experiments essentially repre- sents VL-2 Training for architectures is divided into two phases: pretraining and fine-tuning For specific settings, please refer the appendix.
A critical challenge emerges during pretraining stemming from asymmetric parameter updates: (1) Insufficient train- ing iterations lead to suboptimal projector learning; (2) Prolonged training induces excessive dependency on the memory pathway for LVLM decoding. This imbalance orig- inates from the disparate update strategies- full fine-tuning of memory parameters versus partial tuning of the context projector. Consequently, the LVLM exhibits inherent bias towards memory-based information retrieval to achieve loss minimization.
To mitigate this optimization bias, we propose three-stage training strategy. In the first stage, we tune the parameters
5

---

CoMemo: LVLMs Need Image Context with Image Memory
Table I: Comparison with other LVLMs of different architectures. indicates results obtained from our own experiments.
Other results are sourced from official reports or third -arty evaluation leaderboards.
Model
Parama.
Caption
Loog Generation
Mutti Imange
Long Contest
Math
General YQA
OCR.Relnn
LLAVARe:
Miltlaeh
MaaVies
MathVicns
COCO
No-Oge
BLINK
Munss
MNY
M06N
MOEE
MMV
A2D
Ulosed Souree
GPT 4o
70.2
68.0
65.4
60.3
63.8
84.5
85.7
GPF-4V
54.6
62.7
53.0
58.1
1925.6
75.2
INLMEX
inPlap-oo11
8B
90'
94.2
50.3
63.1
73.4
11B
30.8
51.5
1520.5
91.1
TREMS
MisiCMM-V
2.88
66.1
41.2
54.5
30
154
62.9
1508.1
64.8
99.6
1
28
7M.1"
16.4"
60.0"
62.5
25.7"
39.2"
48.3"
50.2"
271"
12.1"
48.0
16.4"
73.1"
1856"
31.5"
74.5"
75.6"
74.5"
InseaVL 2.
28
114
50.2
108.6"
33.6"
44.0
54.8
54.5
22."
50.1"
51.3
2138.2
74.9
79.2
Quee2 VL
28
52.5
43.0
2336.8
73.5
CoMann
26
25.6
387
453
30.6
345
30
1934
350
762
Table 2: The results on Generation and Math benchmarks. The test settings are described in Section 4.3. The highest scores are highlighted in bold.
Table 3: The results on Multi-image and Long-context benchmarks. The test settings are described in Section 4.4. The highest scores are highlighted in bold.
Model
Caption
Long Generation
Math
Model
Multi- Image
Long-Context
CD00
Flick-
Ne-Caps
..aVARench
MMDU
MathVista
MatiVision
BLINK
Manti
MMT
MM-NIAH-M
MM-NIAH-T
MileBench
LVLMIX INLMS
Ours
98.6
79.1
84.9
78.5
08.9
65.4
788
62.9
60.0
62.9
50,8
66.9
38,7
28.7
31.5
48.0
59,0
44.2
16.4
17.0
15.8
LVLM-X LVLM-S
38.2
41.5
46.5
48.3
47.8
39,5'
29.5
27.3
$3.2
52.1
Ours
43.5
50.6
51.3
33.7
34.6
54.6
4.2. Comparison with Other LVLMS
lengthy textual context (avg. 6.4k tokens) and multiple images (ranging from 2 to 20 images).
As shown in Table 1, we benchmark CoMemo against lead- ing open-source and proprietary LVLMs. It is important to note that our primary objective is architectural ablation rather than absolute performance maximization. The com- parison with other models serves to demonstrate that our model was trained on a large-scale dataset and achieves top- tier performance, rather than being tested on toy datasets.
Math: Measures model reasoning ability on math diagrams and visual math problems using Math Vi- sion (Wang et al., 2024a) and Math Vista (Lu al., 2023). During evaluation, vide step-by-step reasoning and extracted the final an- swer to calculate accuracy.
asked the model to pro-
4.3. Generation and Math Benchmark
As shown in 2, our analysis reveals three key findings. First, LVLM-X's continuous attention mechanism demonstrates superiority in pure visual captioning tasks, achieving a +4% average improvement. Second, LVLM-S's causal atten- tion architecture achieves better performance in knowledge- intensive scenarios, demonstrating enhanced contextual rea- soning capabilities from its I.I.M backbone. Our proposed CoMemo combines the advantages of both approaches, out- performing the original architectures across various tasks. This supports our hypothesis that dual-path attention alloca- tion effectively integrates the benefits of both architectures: maintaining visual grounding while enabling complex rea- soning.
Contemporary multimodal benchmarks primarily rely on constrained response formats (e.g., multiple-choice ques- tions and Yes/No queries) to facilitate evaluation. However, such approaches inadequately assess open-ended reasoning capabilities, as free-form responses introduce higher diver- sity and complexity. To comprehensively evaluate model capabilities across different granularities, we collected sev- eral benchmarks for open-ended evaluation:
Evaluates the model's abil- ity to generate concise image descriptions (10-15 words) using CIDEr scores COCO (Lin et al., 2014a), Flickr30k (Plummer et al., 2015), and No- Caps (Agrawal et al., 2019) datasets.
Caption Generation:
4.4. Multi-image and Long- Benchmark
Long Generation: Evaluates extended inference using LLaVABench (Liu et al., 2024b) and MMDU (Liu et al., 2024e). LLaVABench focuses on single-image context reasoning, involving a few hundred context tokens, while MMDU is multi-image analysis with
To systematically evaluate multimodal long-context under- standing, we establish two complementary evaluation di- mensions:
LVLM--X's single image token compression reduces average context length by 50% (c.g., 32k- 166)..
6

---

CoMemo: LVLMs Need Image Context with Image Memory
Multi-image: When DHR is enabled, each image con- tributes approximately 2k tokens to the context length, thereby necessitating extended context capacity for mulli-image analysis. We evaluate performance on the BLINK (Fu et al., 2025), Mantis (Jiang et al., 2024), and MMT (Ying et al., 2024) datasets.
10
30
Androgo 500r
Long-context: Tests information extraction in long context scenarios. We select MM-NIAH (Wang et al., 2024c) evaluation that detects image/text needles within hybrid long contexts. And MileBench (Song et al., 2024) progressively challenging tasks with 2- 109 images. These benchmarks systematically quantify long-cone long-context capabilities from both textual token and visual token perspectives.
32
86. 12k uk 20ki 24k abi xak 3.
4.. 8K a8k 20: 244 2bk sok
Teken
(a) LVLM-S.
(b) CoMemo
Figure Heatmap of results for the NIAH evaluation on MileBench benchmark. The depth percentage indicates the position of the target information (needle) relative to the entire sequence.
As shown in Table 3, our proposed architecture achieves the best performance in scenarios involving multiple images and long contexts. Specifically, MM-Niah-T represents the needle that is the key information placed in the text, while MM-NIAH-M represents the needle placed in the image. In the evaluation of MM-NIAH-T, the memory structure stores image data that is unrelated to the needle, redundant information. Nevertheless, our model still achieves the best performance. This not only demonstrates that compress- ing the image token position space through RoPE-DHR enhances the model's ability to understand long sequence texts but also indicates that the Memory structure does not cause the model to overfocus on image information, thereby preserving its ability to retrieve and reason with language effectively.
Table 4: benchmarks. The test settings are described Section 4.5, The highest scores are highlighted in bold.
The results on General VQA and OCR-related
General VQA
ACR-Related
Model
MMBench
MME
MMVP
A12D
ChartQA
TextVQA
IVLM-X LVLM-S
69.6
1812
27.3
69.9
69.4
68.8
73.1
1869
31.3
74.3
75.6
74.2
Ours
74.2
1904
36.0
74.2
73.6
72.6
4.5. General VQA and OCR Benchmark
To comprehensively evaluate CoMemo's multimodal un- derstanding capabilities, we conduct extensive experiments on conventional vision-language benchmarks covering two main categories:
In the MileBench evaluation, due the potential for ex- cessive image tokens leading long context sequences and out-of-memory issues, did not enable DHR set- tings. Therefore, in this evaluation, each image in the input sequence has only single image thumbnail. In this sce- nario, our architecture's positional encoding is the same as LVLM-S, primarily reflecting the role of the memory structure. Despite this, our architecture still achieved 2.5% improvement over LVLM-S.
General VQA: Assessing visual perception and reason- ing abilities through single-image question answering, including MMBench (Liu et al., 2025b), MME (Fu al., 2023), and MMVP (Zhong et al., 2023).
OCR-Related: Requiring fine-grained information extraction from diagrams and charts, evaluated on A12D (Hiippala et al., 2021), TextVQA (Singh et al., 2019b). and ChartQA (Masry al., 2022b).
The MileBench benchmark also includes needle in baystack tasks for both text and image scenarios. In Figure 8, we visualize the average results for these two types of NIAH tasks in MileBench. As mentioned earlier, we observed that the attention mechanism of LLMs and LVLMs exhibits a bimodal distribution, where LVLMs tend focus more on the beginning and the most recent tokens, leading the "Lost in the middle" phenomenon. This means that when the needle is placed in the middle of the sequence, the model's 's performance on NIAH tasks deteriorates. Our architecture, however, addresses this issue by continuously focusing on the "middle image information" during the token generation process, effectively mitigating this problem.
The input sequences in these two types of benchmarks are relatively short, and responses typically consisting of single words or short phrases. In contrast, CoMemo incorporates memory structure that alleviates image neglect by intro- ducing additional image focus mechanism. Additionally, RoPE-DHR addresses image neglect by compressing im- age information to reduce the long-range decay caused by positional encoding. While these techniques are not specif- ically tailored for the benchmarks mentioned above, our architecture still performs competitively when compared to

---

CoMemo: LVLMs Need Image Context with Image Memory
LVLM-S.
after incorporating RoPE-DHR, the model's capabilities in different dimensions were further enhanced (Variant 5). However, since RoPE-DHR is essentially compression- oriented encoding, it may affect scenarios like OCR that require fine-grained information.
As shown in Table 4, our architecture performs slightly better than LVLM-S on some general multimodal bench- marks. However, in tasks such as text and OCR, which require high-resolution image information, traditional ap- proaches typically rely on more granular image representa- tions. This approach contrasts with the philosophy underly- ing our approach, which focuses on improving the model's long-context and long- generation capabilities.
Ablation on compression ratio of ROPE-DHR, In the main experiments, RoPE-DHR uses shared position IDs for image tokens in the thumbnail and their corresponding subimage tokens. This approach effectively compresses the position encoding. Therefore, we propose variant, RoPE- DHR without compression, where the position encodings for subimage tokens corresponding to a two-dimensional position increment by relative to the thumbnail image tokens, while the position IDs between thumbnail image tokens increment based on the number of tokens at their corresponding positions, rather than by 1. The experimental results are shown in Table 7 in Variants 4. It can be ob- served that, without compression, the model outperforms the LVLM-S architecture across all dimensions.
4.6. Training and Inference Efficiency Comparison
Table 5: Training efficiency comparison across different 2B parameter models.
Model
Batch size
100 GPUS
Train stepels
Train samples/s
LVLM-X
1024
64
0.123
15.71
LVLM-
1024
64
0,105
13.4
CoMemo
1024
64
0,096
12.26
Table 6: Inference speed comparison across different 2B parameter models (lower is better).
Ablation on different scale models. To verify that CoMemo adheres to the scaling law, we select InterLM-- 7B as the language model for experiments at the 7B scale. As shown in Table 7 in Variant 6 and 7, at the 8B scale, CoMemo's average performance across all dimensions re- mains superior to the LVLM-S architecture. The CoMemo architecture continues to deliver outstanding performance in both Cation and Long-context tasks. As language mod- els scale up. the impact of compressed position encoding becomes more pronounced in OCR tasks.
CPUN
Caption COCO
(9e)
MMlBcach
MMINIAI
L CoMero
1
200
90
25
100
280
130
30
120
To comprehensively compare the training and inference ef- ficiency of three architectures, we measured their sample throughput during training and inference times across mul- tiple benchmarks. As shown in Table 5 and Table 6, we report the training and inference efficiency for each archi- tecture. The results demonstrate that CoMemo achieves latency comparable to LVLM-S. Although LVLM-X ex- hibits higher efficiency due to the use of fewer image tokens, its performance is significantly inferior to both CoMemo and LVLM-S.
Consistency on Different Datasets. We also con- ducted dataset-switching experiments using the open-source Intern VL- 1.2 1.2M fine-tuning dataset (OpenGVLab, 2024) during the SFT stage, while keeping the pretraining data consistent, As shown Variants to 9 Table 7, even with the changes in the dataset, our CoMemo consistently outper- forms the LVLM-S architecture across various dimensions.
4.7. Ablation Study
5. Related Works
Ablation on Components of CoMemo. We conducted complete ablation study on components using a 2B-scale model, as shown in Variants I to 5 in Table 7. When only RoPE-DHR is introduced, the compressed position encoding significantly improves performance on both long- generation and long-context tasks (Variant 2). When only the memory path was introduced, it addressed issues related to neglecting image information, leading to some improve- ment in the model's performance (Variant 3). However, since the image tokens in the memory path lack positional information, during cross-attention computation, there is no positional correspondence between image tokens and text tokens. Moreover, the lack of distinguishing features be- tween image tiles, thumnails, and multiple images hindered the model's capabilities in different dimensions. Therefore,
5.1. Mainstream LVLMs and Their Architectures
Contemporary LVLMs typically employ pre-trained lan- guage models as decoders, utilizing two dominant strategies for visual-text aligmment: (1) cross-attention mechanisms and (2) joint
architectures.
Fully Autoregressive LVLMs: LLaVA (Liu al., 2024b) pioneered this approach by projecting image representa- tions into the LLM's space and jointly decoding them with text tokens. Subsequent models like VILA-v1.5 (Lin et al., 2024) and LLaVA-Next (Liu et al., 2024a) built on this architecture, with L.aVA-Next introducing dynamic high- resolution techniques for improved performance. Other ad-
8

---

CoMemo: LVLMs Need Image Context with Image Memory
Table 7: Ablation Study. The test settings are described in Section 4.7. "NC" means to use RoPE-DHR without compression.
Name
Furums.
Memory
ROPL-DHRR
Dutasets
Overall
Caption
Long- Sungg-Geerratin
Multi-
Long-Coriest
Math
General VQA
OCR.
Ouns
55.4
Varian
Ouns
56.7
67.2
51.4
47.5
42.0
31.7
69.
72.9
Varian: 3
2B
X
Ouss
59.0
$2.8
48.6
46.0
43.0
32.2
66.7
75.6
Variam 4
28
NC
Dus
59.5
79.7
51.9
48.5
41.1
34.7
67.3
74.8
Varian 5
28
Ous
60.5
85.3
5288
48.5
44.4
33.5
68.4
73.4
Variant
Ours
65.4
57.4
36.7
82.3
Ouns
68.0
92.1
57.7
50.8
38.9
78.1
Vanam:
1.2M
51.1
79.7
36.1
64.1
28.3
583
613
Variam 9
28
1.2M
54.6
89.5
38.4
46.2
31.5
26.7
623
63.5
vancements include Qwen2-VI. VL (Wang al., 2024b), which introduced M-RoPE, and Intern VL-2.5 (Chen et al., 2024c), which used pixel shuffle to reduce token count after DHR. DeepSeek-VLL VL2 (Wu et al., 2024) employed a pretrained LLM. However, this alignment approach inher- its the LLM's generation mechanism, which can lead to issues such as "image neglect" or the "lost in the middle" problem.
customized ViT and thus cannot be applied to LVLMs employing DHR.
Our proposed RoPE-DHR, based on ID principles, offers a 2D-aware encoding scheme that addresses these challenges without the extra computational burden.
6. Conclusion
LVLMs: Flamingo (Alayrac et al., 2022) is an early example of LVLMs using cross-attention mechanisms, Later models, like Idefics (Laurençon et al., 2024) and LLaMa-3.2. 2- Vision (Dubey et al., 2024), adopted its mixin layer design, which introduces cross-attention and gating mechanisms. EVLM (Chen al., 2024b) experi- mented with using intermediate Vision Transformer repre- sentations as inputs to the mixin layer. mPlug-ow13 (Ye et al., 2024) added adaptive gating and hyper-layer fusion to combine cross-attention and self-attention. This approach enables visual understanding while maintaining the LLM's frozen language ability, as seen in ALaMM-3.2-Vins Vision. How- ever, in these models, image representations are aligned directly to the LLM'' hidden state, whereas LLaa---iik methods align them with the text token space, better lever- aging autoregressive decoding capabilities and improving performance.
We present CoMemo, a novel architecture for Large Vision- Language Models specifically designed for long-form gen- eration and extended context understanding. Our approach features a dual-path image processing mechanism and in- troduces RoPE-DHR to alleviate remote decay in DHR sce- narios while restoring critical wvo-dimensional spatial infor- mation. These innovations significantly enhance model per- formance across multiple tasks including image captioning, long-form generation, long-context understanding, multi- image analysis, and general visual question answering. We hope this work will contribute to the advancement of the vision-language modeling community.
Impact Statement
This paper presents work whose goal is to advance the field of Large Vision-Language Model. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
5.2. Position Encoding Schemes in LVLMs
Most LVLMs use position encoding methods inherited from LLMs, primarily RoPE. In these models, each image patch token is treated like a text token and assigned position IDs for RoPE computation. However, several advancements address specific challenges. MiniCPM-V (Yao et al., 2024) introduced an absolute position encoding for each image tile in the context of DHR, while LLaMA-3.2-V (Yao et al., 2024) designed encodings for both image tiles and patch tokens. NVLM (Yao et al., 2024), also leveraging DHR, added special tokens before each tile to convey positional information. While effective for predefined DHR ratios, these methods lack scalability.
Acknowledgments
The work is supported by the National Key R&D Program of China (NO. (022/D0161301) by the National Natu- ral Science Foundation of China (U24A20325, 62321005, 62376134).
References
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., John- son, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. Nocaps: Novel object captioning scale. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8948-89889 8957, 2019.
In contrast, VL.2 (Wang et al., 2024b) introduced M-RoPE, a dimeniddimensional position encoding extending RoPE to three channels (temporal, height, width) for im- ages and videos. However, this position encoding requires
Alayrac. J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds,

---

CoMemo: LVLMs Need Image Context with Image Memory
M., al. Flamingo: visual language model for few- shot learning. Advances in neural information processing systems, 15:23716-23755 2022.
with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 2024c.
Chi, Z.., Huang, H., Xu, H.-D., Yu, H., Yin, W., and Mao, X.-L. Complicated table structure recognition. arXiv preprint 190.v:9908.047200 2019.
Amini, A., Gabriel, S., Lin, P., Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:I905.0 13319, 2019.
R., Choi,
Chng. C. K., Liu, Y.. Sun, Y., Ng. C. C., Luo, C., Ni, Z..
Fang, C., Zhang, S., Han, J., Ding, E., al. lcdar2019 robust reading challenge on sebitrary-shaper text-rrc-art. In ICDAR, pp. 1571-1576. IEFF, 2019.
Ben Abacha, A., Hasan, S. A., Datla, V. V., Demner- Fushman, D., and Müller, H. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum)2200 2019 Working Notes. 9-12 Septem- ber 2019, 2019.
Clark, C. and Gardner, M. Simple and effective multi- paragraph reading comprchension. arXiv preprint 1rXiv:7710.102533 2017.
Dai, W., Lee, N., Wang, B., Yang, Z., Liu, Z., Barker, J., Rintamaki, T., Shoeybi, M.. Catanzaro, B., and Ping, W. Nvlm: Open frontier-class multimodal llms. arXiv preprint arXiv:2409. /402, 2024.
Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D. Scene text visual question answering. pp. 4291 1990-, 2019.
Byeon, M., Park, B., Kim, H., Lee, S., Back, W., and Kim, S. Coyo-700m: Image-text pair dataset. https: /tte://ggthub...// con//kakobbain/ coyo-dataset, 2022.
Davis, B., Morse, B., Cohen, S., Price, B., and Tensmeyer, C. Deep visual template-free form parsing. In ICDAR, pp. 134-141. IEEE, 2019.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Cao, J. and Xiao, J. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In COLING, pp. 151 1511-1520, 2022.
Carter, J. Textocr-gpt4v. https://hugginggann coo/atasett htta://tttss/mmmatteeeeeeeeeeeee htmyyaateer/ttextorgggettteee 2024.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Chang, S., Palzer, D., Li, J., Fosler-Lussier, E., and Xiao, N. Mapqa: A dataset for question answering choropleth maps. arXiv preprint arXiv:2211.08545: 2022.
Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., and Krishna, R. Blink: Multi- modal large language models can see but not perceive. In European Conference Computer Vision, pp. 148-166. Springer, 2025.
Chen, G. H.. Chen, S., Zhang. R., Chen, J., Wu, X., Zhang, Z., Chen, Z., Li, J., Wan, X., and Wang, B. Allava: Har- nessing 8pt4v-synthesized data for lite vision-language model. arXiv preprint arXiv:2402. 11644 2024a.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. pp. 6904-6913, 2017.
Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., and Liang, X. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746 2022.
Gu, J., Meng, X., Lu, G., Hou, L., Minzhe, N., Liang, X., Yao, L., Huang, R., Zhang, W., Jiang, X., et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark.
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal l1m's referential dialogue magic. arXiv preprint ar arXiv:2306. 15/95, 2023.
0226418-26433 2022.
Chen, K. Shen, D., Zhong, H., Zhong, H., Xia, K., Xu, D., Yuan, W., Hu, Y., Wen, B., Zhang, T., et al. Evlm: An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.1 / /4177, 2024b.
Guo, H., Qin, X., Liu, J.. Han, J.. Liu, J., and Ding, E. Eaten: Entity-aware attention for single shot visual text extraction. In ICDAR, pp. 259-259. IEEE, 2019.
He, M., Liu, Y., Yang, Z., Zhang, S., Luo, C., Gao, F. Zheng, Q., Wang, Y., Zhang, X., and Jin, L. Icpr2018 contest on robust reading for multi-type web images. pp. 7-12. IEEE, 2018.
Chen, Z., Wang. W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models
10

---

CoMemo: LVLMs Need Image Context with Image Memory
He, X., Zhang, Y., Mou, L.,, Xing, E., and Xie, P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring objects in photographs of natu- ral scenes, pp. 787-798, 2014.
Hiippala, T., Alikhani, M., Haverinen, J., Kalliokoski, T., Logacheva, E., Orekhova, S., Tuomainen, A., Stone, M., and Bateman, J. A. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Re- sources and Evaluation, 55:661-688, 2021.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. pp. 235-251, 2016.
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. pp. 4999-5007, 2017.
Hosu, V., Lin, H., Sziranyi, T., and Saupe, D. Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. 29:4041-40555 2020.
Kuang, J., Hua, W., Liang, D., Yang, M., Jiang, D., Ren, B., and Bai, X. Visual information extraction the wild: practical dataset and end-to-end solution. In ICDAR, pp. 36-53, Springer, 2023.
Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Li, C., Zhang, J., Jin, Q.. Huang, F., et al. unplu-ddcowl 1.5: Unified structure learning for ocr-free document under- standing. arXiv preprint arXiv:2403. 12895, 2024.
LAION. Laion-gpt4y dataset. https:://hugiingg co/ www.aatssssaaaaoo hoo//ataaetts//aaioogg------aaas 2023.
Hu, X., Gu, I.,, An, Q.. Zhang, M., Liu, I.., Kobayashi, K., Harada, T., Summers, R., and Zhu, Y. Medical-diff-qqa: a large-scale medical dataset for difference visual question answering on chest x-ray images, 2023.
Lau, J. J., Gayen, S., Ben Abacha, A., and Demner- Fushman, D. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 201.
Huang, Q. Xiong, Y., Rao, A., Wang, J., and Lin, D. Movienet: A holistic dataset for movie understanding. pp. 709-727. Springer, 2020.
Laurençon, H., Tronchon, L., Cord, M., and Sanh, V. What matters when building rision-languse models? arXiv preprint arXiv:2405.02246, 2024.
Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., and Jawahar, Icdar2019 competition on scanned receipt ocr and information extraction. In ICDAR, pp. 1516-1520. IEEE, 2019.
Lerner, P., Ferret, O., Guinaudeau, C., Le Borgne, H., Besançon, R., Moreno, J. G., and Lovón Melgarejo, J. Viquae, a dataset for knowledge-based visual question an- swering about named entities. In SIGIR, pp. 2022.
Hudson, D. A. and Manning, C. Gqar A new dataset for real-world visual reasoning and compositional question answering. pp. 6700-6709, 2019.
Li, J., Zhang, D.,, Wang, X., Hao, Z., Lei, J., Tan, Q., Zhou, C., Liu, W., Yang, Y., Xiong, X., et al. Chemvim: Explor- ing the power of multimodal large language models in chemistry area. arXiv preprini 2xXiv:2408.07246. 2024.
Jiang, D., He, X., Zeng, H., Wei, C., Ku, M., Liu, Q., and Chen, W. Mantis: Interleaved multi- image instruction tuning. arXiv preprint urXiv:2405.01.
2024.
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diag- nostic dataset for compositional language and elementary visual reasoning. pp. 2901-2910, 2017.
Li, Z., Wang, X., Stengel- Eskin, E., Kortylewski, A., Ma, W., Van Durme, B., and Yuille, A. L. Super-clevr: A vir- tual benchmark to diagnose domain robustness in visual reasoning, pp. 14963-14973, 2023.
Kafle, K., Price, B., Cohen, S., and Kanan, C. Dvqa: Under- standing data visualizations via question answering. pp. 5648-5656, 2018
Lin, J.. Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689- 26699, 2024.
Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., and Bengio, Y. Figureqa: An anno- tated figure dataset for visual reasoning. arXiv preprint 2017.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014 4a.
Kantharaj. S., Leong, R. T. K., Lin, X., Masry, A., Thakkar, M., Hoque, E., and Joty, S. Chart-to-text: A large- scale benchmark for chart summarization. arXiv preprint arXiv:2203.0 06486, 2022.

---

CoMemo: LVLMs Need Image Context with Image Memory
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. pp. 740-755. Springer, 2014b.
Liu, Z.., Chu, T., Zang, Y., Wei, X., Dong, X., Zhang, P. Liang, Z., Xiong, Y., Qiao, Y., Lin, D., et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tun tunnnng dataset for Ivlms. arXiv preprint arXiv: 1133, 2024e.
Liu, B., Zhan, L.-M., Xu, L., Ma, L.. Yang. Y., and Wu, X.- M. Slake: A semantically- labeele unowldde--enhance dataset for medical visual question answering. In ISBI,
Lu, P., Gong, R., Jiang, S., Qiu, L., Huang, S., Liang, X., and Zhu, S.-C. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint wxXi:22055004101 2021.
pp. 1650-1654. IEEE, 2021.
Liu, C.-I., Yin, F,, Wang, D.-H., and Wang, Q.F.. Casia online and offline chinese handwriting databases. In ICDAR. pp. 37-41. IEEE, 2011.
Lu, P., Mishra, S., Xia, T., Qiu, I.., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. 35:2507-2521, 2022a.
Liu, F., Emerson, G., and Collier, N. Visual spatial reason- ing. Transactions of the Association for Computational Linguistics, 11:65-651,, 2023a.
Lu, P., Qiu, L., Chang, K.-W., Wu, Y. N., Zhu, S.-C., Ra- jpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for semi-structured math- ematical reasoning. arXiv preprint (tti::2209.161111 2022b.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. Mitigating hallucination in large multi-modal models via robust instruction tuning. 2023b.
Liu, F., Wang, X., Yao, W. Chen, J., Song, K., Cho, S., Yacoob, Y., and Yu, D. Mme: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311, 10774, 2023c.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Llava-next: Improved reason-
and Lee, Y. ing, ocr, and world knowledge, January 2024a. URL https://llawe htw.llaaaavl.gihuub... io/blog/ 2024-015 01-30-11ava next/.
Mao, H., Cheung, M., and She, J. Deepart: Learning joint representations of visual arts. pp. 1 1183-1191, 2017.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok- vqa: A visual question answering benehmark requiring external knowledge. pp. 3195-3204, 2019.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M.. Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173. 2024c.
Marti, U.-V. and Bunke, H. The iam-database: english sentence database for offline hand recognition. In- ternational jonatioonaJourralll journal on document analysis and recognition, 5:39 46, 2002.
Masry, A., Do, X. L., Tan, J. Q., Joty, S., and Hoque, Chartqa: benchmark for question answering about charts with visual and logical reasoning. In ACL, pp. 2263-2279, 2022a.
Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free free method for alleviating halluci- nation in Ivims. In European Conference on Computer Vision, pp. 125-140. Springer, 2025a.
Masry, A., Long, D. X., Tan, J. Q., Joty. S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint 1:Xh:22033110444 2022b.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao W., Yuan, Y., Wang, J., He, C., Liu, Z., al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2025b.
Masry, A., Kavehzadeh, P., Do, X. L., Hoque, E.,, and Joty, S. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv: /4761, 2023.
Liu, 7., Chu, T., Zang, Y., Wei, X., Dong, X., Zhang, P., Liang, Z., Xiong, Y., Qiao, Y., Lin, D., et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuminn dataset for lylms. arXiv preprint arXiv:2406. 1833, 2024d.
Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. Infographievqa. pp. 1697-1706, 2022.
12

---

CoMemo: LVLMs Need Image Context with Image Memory
Methani, N., Ganguly, P., Khapra, M. M., and Kumar, P.
Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., and Hassner, T. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. pp. 8802-8812, 12, 2021.
Plotqa: Reasoning over scientific plots. pp. 1527-1536, 2020.
OpenGVLab.
Opengyyaa/neerr//ccavvvv.aattite Jan 2024. URL butpps:/huggiigg:tee co/
Song, D., Chen, S.,, Chen, G. H., Yu, F., Wan, X., and Wang. В. Milehench: Benchmarking milms in long context. arXiv preprint arXiv:2404. 18532, 2024.
OpenGVLab/ httpaaab/nnntee///-at------
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang. C.,, Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277 27-4444 2022.
Su, Transformer roadmap: 2. rotary position embed- ding. Mar 2021. URL https: spaces ac. en/ archives/8266
Sun, Y., Ni, Z., Chng, C.-K., Liu, Y., Luo, C., Ng. C. C., Han, J., Ding, E., Liu, J., Karatzas, D., et al. Icdar 2019 competition on large-scale street view text with partial labeling-rre-lstt In ICDAR, pp. 1557-1562. IEEE, 2019,
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Team, O. Internv12: Better than the best expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024. URL https://internvl. wwtperrnii..iiittteeee ritthb..o/blee.... 2024-07-02-Interrv---
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C.,, Hockenmaier, J., and I Lazebnik, Flickr30k entities: Collecting region-to-phrase correspondences for richer seage-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641 2649, 2015.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.,, Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open aarge-ccale dataset for training next generation image-text models. 35:25278-25294, 2022a.
Veit, A., Matera, T., Neumann, L., Matas, J., and Belongie, S. Coco-text: Dataset and benchmark for text detec- tion and recognition in natural images. arXiv preprint arXiv: 1o0::6601.714 2016.
Schuhmann, C., Köpf, A., Veneu, R., Coombes, T., and Beaumont, R. Laion coco: 600m synthetic cap- tions from laion2b-en. /tttps://la ai..o...a//lo// lai daion-coca 2022b.
Wang. J., Zhang, P., Chu, T., Cao. Y., Zhou, Y., Wu, T., Wang, B., He, C., and Lin, D. V3det: Vast vocabulary visual detection dataset. pp. 19844-19854, 2023a.
Seo, M. Hajishirzi, H., Farhadi, A., Etzioni, O., and Mal- colm, C. Solving geometry problems: Combining text and diagram interpretation. pp. 1466-1476, 2015.
Wang, K., Pan, J., Shi, Lu, Z., Zhan, M., and Li, H. Mea- suring multimodal mathematical reasoning with math- vision dataset. arXiv preprint arXiv:2402. /4804, 2024a.
Shah, S., Mishra, A., Yadati, N., and Talukdar, P. P. Kvqa: Knowledge-awee aware visual question answering. volume 33, pp. 8876-8884, 2019.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-languag model's perception of the world at any resolution. arXiv preprint arXiv:2409, 120.499..199991 12191, 2024b.
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. Objects365: A large-scale, high-quality dataset for object detection. pp. 1430-8439, 2019.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. pp. 8317-8326, 2019a.
Wang. W., Shi, M., Li, Q., Wang, W., Huang, Z., Xing, I.., Chen, 7., Li, H., Zhu, X., Cao, Z., et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. arXiv preprint 2t0.:230000191 2023b.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019b.
Wang. W., Zhang, S., Ren, Y., Duan, Y., Li. T., Liu, S., Hu, M., Chen, Z., Zhang, K., Lu, I., et al. Needle in a
13

---

CoMemo: LVLMs Need Image Context with Image Memory
multimodal haystack, arXiv preprint arXiv:2406.07330, 2024c.
for medical visual question answering. arXiv preprint arXiv:2305. 10415, 2023a.
Wang, X., Liu, Y., Shen, C., Ng. C. C., Luo, C., Jin, L.., Chan, C. S., Hengel, A. v. d., and Wang, L. On the general value of evidence, and bilingual scene-text visual question answering. pp. 10126-10135, 2020.
Zhang. Y., Zhang. R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T. Llavar: Enhanced visual instruction tuning for text-rich image understanding, arXiv preprint arXiv:2306. 20222336677711 17107, 2023b.
Wu, C. Pme-casereport. co httptaastssscaaalllioooo-------- 2023.
Zheng, X., Burdick, D., Popa, L., Zhong, X., and Wang, N. R. Global table extractor (gte): framework for joint table identification and cell structure recognition using visual context, pp. 697-706, 2021.
Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. Deepseek-vi2: Mixture- of-experts vasion-language models for advanced multi- modal understanding. arXiv preprint arXiv:2412.10300 2024.
Zhong, Y., Liang, L., Zharkov, I., and Neumann, U. Mmvp: Motion-matrix-bann video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4273-4283, 2023.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 172359991744,,.. 2023.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint vrXhv:2408.01800 2024.
Ye. J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, mplug-ow13: Towards long image-sequence understanding in multi-modal large lan- guage models. arXiv preprint arXiv:2408.0 04840, 2024.
Ying, K., Meng, F., Wang, J., Li, Z., Lin, H., Yang, Y., Zhang, H., Zhang, W., Lin, Y., Liu, S., et al. Mmt- bench: A comprehensive multimodal benchmark for eval- uating large vision-language models towards multitask agi. arXiv preprint arXiv:2404. /6006, 2024.
Yu, I.., Jiang, W., Shi, H., Yu, J., I. Liu, 7., Zhang, Y., Kwok, T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309.
2023.
Yuan, T.. Zhu, Z., Xu, K., Li, C., Mu, T., and Hu, S. A large chinese text dataset in the wild. 34(3):509-521, 2019.
Yuan, Y.. Liu, X.. Dikubab, W. Liu, H., Ji, Z., Wu, Z., and Bai, X. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601, 2022.
Zhang, R., Zhou, Y., Jiang, Q. Song, Q., Li, N., Zhou, K., Wang, L.. Wang, D., Liao, M., Yang, M. et al. Icdar 2019 robust reading challenge reading chinese text signboard. In ICDAR, pp. 1577-1581. IEEE, 2019.
Zhang, X., Wu, C., Zhao, Z. Lin, W., Zhang, Y., Wang, Y., and Xie, W. Pmc-vqa: Visual instruction tuning
14

---

CoMemo: LVLMs Need Image Context with Image Memory
A. Theoretical Derivation of Remote Decay
Definition A.1 (RoPE-induced Inner Product), Given a query vector & E Cd and key vector E Cd, their inner product under Rotary Position Embedding (RoPE) defined as:
4/2-1
(Rmq) (Rm))((R, (Rn,) & Re
Σ
where 1/ denotes the i-th 2D subvector of g, 0, is the preset rotational frequency, and represents complex conjugation.
Building on Definition A.I, we establish the following core lemma:
Lemma A.2 (Abel Transform Representation), Let h, 9p and Sj - E/- e ((m-n)) with boundary conditions ha/2 and So O. Then:
d/2-1
d/2-1
d/2-1
Σ
hee (m- =
Σ 1=
ha(Sit1 - S:) = Σ
S - h;)
2=0
1=
Proof. By Abel summation by parts:
d/2
d/2-1
d/2-1
ha(S++1 - -;) = Σ
h1841
Σ
h
1m0
0m0
4/2
d/2-1
Σ
h S
Σ
h₂S
(index shift)
d/2-1
Σ
S.(h:
14-1)
ha/2-1 h
k=0
d/2-1
S (hi+1 h4) (using boundary conditions)
0
Theorem A.3 (Decay Rate Bound). Under the assumptions of Lemma A.2, the absolute value of the inner product satisfies:
4/2-1
d/2-1
Σ
hge (m-
<
max 00iid/2-11
|h441
his
Σ
|S
Proof. From Lemma A.2 and the triangle inequality:
4/2-1
d/2-1 Σ
Σ
hec (m
S (h+41 hi)
k=0
0=0
d/2-1
s
Σ
S1+1l |P:41 hip
-=
4/2-1
< (max 1 hul)
Σ
S
15

---

CoMemo: LVLMs Need Image Context with Image Memory
B. Training Details
The hyperparameters used for pretraining and finetuning across the three architectures are listed in Table We observed that the 2B model, with its smaller hidden dimension, requires less extensive training, making Phase optional. Therefore, in the ablation experiments in the main text, we only used Pretraining Phase 1 for the 2B model, while the 8B model utilized both pretraining phases.
Table Hyperparameters for Training and Inference
Parameter
Pretraining Phase
Pretraining Phase
Finetuning
Max sequence length
8192
8192
8192
Max tile/image
12
12
12
Optimizer
Adam W x 10-
AdamW W
Adam W 4 10
Learning
1 10
Weight decay
0.01
0.01
0.01
Optimizer momentum
31, B2 0.9, 0.999
81, 82 0.9, 0.999
81, 02 = 0.9, 0.999
Learning rate schedule
Constant with warmup Constant with warmup
Cosine decay
Warmup ratio Training steps
0.03
0.03
0.03
2000
2000
9000
Batch size
1024
1024
1024
Number of mixin layers
4
4
4
Trainable weights
LVLM-S: MLP
All
LVLM-X: Mixin layers MLP
CoMemo: Mixin layers + MLP
(Freeze gate in Phase 2)
C. Detailed Expereiment Results
We provide the detailed ablation study results in Table 9.
Table 9: Detailed results on albation study.
Caption
I Long-Generatin
Multi-Image
Long-Context
Math
General VQA OCR-Related
Model
CoCO
Plickr
No-Capp Co-Caps
LL.VABen.
MMDU
BLINK
Mantis
MMT
MM-NIAH
MileBench
Math Vista
Math
MMBench
MME
MMVP
A12D
CharQQA
TexIVQA
Variant 1 79.1
65.4
60.0
62.9
28.7
38.2
48.3
50.2
27.0
52.1
48
16.5
73.1
1869
31.3
74.3
75.6
74.2
Variant 2 74.1 63.5 64.0
64.0
38,8
42.7
48.8
51.1
30.2
53.8
48.1
15.2
72.8
1899 40
73.4
72.5
72.9
Variant 3 98.0 74.4 75.7
65.2
34.1
51.6
49.4
31.9
54.2
50.1
14.3 73.1 1834
35.3
75.9
76.2 74.6
Variant 4 90.2 67.2 81.9 64.2
39.6
42.7
52.1
50.6
30.6
55.6
49.9
19.4 74.0 1826
36.7
74.2
76.1 74.2
Variant 5 98.6 78.5 78.8 66.9
38.7
43.5
50.6
51.3 34.2
54.6
50.0
17.0 74.2 1904
74.2
73.6 72.6
Variant 6 90.1 73.0 69.9 73.7
48.2
49.2 65.0 57.9 29.8
60.8
58.3
19.1 79.1 2210
45.3
83.5
84.6 78.7
Variant 7 102.0 82.2 92.2 76.2
46.5
49.7 65.4 58.0 38.2
63.4
59.7 18.1 79.8 2182
45.3
83.5 77.2 76.6
Variant 8 92.0 61.7 85.6 42
30.2
40.5
44.2 47.8 15.6
41.9
39.9 17.7 63.8 1674
28
63.2 63.8 58.6
Variant 9 100.5 71.5 96.6 44.8
32.0
40.6 48.8 49.2
17.5
45.6
44.2 13,2 67.6 1759
31.3
63.3
64.7 62.6
D. Dataset Details
The data used in the pre-training stage are listed in Table 6. And datasets used for instruction tuning are listed in Table 7.
16

---

CoMemo: LVLMs Need Image Context with Image Memory
Table 6: Summary of datasets used in the pretraining stage.
task
dataset
Short Caption
Laion (en&zh) (Schuhmann et al., 2022a), COYO (Byeon et al., 2022), COCO (Lin et al., 2014b)
OCR
Wukong-OCC (Gu et al., 2022), LaionCOCO-OCR (Schuhmann et al., 2022b)
Detection
GRIT (Peng et al., 2023), Objects365 (Shao et al., 2019)
Conversation
All-Secing (en&zh) (Wang et al., 2023b)
Image-text instruction data
(see Table 7)
Table 7: Summary of datasets used in the instruction tuning stage.
task
dataset
General QA
VQAv2 (Goyal al., 2017), GQA (Hudson Manning, 2019), OKVQA (Marino et al., 2019), VSR (Liu al., 2023a)
A12D (Kembhavi et al., 2016), ScienceQA (Lu al., 2022a), Chemistry Data (Li et al., 2024)
Science
TQA (Kembhavi al., 2017)
PMC-VQA (Zhang et al., 2023a). VQA-RAD (Lau et al., 2018), VQA-Med (Ben Abacha al., 2019)
Medical
Medical-Diff-VOA (Hu et al., 2023), PathVQA (He et al., 2020),
SLAKE (Liu et al., 2021), PMC- RepcCaasReeoorttt (Wu, 2023)
ChartQA (Masry al., 2022a), LRV-Instruction (Liu al., 2023b), PlotQA (Methani et al., 2020)
Unichart (Masry et al., 2023), MMC-Inst (Liu et al., 2023c), DVQA (Kafle al., 2018)
Chart
TableMWP (Lu et al., 2022b), FigureQA (Kahou al., 2017). MapQA (Chang al., 2022)
SciTSR (Chi et al., 2019), Fintabnet (Zheng et al., 2021)
CLEVR (Johnson al., 2017), MetaMath (Yu et al., 2023), GeoQA+ (Cao & Xiao, 2022)
Mathematics
Geometry3k (Lu et al., 2021), GeoS (Seo et al., 2015), Unigeo (Chen et al., 2022)
Super-CLEVR (Li et al., 2023), MathQA (Amini et al., 2019)
Ar1500k (Mao al., 2017) 7), MovieNet (Huang et al., 2020), KonIQ-10k (Hosu et al., 2020)
Knowledge
KVQA (Shah al., 2019), ViQuAE (Lerner al., 2022)
Info (Mathew et al., 2022). TexLVQA (Singh al., 2019a), ArT (Chng et al., 2019)
CASIA (Liu et al., 2011) 1), Chart-to-text (Kantharaj et al., 2022), COCO-text (Veit et al., 2016)
CTW (Yuan et al., 2019), EATEN (Guo et al., 2019), ICDAR2019-LSVT (Sun et al., 2019)
OCR
ICPR MTWI (He et al., 2018) 8), NAF (Davis et al., 2019), ReCTS (Zhang et al., 2019)
Tex1OCR (Singh et al., 2021), LLaVAR (Zhang et al., 2023b), HME-100K (Yuan et al., 2022)
POIE (Kuang et al., 2023), SROIE (Huang et al., 2019), ST-VQA (Biten et al., 2019)
EST-VQA (Wang al., 2020), IAM (Marti & Bunke, 2002)
Document
DocVQA (Clark & Gardner, 2017), DacReason25k (Hu al., 2024)
RefCOCO (Kazemzadeh al., 2014), RefCOCO+ (Kazemzadeh al., 2014), ReiCOCOg (Kazemzadeh al., 2014)
Grounding
RD-BoxCoT (Chen al., 2023)
ALLaVA (Chen et al., 2024a), LAION-GPT4V (LATON, 2023)
Conversation
MMDU (Liu et al., 2024d), TexICCR-GPT4V (Carter, 2024)
Detection
Objects365 (Shao et al., 2019), V3Det (Wang et al., 2023a)
17


Reviews:
[{"review_id": "Reg7s2ya5g", "reviewer": "Reviewer_vTuL", "summary": "This paper introduces CoMemo, provides two key design choices into MLLMs, 1) adding additional cross-attn like Flamingo, besides the origianl llava-style approach, but to mask part of the info to prevent over-reliance 2) add 2d rope to LLM backbone for image features. The approach ourperforms llava-style and flamingo-style. The author also provides detailed study into MLLMs: 1) study of the lost in the middle phenomenon 2) The remote decay of rope 3) the balance between two pathways", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback to improve our work's clarity. As you rightly pointed out, proposing a framework that combines Flamingo-style and LLaVA-style approaches represents a highly impactful contribution. Thank you for recognizing the value of our work.\nBelow, we address your concerns point by point, and all suggested revisions will be incorporated in the next version. If you feel we have adequately addressed your feedback, we would be most grateful if you could consider adjusting your evaluation score accordingly. Thank you for your time and consideration.\n\n`Q1: I still dont know what is the detailed calculation of gates avg.`\n\nIn L143 of the original manuscript, we have already mentioned how the avg. gates are calculated:\n> Our analysis averages the attn_gate and ffw_gate values to quantify pathway preference.\n\nSpecifically, the average gates value is obtained by calculating the mean of the absolute values of the gate values from each mixin layer, corresponding to the data shown in the line chart in Figure 5. \n\n`Q2: Authors mentioned attn_gate and ffw_gate at Section 2.3, but first introduce it in 3.2.`\nThis is because attn_gate and ffw_gate are not novel concepts introduced in our paper. However, we believe that adding citations when first mentioning these terms, or restructuring the content could significantly reduce potential confusion.\n\n`Q3: The writing of Balance in DHR Allocation is also unclear, the author should point out that 1k, 2k, 4k means steps.`\nThey appear in Figure 5, as we stated in the caption of Figure 5:\n> pretrained checkpoint corresponding to the x-axis.\n\nThe values 1k, 2k, and 4k indeed denote to the training steps during pretraining. \n\n`Q4: Algo1's format`\n\nWe apologize for the poor reading experience caused by the formatting issue. In the next version, we will adjust the layout to alleviate the impact of the line-breaks on readability.\n\n`Q5: The author can also test more populat benchmark like mm-vet`\n\nSure! As evident from the [Table 4](https://drive.google.com/file/d/1_OsHi8-Ahy1YhaYpS6NK98q7ZOE-N0QM/view?usp=sharing), since MMVet is a small-scale evaluation set (200 Q&A pairs), CoMemo does not significantly outperform LVLM-S in overall score. However, it still demonstrates superiority in certain dimensions emphasizing visual perception (Recognition/Spatial/Math).\n\n`Q6: It seems the rope-2d is harmful to OCR in Table5, Could you provide some justification on this point?`\n\nWe have previously addressed this trade-off in the manuscript (L361-367, L392-394). However, we now propose an improved RoPE-2D(no-overlaps) variant that achieves more balanced improvements across all capabilities.\n\nWhy RoPE-2D affects OCR performance? The performance trade-off occurs because we compress positional encoding IDs to mitigate the remote decay problem. While this compression significantly benefits tasks that don't require fine-grained image information, it inevitably reduces information richness for OCR tasks. For instance, in ChartQA evaluations with a maximum number of 12, our compression maps 3,328 image tokens to only 256 RoPE position IDs.\n\nThe computation of RoPE-2D (no-overlaps) involves the following two steps:\n\n1. Subimage position IDs are mapped to their corresponding thumbnail patch positions.\n(In the initial version of RoPE-2D, all subimage position IDs matched their corresponding thumbnail patch IDs. Here, however, they are sequentially incremented starting from the thumbnail patch IDs.)\n\n2. Thumbnail position IDs accumulate based on the number of mapped patch tokens\nAs shown in [Table 2](https://drive.google.com/file/d/1lXmFL1Nl0YYsIu6FStLt7v7VjMFUm1f5/view?usp=sharing), this approach demonstrates improvements across all evaluation metrics compared to the baseline (LVLM-S vs CoMemo+RoPE2D(no-overlaps)). However, it shows significantly weaker performance on long-context and generation tasks compared to the compressed RoPE-2D version.\n\n`Q7: Can CoMemo still outperforms -S variant if applying rope-2d to -S variant.`\n\nYes, RoPE-2D can indeed be applied to the -S variant. \nAs shown in Table 2, our 2D positional encoding reconstruction enhances LVLM-S's performance across multiple capabilities: multi-image processing, general VQA, and image captioning tasks. This approach also improves long-context understanding and extended text generation by mitigating the remote decay issue through compressed positional encoding. However, as previously noted, we observed some degradation in OCR performance. Due to time constraints, we were unable to conduct experiments with the RoPE-2D(no-overlaps) variant. Based on the consistent improvements demonstrated in Table 2, we anticipate this variant would similarly provide stable performance gains across various capabilities.\n\nWe appreciate the constructive comments from the reviewer regarding RoPE-2D. We believe that these experiments and analyses will help followers gain a more comprehensive understanding of RoPE-2D.", "claims_and_evidence": "See below", "methods_and_evaluation_criteria": "See below", "theoretical_claims": "See below", "experimental_designs_or_analyses": "See below", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The writing is very unclear, and it's a bad experience to read this paper, an example is that the authors mentioned attn_gate and ffw_gate at Section 2.3, but first introduce it in 3.2. The writing of Balance in DHR Allocation is also unclear, the author should point out that 1k, 2k, 4k means steps. and I still dont know what is the detailed calculation of gates avg. Algo1 should be format using a smaller font size, now the line-break is very weird.\nThe experiments are solid, and the results are promising, the prosoed methods consistently outperforms -X and -S variants, this paper provides a promising solution to integrate flamingo and llava style MLLMs.", "other_comments_or_suggestions": "There is a long standing debate between the flamingo-style and llava-style, It's very exciting to see a good solution to leverage both advantages. The experiments results are good, while the paper is not ready, and the writing should be improved. I think with major revision, this paper can be a very good and impactful work, but the current submission is not there yet.", "questions_for_authors": "To test free-form responses, the author can also test more populat benchmark like mm-vet, can authors provide justifications on this point.\nIt seems the rope-2d is orthogonal to the -S variant, can CoMemo still outperforms -S variant in Table 2, Table3 and Table4 if applying rope-2d to -S variant.\nIt seems the rope-2d is harmful to OCR in Table5, also in Table 4, CoMemo is worse than -S variant, Could you provide some justification on this point?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SOIHq9EjhD", "reviewer": "Reviewer_HPwo", "summary": "This paper thoroughly investigates the flaws of LLM architectures when processing multimodal inputs, including the progressive neglect of central visual content as context expands and the failure of conventional positional encoding schemes in preserving 2D structures. To address these issues, this paper presents CoMemo to decouple the memory path for mitigating visual neglect and RoPE-2D to maintain 2D spatial awareness. The experimental results indicate the effectiveness of the proposed methods in multiple benchmarks.\n\n###update after the rebuttal#####\n\nThanks to the authors for responding to my arguments in this paper. The newly added experimental results in the provided link are sufficient. It addressed my concerns. I tend to raise my score.\n\n############################", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your recognition of our work and your positive comments regarding the well-orgnized and insightful findings of our paper. \nBelow, we will address your concerns point by point, and all suggested revisions will be incorporated into the next version of the manuscript. If our responses adequately address your concerns, we would sincerely appreciate it if you could consider adjusting your evaluation score. Thank you again for your time and thoughtful review.\n\n`Q1: The performance of CoMemo still has significant gaps compared with other models on certain benchmarks. What is the reason?`\n\nThe main reason for CoMemo's performance gap compared to other models is the training data. As seen in the evolution of open-source models like InternVL, QwenVL, and LLaVA, architectural improvements were largely driven by data strategies, rather than changes in architecture. Differences in training data can obscure architectural distinctions, complicating the selection of the optimal model. As discussed in Section 4.2, the goal of this paper is to conduct an ablation study on model architectures using the same dataset, not to maximize absolute performance. Comparing with SOTA models highlights that our training dataset is not based on toy datasets, demonstrating the generalization ability of our results.\n\n`Q2: In Table 1, the reviewer noticed that lots of results are missing for the majority of the models. What is the reason?`\n\nWe apologize for the confusion caused by the missing values in Table 1. Due to time constraints, we were unable to conduct a comprehensive evaluation, but we ensured that most benchmarks have at least three or more models for comparison in this [url](https://drive.google.com/file/d/1jCKPKPMcAjnJc_1FCwqQsSN9W9luK7qP/view?usp=sharing). Additionally, we will consider including a comparison of different models at the 8B scale in the next version.\n\n`Q3: The authors only employ a small-scale model, i.e., InternLM-1.8B, to conduct experiments, lacking the verification of the universality of the proposed schemes.`\n\nThank you for your constructive and insightful suggestions. We agree that evaluating our framework at larger scales can better demonstrate its effectiveness. Therefore, we have conducted additional experiments at the 7B scale see [Table 3](https://drive.google.com/file/d/1O6jIbebOjUmgPlfTAa1ytDVRuY_1IDTt/view?usp=sharing).\n\nFor the 7B-scale model, we adopted InternLM2-7B as the language backbone and InternViT-300M as the visual encoder. The results at this scale are largely consistent with those observed in the 2B experiments, further validating that our architecture follows the scaling law. As shown, performance improved across most vision tasks at the 7B scale. However, we observed a slight drop in performance on OCR-related tasks, which we attribute to the compression characteristics of our RoPE-2D positional encoding.\n\nTo address this issue, we also propose a variant called RoPE-2D (no-overlap). Our experiments at the 2B scale show that this variant provides more stable improvements across different tasks in [Table 2](https://drive.google.com/file/d/1lXmFL1Nl0YYsIu6FStLt7v7VjMFUm1f5/view?usp=sharing). Due to time and resource constraints—training a 7B-scale model on our dataset requires nearly two days with 128 A100 GPUs—we were unable to explore more positional encoding strategies or conduct further large-scale experiments at this time. However, we plan to include more extensive evaluations to further demonstrate the generalizability of our method in the next version.\n\n`Q4: Can the authors provide more explicit analyses and visualizations to indicate the effective mechanism of the proposed methods?`\n\nIn [Figure 1](https://drive.google.com/file/d/1gMQOcUC4qkBur4RVMLw6PbDCUC8JtQt0/view?usp=sharing), we compare the average gradients of input image tokens between LVLM-S and CoMemo during inference across different benchmarks. Specifically, we compute the gradients with respect to the input tokens based on the logits corresponding to the model's response token IDs, then extract the image tokens from the indexed results. Since we only consider the magnitude of influence, we take the absolute value of the computed gradients before averaging across all image tokens to obtain the final result. \n\nThe comparison shows that through architectural adjustments and positional encoding modifications, the model's responses indeed demonstrate stronger focus on visual information, which verifies our mitigation of the image neglect problem.\n\n`Q5: Missing a space between words in the abstract part. Please pay attention to keeping consistent decimal places in experimental results.`\n\nThank you for pointing out the formatting issues in our paper. We will address these problems and make the necessary corrections in the next version.", "claims_and_evidence": "The claims are reasonable.", "methods_and_evaluation_criteria": "The employed benchmarks are comprehensive and reasonable for evaluating performance.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The authors only employ a small-scale model, i.e., InternLM-1.8B, to conduct experiments, lacking the verification of the universality of the proposed schemes. The authors are advised to verify the effectiveness across multiple architectures to ensure reliability.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "This paper is well-organized, and the listed key findings are insightful.", "other_comments_or_suggestions": "Missing a space between words in the abstract part.\n\nPlease pay attention to keeping consistent decimal places in experimental results.", "questions_for_authors": "1. In Table 1, the reviewer noticed that lots of results are missing for the majority of the models. What is the reason?\n2. The performance of CoMemo still has significant gaps compared with other models on certain benchmarks. What is the reason?\n3. Can the authors provide more explicit analyses and visualizations to indicate the effective mechanism of the proposed methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UmYHgYuTen", "reviewer": "Reviewer_PsDG", "summary": "The paper introduces CoMemo to improve visual information retention in multimodal tasks. Specifically, CoMemo includes a memory path for image tokens that operates independently of the main text path. This helps prevent visual information loss during long-context reasoning. Then CoMemo uses RoPE-2D encoding that maintains 2D spatial relationships in high-resolution images. This reduces performance degradation in long sequences. Finally, CoMemo uses the memory mixin strategy during training to ensure both the context path and memory path contribute effectively.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1: The dual-path architecture and RoPE-2D introduce additional computational overhead, potentially impacting real-time applications.`\n\nWhile CoMemo introduces some computational overhead, we have designed mechanisms to minimize latency, making the time difference between CoMemo and LVLM-S nearly negligible.\n\n1. Shared image token representation: Both the memory and context paths share the same image token representations, so the ViT image encoding is performed once without adding latency.\n\n2. Fewer cross-attention layers: Unlike Flamingo and MLLama-3.2, which insert cross-attention after each transformer block, we use a 4:1 ratio, inserting only 6 cross-attention layers in InternLM2 (24 layers in total). Cross-attention computes interactions only between input tokens and image tokens, reducing computation since the image token sequence is shorter than the input sequence.\n\n3. No KV-cache required for cross-attention: Cross-attention eliminates the need for key-value caching during inference. Decoding only requires computing the query for the current token, whereas self-attention involves additional memory for KV-cache and quadratic time complexity (O(N²)).\n\nAs shown in [Table 4](https://drive.google.com/file/d/1hcnKbAzDiZnCSX9GFJY2l2jObVkDYcDa/view?usp=sharing) and [Table 5](https://drive.google.com/file/d/1MPYN2gf7sCnyZA8os0DGaSsDidYKqxX2/view?usp=sharing), we report training and inference efficiency across different architectures. The results confirm that CoMemo has nearly the same latency as LVLM-S. Although LVLM-X achieves higher efficiency due to using fewer image tokens, its performance is significantly weaker than both CoMemo and LVLM-S.\n\n`Q2: RoPE-2D’s compression may degrade performance in fine-grained tasks like OCR.`\n\nThis is an important point, and we did not provide a detailed discussion in the original manuscript. The compression in RoPE-2D is a trade-off between improving long-context and long-generation tasks while potentially degrading performance on fine-grained tasks like OCR.\nWe propose a variant, RoPE-2D (no-overlap), which removes the compression. This version shows consistent improvements across various benchmarks, including OCR tasks.\nWhy does RoPE-2D affect OCR performance?\n\n1. OCR is a fine-grained visual task that requires detailed information. The compression of positional information in RoPE-2D can reduce performance on tasks needing high resolution.\n\n2. In OCR evaluations, excessive compression may occur, especially when the dynamic number for DHR (set based on InternVL and VLMEvalKit) is large. For example, in ChartQA, the compressed positional encoding causes 3328 image tokens to correspond to only 256 position IDs.\n\nWhat if we map positional relationships without compression? \n\nWe explored RoPE-2D (no-overlap), where:\n\n1. Subimage position IDs are mapped to thumbnail patch positions.\n\n2. Position IDs accumulate based on the number of mapped patch tokens.\nIn this variant, position IDs are incrementally assigned, ensuring uniqueness across subimages. As shown in [Table 2](https://drive.google.com/file/d/1lXmFL1Nl0YYsIu6FStLt7v7VjMFUm1f5/view?usp=sharing), this approach improves all evaluation metrics compared to the baseline. However, it performs worse on long-context and generation tasks compared to the compressed RoPE-2D version.\nWe will include a more detailed analysis of RoPE-2D in the next version.\n\n`Q3: The authors show that LVLMs have the \"lost in the middle\" phenomenon with gradient and attention weights in Fig 3. I would like to know if the proposed method truly improves these aspects and if it can provide clearer visual results.`\n\nThank you for this insightful question. We analyze the “lost in the middle” phenomenon from two perspectives and evaluate CoMemo’s improvements in these areas:\n\n1. Attention Weights: CoMemo retains the self-attention mechanism from LVLM-S, so the image attention remains largely unchanged. However, CoMemo introduces an additional cross-attention mechanism, allowing input tokens to attend to image tokens, providing explicit visual grounding. As discussed in Section 2.3, the average gate value quantifies the strength of this visual attention, which is unaffected by the input context and thus does not suffer from the “lost in the middle” issue.\n\n2. Image Token Gradients: In [Figure 1](https://drive.google.com/file/d/1gMQOcUC4qkBur4RVMLw6PbDCUC8JtQt0/view?usp=sharing), we compare the average gradients of image tokens during inference between LVLM-S and CoMemo across different benchmarks. We compute the gradients of output logits with respect to input image tokens, take the absolute value, and average across all image tokens. The results show that CoMemo significantly strengthens visual grounding. On the MMNIAH benchmark, where the “lost in the middle” issue is prominent, image token gradients in CoMemo nearly double compared to LVLM-S, clearly indicating that CoMemo mitigates this problem.\n\nWe will correct the typos issue in the next version.", "claims_and_evidence": "This paper is well-written, systematically introducing the limitations of existing models before presenting effective solutions. For instance, to illustrate the \"lost in the middle\" problem, the authors provide clear evidence through gradient heatmaps and evaluation results on long-context benchmarks, demonstrating CoMemo's improved performance over LVLM-X and LVLM-S.", "methods_and_evaluation_criteria": "The evaluation process used in this paper follows a popular setup.", "theoretical_claims": "The theoretical claims presented in the paper are solid.", "experimental_designs_or_analyses": "The experimental setup is reasonable and the authors give sufficient analysis.", "relation_to_broader_scientific_literature": "The authors introduce their proposed method by addressing three key challenges: the \"lost in the middle\" phenomenon, \"remote decay in dynamic high-resolution models\", and \"the balance between two pathways.\" The \"lost in the middle\" issue has been identified in previous studies [1,2], while \"remote decay in dynamic high-resolution models\" is a well-recognized challenge in the field.\n\n[1] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua,M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts.\n[2] Song, D., Chen, S., Chen, G. H., Yu, F., Wan, X., and Wang, B. Milebench: Benchmarking mllms in long context.", "essential_references_not_discussed": "The references are comprehensive and well-organized.", "other_strengths_and_weaknesses": "* RoPE-2D’s compression may degrade performance in fine-grained tasks like OCR. \n* The dual-path architecture and RoPE-2D introduce additional computational overhead, potentially impacting real-time applications.", "other_comments_or_suggestions": "“2.3. The balance between tow pathways” -》 “2.3. The balance between two pathways”", "questions_for_authors": "* RoPE-2D’s compression may degrade performance in fine-grained tasks like OCR. This doesn't seem very reasonable. Does the authors have any opinion on it?\n* The authors show that LVLMs have the \"lost in the middle\" phenomenon with gradient and attention weights in Fig 3. I would like to know if the proposed method truly improves these aspects and if it can provide clearer visual results.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n4BcUOVoC0", "reviewer": "Reviewer_t4ai", "summary": "- This paper proposes CoMemo, a hybrid architecture of LLaVA and Flamingo’s cross-attention module.\n- It also has a 2D positional encoding mechanism (RoPE-2D) to better preserve spatial relationships, addressing the issue of visual information neglect in long contexts.\n- Experimental results across multiple benchmarks indicate that CoMemo improves performance on tasks requiring long-context comprehension and multi-image reasoning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`Q1: CoMemo is essentially a hybrid of LLaVA and cross-attention (Flamingo-style). `\n\nFirst, neither open-source nor closed-source models currently offer such an architecture. As Reviewer vTuL noted, \"It is exciting to see an effective solution that leverages the advantages of both approaches.\" Furthermore, in L45-46 of our paper, we emphasize that a naive combination of these two structures would fail. It is precisely the novel techniques we introduce—beyond simple hybridization—that differentiate CoMemo from merely hybrid cross-attention and LLaVA framework.\n\n`Q2: The same architecture was introduced earlier, e.g., Q-Former in BLIP-2.`\n\nThe most fundamental reason is that the Q-Former is essentially a projector. In BLIP, the representations between text tokens and image tokens are still updated via self-attention, which differs significantly from CoMemo’s approach of directly using cross-attention for representation updates. Therefore, BLIP is not orthogonal to LVLM-S—in essence, it is a special case of the LVLM-S architecture. \n\n`Q3: The authors claim to be the first to extend RoPE to 2D, but this is incorrect. M-RoPE means no additional computational cost.`\n\nThis comment misinterprets our claim and overlooks an important technical distinction. As stated in L26–L32 and L82–L84 of our paper, we introduce a RoPE-based 2D encoding scheme specifically designed for the DHR (Dynamic High-Resolution) technique.\n\nKey differences from M-RoPE are as follows:\n\n1. LLM Training Requirements: M-RoPE operates in 3D space, requiring dedicated training. In contrast, our RoPE-2D uses 1D computation, directly reusing the original RoPE from the LLM without additional alignment training.\n\n2. Resolution Schemes: M-RoPE was designed for Naive Dynamic Resolution, not DHR. In Qwen-VL, RoPE replaces ViT’s absolute position encoding to support varying resolutions, but this requires extra training. \n\n3. Additional RoPE Cache Computation: M-RoPE’s 3D RoPE requires cache computation for each dimension, which adds complexity.\n\n`Q4: Missing values in Table 1.`\nThis table is only intended to show that our experimental data is not from a small-scale dataset. Additionally, we have now completed the [most missing values](https://drive.google.com/file/d/1jCKPKPMcAjnJc_1FCwqQsSN9W9luK7qP/view?usp=sharing).\n\n`Q5: Qwen2-VL-2B explicitly reports an MMMU score of 41.1 in its paper, but the authors omit this unfavorable result.`\nThe benchmark we evaluated does not include MMMU. \n\n`Q6: The appropriate comparison for InternVL-2.5 should be MiniCPM V2.5 or V2.6.`\nSince the CoMemo model in our main experiments is at the 2B scale, and MiniCPM V2.5 and V2.6 are 7B-scale LVLMs, we chose to compare with MiniCPM-V-2, which is closer in model size.\n\n`Q7: Too few baselines are considered. For instance, BLIP [2] and BLIP-2 [1], which are closely related, should be included.`\n\nWe selected the two baselines because they are currently the only two mainstream architectures in this field. As for why we did not choose BLIP as a baseline, there are two main reasons:\n\n1. The BLIP architecture can be considered a refined form of the LLaVA-like architecture and should essentially be covered under the LVLM-S framework.\n\n2. Previous studies have shown that, with the same data scale, the performance of Q-Former in BLIP is inferior to that of MLP[1, 2]. In BLIP-3, Q-Former is no longer used[3].\n[1] DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models\n[2] Honeybee: Locality-enhanced Projector for Multimodal LLM\n[3] xGen-MM (BLIP-3): A Family of Open Large Multimodal Models\n\n`Q8: Questions about mask during pretrain.`\n`Q8.1: How is the masked information selected?`\n\nWe use a random masking strategy (L261–267) to remove image information by masking image tokens post-tokenization.\n\n`Q8.2: Since the approach first introduces image context and then applies random dropout, doesn’t this introduce inefficiency?` \n\nL186–189 clarify that the Memory Path and Context Path reuse the same image representations, incurring no extra computation. The minor latency from dropout (masking) in the Memory Path is negligible.\n\n`Q8.3: Why mask the cross-attention path while treating the autoregressive path as the primary method?`\n\nIn L13–20, we briefly introduced and cited findings from Idefics-2, indicating that under the same parameter count and training data, the LVLM-S approach outperforms LVLM-X.\nIn Section 2.3 of the paper, we further conducted an ablation study on this. As shown in Figure 5, as the number of pretraining steps increases, the model exhibits stronger reliance on the cross-attention path, yet this leads to suboptimal performance.", "claims_and_evidence": "The authors claim their contributions are equally split (50% each), but both are weak:\n\n1. Novelty of CoMemo: The authors present CoMemo as a novel architecture, but it is essentially a hybrid of LLaVA and cross-attention (Flamingo-style). While hybridity itself isn't an issue, the same architecture was introduced earlier, e.g., Q-Former in BLIP-2 [1] (published January 2023), which is neither cited nor included in comparisons.\n2. Extending RoPE to 2D: The authors claim to be the first to extend RoPE to 2D, but this is incorrect. The concept was already introduced as M-RoPE in Qwen2-VL (published September 2024), which they cite and include in Table 1. They attempt to differentiate their work in lines 412–418, arguing:\n    \n    > \"In contrast, Qwen-VL2 (Wang et al., 2024b) introduced M-RoPE, a multi-dimensional position encoding extending RoPE to three channels (temporal, height, width) for im- ages and videos. However, this increases computational costs, tripling the original RoPE overhead and introducing redundancy by requiring these channels for text tokens.\"\n    > \n    \n    However, this claim is incorrect. The Qwen-VL2 paper explicitly states that when processing 2D images, the temporal dimension remains constant, meaning no additional computational cost:\n    \n    > \"When processing images, the temporal IDs of each visual token remain constant, while distinct IDs are assigned to the height and width components based on the token’s position in the image.\"\n    > \n3. Additionally, the evaluation is weak, with few baselines, missing values in Table 1, and unclear figures (e.g., missing axis ranges in Figure 1). Overall, the work requires significant improvement.", "methods_and_evaluation_criteria": "Lines 51–52 state: \"masks out part of the image information in the mixin layer to prevent over-reliance on this path,\" raising the following questions:\n\n- How is the masked information selected?\n- Since the approach first introduces image context and then applies random dropout, doesn’t this introduce inefficiency? Was this design choice specifically considered?\n- Why mask the cross-attention path (MixinLayer) while treating the autoregressive path as the primary method for incorporating image context, rather than the reverse?", "theoretical_claims": "- It is unclear how Findings 1–3 in Section 2 are derived.\n- The main claims are questionable; see the *Claims and Evidence* section above for details.", "experimental_designs_or_analyses": "- Table 1 has too many missing values. For example, Qwen2-VL-2B explicitly reports an MMMU score of 41.1 in its paper, but the authors omit this unfavorable result.\n- The comparison includes an outdated version of MiniCPM (V-2); the appropriate comparison for InternVL-2.5 should be MiniCPM V2.5 or V2.6.\n- Too few baselines are considered. For instance, BLIP [2] and BLIP-2 [1], which are closely related, should be included.", "relation_to_broader_scientific_literature": "See below the ‘Essential References Noe Discussed’ section", "essential_references_not_discussed": "The following key literatures are missing:\n\n[1] Li, J., Li, D., Savarese, S. and Hoi, S., 2023, July. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning (pp. 19730-19742). PMLR.\n\n[2] Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888–12900. PMLR, 2022.", "other_strengths_and_weaknesses": "Despite the weaknesses in theoretical claims, literature coverage, and evaluation results, the overall writing is poor and unpolished, resulting in broken logic and a lack of clarity.", "other_comments_or_suggestions": "- Figure 1 is difficult to interpret without axis scales.\n- If the Mixin layer is essentially cross-attention, renaming it adds unnecessary complexity.\n- **Typo:** l.137 – *\"tow pathways\"* → *\"two pathways\"*\n- **Missing reference:** l.155 – *\"As shown in 5\"* should specify the correct figure number.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CoMemo, a dual-path architecture addressing the "lost in the middle" issue in VLM. This extra branch processes image tokens through a mixin layer to preserve central visual details. It also introduces a RoPE‑2D to enhance 2D spatial relationships. Experiments show this approach enhances long-context reasoning and multi-image understanding in LVLMs.

The paper received 2 weak accept, 1 accept, and 1 weak reject. After carefully read the paper, the reviews and the rebuttal/feedback, the ACs decided to put the paper as `weak accept`, considering the following points can be improved further:

1. Presentation clarify, as pointed out by reviewer `vTuL`.

2. The training data details seem not provided; not sure if the comparison with the other 2B models is fair.

3. The selection of benchmarks appears limited and selective, which weakens the convincingness of the reported improvements; expanding the evaluation to include a wider range of tests—and incorporating more 7B-scale model results—would better demonstrate the effectiveness and scalability of the proposed modules.
Decision: Accept (poster)