MMReD: a Cross-Modal Benchmark for Dense Context Reasoning
Authors: Maxim Kurkin
Boris Shirokikh
Irina Abdullaeva
Viktoriia A. Chekalina
Andrey Kuznetsov
Venue: ICLR 2026 Poster (2026)
Keywords: long context
reasoning
LLM
LVLM
MLLM
benchmark
Primary area: datasets and benchmarks
TL;DR: We present MMReD, a benchmark revealing that state-of-the-art LLMs and LVLMs struggle with dense multi-modal reasoning over long contexts, highlighting critical architectural limitations and the need for fundamentally new approaches.

Abstract: Despite recent advancements in extending context windows of large language models (LLMs) and large vision-language models (LVLMs), their ability to perform complex multi-modal reasoning over extended contexts remains critically limited. To underline this challenge, we present \textbf{MMReD}, a benchmark specifically designed to assess reasoning abilities within dense, information-rich scenarios where simple retrieval is not enough. Unlike traditional Needle-in-a-Haystack evaluations, MMReD challenges models to identify and interpret global patterns across entire contexts. Our benchmark comprises 24 tasks of varying complexity, ranging from standard passkey retrieval setups to those requiring selective or uniform attention to all context chunks. The evaluation reveals a consistent performance drop across all tested models -- including the most advanced LLMs, LVLMs, and architectures specializing in code and reasoning -- as the number of observations increases. Notably, even the leading reasoning-specialized models achieve 0\% accuracy on certain tasks at the maximum context length of 128 observations. Conventional fine-tuning techniques, such as SFT and GRPO, also fail to generalize effectively to longer contexts. These observations reveal an inherent limitation in current model architectures, emphasizing the need for innovative approaches to enable competent dense context reasoning in multi-modal AI systems.

Published as a conference paper at ICLR 2026
MMRED: A CROSS-MODAL BENCHMARK FOR DENSE CONTEXT REASONING
Maxim Kurkin .1,2, Boris Shirokikh"!, Irina Abdullaeva! 1.3 Viktoriia Chekalina 144 & Andrey Kuznetsov
FusionBrain Lab, AXXX, Moscow, Russia
2 AApplied AI Institute, Moscow, Russia
3 RResearch Center of the Artificial Intelligence Institute, Innopolis University, Innopolis, Russia
"I "Lomonosov Moscow State University, Moscow, Russia
ABSTRACT
Despite recent advancements in extending context windows of large language models (LLMs) and large vision- languaee models (LVLMs), their ability to per- form complex multi-modal reasoning over extended contexts remains critically limited. To underline this challenge, we present MMReD, benchmark specifi- cally designed assess reasoning abilities within dense, information-rict scenar- ios where simple retrieval is not enough. Unlike traditional Needle-in-a-Haysthe evaluations, MMReD challenges models to identify and interpret global patterns across entire contexts. Our benchmark comprises 24 tasks of varying complex- ity, ranging from standard passkey retrieval setups to those requiring selective or uniform attention to all context chunks. The evaluation reveals a consistent per- formance drop across all tested models including the most advanced I.L.s, LVLMs, and architectures specializing in code and reasoning observations increases. Notably, even the leading reasoning -ssooiinn-ppeializeed models achieve 0% accuracy on certain tasks the maximum context length of 128 ob- servations. Conventional fine-tuning techniques, such as SFT and GRPO, also fail generalize effectively to longer contexts. These observations reveal an inherent limitation in current model architectures, emphasizing the need for innovative ap- proaches enable competent dense context reasoning in multi-modke systems.
the number of
INTRODUCTION
sequence length N 11.2 8,8, 26. 32, 64, 128
Question: How many characters acterss were the Bathroom when Mary first appeared the Kitchen?
Answar: 3
Preciction: ('ansver" '3")
Question
(i)?
Precicton: ('ansuer:
Question: Which room was empty for the least amount steps?
Answer: Bathroom
Precictio:: ( ('ansver: "Hallway
Figure Overview of multi-modal dense- sesueee reasoning task from MMReD, Models are evaluated on queries that require tracking entities, spatial relationships, basic counting, and event- based reasoning over varying sequence lengths N, evaluating the models' ability process and retain long-term dependencies. Example questions cover all designed answer types: numbers, char- acters, and rooms. In the images and questions, we replace i, + 1, etc, with the real step numbers.
"Equal contribution. Correspondence to: kurki fusi onbra n1ab.

---

Published as a conference paper at ICLR 2026
Large language models (LLMs) have demon- strated a remarkable ability to reason in both short and long contexts. Long-context exten- sion, in particular, has gained significant at- tention models increasingly deployed in tasks requiring memory, retrieval, and sequen- tial reasoning.
Standard practice on reasoning evaluation is to rely on range of public benchmarks that test reasoning capabilities on a variety of topies. Short-context reasoning includes general language understanding and reading comprehension (DROP (Dua al.) 2019),
20-
ARC (Clark et al.) 2018), BBH (Suzgun 2022)), academic reasoning (MMLU- Wang et al.] 2024dh, GPQA (Rein
et al.
PRO
Figure 2: All evaluated LLMs and LVLMs share
ct al.
2024), MATH (Hendrycks et al. 2021),
common performance profile, decreasing with the context length of MMReD tasks. The results are grouped and averaged across question types.
AIME (MAA 2024) and code comprehension
(HumanEval (Chen et al.] (022), CRUX (Gu
al.
2024). MBPP (Austin et al.) 2021)).
Long-context reasoning includes recent bench-
marks RULER (Hsich al. 2024). BABILong (Kuratov al. 2024), and Michelangelo (Vodrahalli
et al. 2024
Large vision-language models (LVLMs) are also rapidly closing the gap in performance on var- ious visual recognition tasks that require reasoning in both text and visual modalities. Existing benchmarks can be categorized by modality of context in the question: single image understand- ing (MMMU Yue et al. 2024a b)). multiple image understanding ( (MuirBench (Wang et al. (025).
BLINK (uu et al. 2024b)), and video understanding (VidcoMME (Fu et al. (002a). MLVU (Zhou al. 2024), EgoSchema (Mangalam et al. 2023). LVBench (Wang et al. 2024b)).
While these benchmarks demonstrate improved performance on longer contexts, they primarily fo- cus, as show below,
serrevall-bases Needle-ing ieeee--in-----ae in-a-Haystack (NIAH) setups, where models locate a specific fact from an otherwise irrelevant or distractor-fille context. However, such tasks do not fully capture model's ability we show that state-of-the-art LLMS and I.V.Ms exhibit no clear correlation between their NIAH performance and their ability to perform deeper, structured reasoning in the information-rich sce- narios. So NIAH performance on existing benchmarks alone is not a reliable indicator of reasoning ability.
across densely distributed information. In analysis,
To bridge this gap, we introduce MMReD (Multi-Modal REasoning in Dense context), a benchmark designed to assess a conceptually different capability: reasoning dense environments where mod- els must attend uniformly across the entire context. Our experiments reveal a consistent decline in performance across all tested models as context length increases (Figure 2 highlighting core limi- tations in current architectures and training approaches for dense context multi-moda reasoning.
Our contributions can be summarized as follows:
1. We develop and release MMReD, a comprehensive benchmark for evaluating long-context multi-modal reasoning, which goes beyond existing NIAH setups.
2. We demonstrate that state-of-the-art LLMs, LVLMs, and aasoning-specialinn architec- tures fail to generalize to dense context reasoning, revealing novel significant limitations.
3. We show that standard fine-tuning methods, e.g., supervised fine-tuning (SFT) and GRPO (Shao et al. 2024), are insufficient for enabling dense context reasoning in current models.
4. We highlight key challenges and propose potential directions for future research aimed at overcoming the current limitations in dense context reasoning for multi-modal Al systems.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Long-context understanding has become a central challenge in the development of LLMs and L.V.Ms.. Prior research in this area spans (i) benchmarks for long-context comprehension and (ii) ar- chitectural and training approaches for improving long-context reasoning. We briefly review both strands, motivating the unique role of MMReD.
Early efforts in long-context evaluation. Initial work focused synthetic or textual setups de- signed to probe whether models can retain and exploit information over extended sequences. For example, Kim & Schuster (2023) introduced trackyg and disambiguation tasks, providing some of the first systematic evidence that transformer models struggle when the relevant signal is deeply buried in the input.
Rise of the -naale--n---Hystackkk paradigm. Building on these insights, subsequent benchmarks formalized the retrieval challenge by embedding small "needle" of task-relevant information inside a large, mostly irrelevant context. BABILong (Kuratov et al., 2024) extended the bAbl reasoning tasks (Weston et al., 2016) into book-length contexts, while Visual Haystacks (Wu al., 2024) adapted this idea to multimodal settings by injecting known objects into distractor-heavy image col- lections. These benchmarks popularized the NIAH paradigm, which has since become the dominant template for evaluating long-context models.
Beyond single-needle retrieval. Recent work has sought to move past simple NIAH formulations. Vodrahalli et al. (204) introduced the Michelangelo benchmark to study the "short-circuitin prob- lem, where models exploit superficial correlations rather than genuinely using the entire context; its most demanding latent list setting requires tracking up to 20 relevant needles among distractors, yet state-of-the-art models still perform well above chance. posed LongBench v2, which grounds evaluation in naturally occurring long contexts rather than synthetic haystacks, thereby improving ecological validity. Nevertheless, both Michelangelo and LongBench v2 remain closer to multi-needi retrieval problems: the bulk of the context is still dom- inated by irrelevant or weakly informative content, By contrast, MMReD is explicitly designed so that all context elements are densely informative, forcing models to integrate global patterns rather than locate sparse signals.
parallel, |ai et al. (2025b) pro-
Architectural approaches and theoretical limitations. Research on long-context modeling has explored architectural innovations and training schemes: memory-augmented transformers (Bula- lov et al. 2022; Rodkin et al. (202),, structured state-space models such as Mamba (Gu & Dao 2024 Dao & Gu (024). and context-extension techniques like YaRN (Peng et al. 2023) and LongVA VA (Zhang et al.] 2024a). While these approaches extend sequence length handling, their evaluation is typically tied to NIAH-style retrieval. More recently, [Limann & Guo (2026) reframe attention through Entropic Optimal Transport, replacing the implicit uniform prior of standard soft- max with learnable continuous prior; beyond providing an EOT-based explanation of attention sinks, this approach yields a prior that combines the flexibility of learned positional embeddings with the geighh-gneraaizztion properties of fixed encodings. Two concurrent theoretical analyses illumi- nate why such failures arise structurally. Veličković et al. (005)) prove that any temperreme softmax-Bi -based circuit must cessperse-ateentiin weights necessarily spread over irrelevant keys as the number of items grows-making sharp, reliable lookups at unseen sequence lengths provably impossible within standard softmax architectures. Ebrahimi et al. (2026) show that transformers learn length-specific solutions with negligible weight sharing across sequence lengths, so the data required to cover new lengths grows far more steeply than for recurrent models that amortize learn- ing across lengths; this directly explains why standard SFT and GRPO fail to generalize to longer contexts in our experiments. Taken together, these results suggest that the performance degradation observed in MMReD is not merely a scaling artifact but reflects a structural limitation of softmax attention in dense, integrative reasoning settings.
Beyond single-needle retrieval. Recent work has sought to move past simple NIAH formulations. Vodrahalli et al. (224) introduced the Michelangelo benchmark to study the "short-circuiting" prob- lem, where models exploit superficial correlations rather than genuinely using the entire context; its most demanding latent list setting requires tracking up to 20 relevant needles among distractors, yet state-of-fe the-art models still perform well above chance. In parallel, Bai et al. (2025b) proposed LongBench v2, which grounds evaluation in naturally occurring long contexts rather than synthetic haystacks, thereby improving ecological validity. In the video domain, BBe-AAi et al. (2025) make
3

---

Published as a conference paper at ICLR 2026
the evidential demand of VideoQA benchmarks explicitly measurable via the Minimum Required Frame-Ser (MRFS) the smallest set of frames that must be fused to answer a question correctly. They show that prior VideoQA datasets have a mean MRFS of only 2.6-4.2, meaning most ques- tions are resolvable from a handful frames, whereas HERBench raises this to 5.5; nevertheless, their hardest setting requires only a sparse subset of frames, and models still fail primarily due to retrieval and fusion deficits rather than an inability to reason over uniformly dense contexts. Nevertheless, Michelangelo, LongBench v2, and HERBench share the same structural property: the bulk of the context is still dominated by irrelevant or weakly informative content, and the task re- duces to locating a sparse evidential subset. By contrast, MMReD explicitly designed so that all context elements are densely informative, forcing models to integrate global patterns rather than locate sparse signals.
Motivation for MMReD. Prior benchmarks have advanced the study of long-context comprehen- sion but converge on retrieval-centric formulations that not directly test a model's ability to reason when all parts of the context are densely informative and must be integrated uniformly. MMReD is designed to fill this gap: complements NIAH evaluations by systematically assessing reasoning in dense, non-retrieval--asse multimodal contexts, and provides diagnostic setting in which the theoretical limitations identified above can be measured empirically.
MMRED BENCHMARK
To address questions above, we create visual environment with randomized and scalable state evolution and sufficient and diverse of tasks to evaluate the key reasoning capabilities of models.
3.1 DESIGN PRINCIPLES
We prioritize minimalist visual representation. By avoiding visual complexity, we ensure that evaluation results reflect dense context multi-modal reasoning rather than visual-only perception, as in Wu et al. (2025). However, our design assumes that models possess basic OCR capabilities.
We also prioritize minimalist linguistic constructions. By doing so, we ensure that our evaluation remains focused on the same dense context multi-modal reasoning rather than language and in- struction understanding, as Kim & Schuster (2023); Vodrahalli et al. (2024). Despite linguistic simplicity, ensure covering key reasoning categories with diverse set of tasks.
Our benchmark designed to scale in complexity on the context length axis. The dataset includes sequences of varying lengths, with longest sequences reaching 128 frames, where best models start to systematically score 0% accuracy in some tasks. With the increasing capabilities of models, the benchmark can be extended larger contexts (e.g., 256 frames or beyond), ensuring that it remains a dynamic and evolving evaluation framework.
the current
To ensure MMReD provides fair and robust evaluation, we enforce strict controls dataset gen- eration. All generated sequences are unique, preventing unintended memorization or retrieval op- portunities. The environment evolves randomly, preventing reliance on simple heuristics. And we balance generated answer distributions to prevent dominance of some frequency-biasee methods.
3.2 DATASET
The MMReD dataset consists structured sequences of frames representing an evolving environ- ment. Each frame depicts spatial arrangement of characters within predefined rooms. These sequences serve as context for various reasoning tasks, such as identifying patterns across time (c.g., "Which room was empty for fewer steps than the other rooms?"). See examples in Figure] [
Environment. We define six distinct rooms: Kitchen, Bathroom, Garden, Office, Bedroom, and Hallway; and five characters: Sandra, Mary, Michael, John, and Daniel. At each time step, charac- ters are assigned to rooms, with the possibility of multiple characters sharing a room or some rooms being empty. All characters remain present in the environment throughout the sequence.
Sequence evolution. Our dataset consists of multiple distinct sequences. sequence con- sists of an initial state and set of state changes. We generate sequences of varying lengths
4

---

Published as a conference paper at ICLR 2026
N E (1, 64, 128). with 1200 sequences per length. When N 1, the sequence consists only of an initial state, with an empty set of state changes.
To generate a sequenee of length N, first sample an initial state by randomly (uniformly) assign- ing each character to one of the six rooms. At each of the following N steps, one randomly selected character moves to different randomly selected room. We generate sequences iteratively and, if an obtained sequence already exists, repeat the process until a unique sequence appears.
Scene construction. Each state is visualized as a 512 x 512 pixel image, where rooms depicted as rectangles located in a 2 x 3 grid, each named at its bottom. Characters are represented as colored circles, with their names displayed above. A character's presence in room is shown by positioning its circle inside the corresponding rectangle. The state number is given at the bottom of the image.
Tasks annotation. Each sequence is paired with exactly one generated question, selected from set of 24 question types. We generate 50 questions per type, resulting in 1200 question-sequence pairs per length N. To generate question, we randomly sample rooms, characters, or a state number as needed for the question template. The answer is precomputed algorithmically using full access to the sequence and environment. If a correct singular answer does not exist, e.g., two characters satisfy the condition simultaneously, we repeat the sequence generation for this question.
3.3 TASKS
MMReD consists of two primary categories of questions: scene-referenced (NIAH) and dense con- text (DC) reasoning questions. Each category is designed to systematically evaluate different aspects of multi-modal reasoning.
NIAH questions. The first three sections of Table M correspond to the NIAH task. These questions are designed to be an- swered based on a single image within the sequence that satisfies the condition.
Table 1: Overview of MMReD benchmark questions categorized by reasoning type. Each question template contains placeholders: [R]. [C], and [X] represent ran- domly sampled rooms, characters, and step numbers, respectively. [comp] is a randomly chosen comparison phrase ("most" or "least amount of").
We created three subgroups of questions: First Appearance (FA), locating the earli- est occurrence of a given entity or event: Final App (FI), focusing on the last occur- rence; Frame X (FX), focusing on an ex- plicitly specified frame. This division al- lows us to isolate and measure the lost-in- the-middle phenomenon (Wu et al. 2024). where models struggle to retrieve informa- tion from intermediate context positions.
Question tamplate
FA-FA-R
which room did [C] (irst appess?
EA-CCEAR
which room [CJ) when tisst appensed the [R]7
FAFRRC
was disst uppear in [R]
FA-RCFA-C
vos the [RI] whes [C] finst appeaned in [R2]]
FA-NRFA-1
many characters were in the [R1] when (C) fiost appeared in [R3]]
FLFA-R
which room was
the step?
R-CCFL-R
which room was (CI) when [C2] made their finel appearance in the [R]]
was list appeas the [R]
F--CFA-C
Who was the [R1] when [C] mode their final appeanmce the [R33
FI-NRFA-1
many chirs were the (Rhi) when [C] mode their final app in the [R3]?
FX-CF-R
which room was at step [X)?
Within each section, we construct multi- ple question templates that target core rea- soning categories such as object tracking. counting, and spatial reasoning, similar bAbl (Weston et al. 2016). We also diver- sify output types- - rooms, characters, and integers- to test model robustness across different answer formats. This ensures that our evaluations remain representative of the model's overall reasoning ability.
FX-RF-C
was the [R] at step (X]
FX-OCR-C
vos the came mam as (C) step [XP
EX NCP
Hew many other charasters were in suste room as (C) at step (X??
FX-NE-1
New many rooms were empty at step [X]?
DC R
Which room was empty for the [oomp] steps?
DC Ne-WS-R R
which room did [C] spend the (comp) time?
DC-CR-R
Which moom was crowded (ilreo or moce people) for the mose stepe?
DC-WHS
Who spent the [comp] time in the [R]
DC-SA-C:
Who spent the (comp] time alone the moms?
DC-ST-C
With whon did [C] spoad the (comp) tine together in the same mout!
DC-SR-1
How many steps dil [C] spead the [R]?
DC-RU-
How many different rooms did (C) visit?
nC-OC-I
How many times did a croed (there er more people in one mam) appear?
DC questions. The final section of Table [ introduces novel dense context (DC) reasoning tasks. Unlike the NIAH questions, which require retrieval from a single image, these questions demand global and uniform attention to the entire sequence. We similarly diversify DC tasks to include the same reasoning categories and output types, resulting in nine question types.
Problem formulation. The final evaluation dataset consists of 9600 triplets (sequence, question, answer). The models' goal is to predict the answer given the input pair sequence -queetiinneeooon.... LVLMs receive sequences transformed to images, described above, while LLMs receive a purely textual
5

---

Published as a conference paper at ICLR 2026
representation. To assess model performance, exact-match accuracy, where predicted answer is considered correct if and only if it matches the ground truth answer.
3.4 EVALUATION SETUP
Models. We evaluated approximately 30 multi-modal models on MMReD, including both open- source and proprietary models. Our primary focus was on image-specialized LVLMs, includ- ing Qwen2-VLL VL (Wang et al. (024a)), Qwen2.5-VL (Bai et al. 2025a), IntervVL-22 (Chen et al.
2024.. IntemVL-2..5MMPP (Wang et al. 2024c), MiniCPM-2.66 1-0.CM--26--0 Yao al. 2024), and VA
Mini (Zhang et al. 2025b). Additionally, we evaluated video-oriented LVLMs, such as LLaVA-
Video (Zhang et al.) 2024b), Video-LLaMA3 (Zhang et al. (025a), and Aria (Li et al. 2024)
We also assessed variety of LLMs: Qwen2 (Yang et al. (024a)). Qwen2.5 (Yang et al. 2024b)
Qwen3 (Yang et al. 2025), and Qwen-Coder (Hui et al. 2024) families, distilled versions of the
DeepSeek-R1 (Guo et al. 2025), and the QwQ (Team 2025) reasoning model from the Qwen se-
ries. Among the proprietay models, OpenAl's GPT-4o (OpenAl (024a) (*2024-11 11-40)1-20") and GPT.
4o-mini (OpenAl 2024b) (<2024-07-18') were tested using their official APIs.
Benchmark representations. LLMs require textual representation of our benchmark. Thus, we transform sequences into JSON files, explicitly writing the frame numbers and corresponding char- acters locations. Furthermore, video-oriented LVLMs use different methods for sampling frames from the input visual sequence. So we fed MMReD images to video-oriented LVLMs as result of frame sampling, ensuring an identical amount of input information to all models.
Fine-tuning. We also tested whether fine-tuning can help generalize to unseen context lengths. Both SFT and GRPO (Shao et al., 2024) training were performed on the same dataset spanning N = [1, 4, 8, 16) sequence lengths, 200 samples per each task of the benchmark per sequence length. We used DDwen2.5-7B-Instreen (Yang et al. 2024b) and Falcon3-Mambe -a-aa-------ntcr (Tem
2024 models for SFT and -----eeee----iislllll Qweenill-Qwen-BB (DeepSeek-Al et al.) 2025) for GRPO.
Technical details, such as system prompt, output formats, generation parameters (consistent across all models), model hosting, fine-tuning hyperparameters, and resources, provided in Appendix.
RESULTS AND ANALYSIS
4.1 MAIN RESULTS
Average performance across all tasks of models, including both multi- and text-only variants, is presented in Figure 3a Results for each model and model group are given in Supplementary materials. Based these results, we draw several key conclusions.
MMReD a universal and cross-modal task for dense reasoning. The performance of all eval- uated models begins to degrade significantly for sequence lengths exceeding 32 steps. Which is even more evident for the DC tasks (Figure (3)). This degradation rate strongly corresponds to the number of parameters larger models demonstrate greater ability to maintain accuracy in con- structing logical chains over longer sequences. Furthermore, reasoning-spest -ppoiiai-peecillzzedd LLMs not only exhibit superior initial performance but also show greater robustness to increases the length of the reasoning chain. It is worth noting that the Qwen2.5-Code model, fine-tuned for poding-speciffi tasks, underperformed compared to the original Qwen2.5. This suggests that training on coding tasks alone does not sufficiently reinforce the ability to construct and infer logical reasoning chains.
Multimodal instruction tuning impairs long-context understanding. Additionally, LVLMs struggle to utilize visual context effectively, even when the context length remains within the claimed supported limits. For instance, Intern VI.2.5, which reports context length of 16,384 tokens and uses 256 tokens to encode single image, should handle tasks involving up to 64 images with op- timal accuracy. But we observe a decline in performance starting from 16 images. For sequences exceeding 64 images, the models' ability to generate coherent reasoning chains deteriorates to the point where extracting a final answer becomes unfeasible.
Impact of LLM reasoning long-context retention. Proprietary GPT-4o and its 'mini' ver- sion outperform most ooen-source models in both textual and multi-modal representations of the benchmark. Their accuracy short sequences is nearly perfect, and as the length of the sequence

---

Published as a conference paper at ICLR 2026
Cole
LIMG
DILRS
Reasiving LLR.
(a) Models' mean accuracy calculated for all MMReD tasks (24 questions).
LIMS
Reasesing LLMS
(b) Models" mean accuracy calculated only for dense context (DC) type of tasks.
Figure 3: Models' mean accuracy on MMReD across all sequence lengths, grouped by model type: fine-tuned LLMs, general-purpose LLMs, LVLMs, and reasoning LLMs. Cell color indicates per- formance (green high, red - low), with models ranked by average score within each group.
increases, the degradation in quality is smoother. However, DeepSeek RI, distilled LLaMA-70B and Qwen2.5-32B outperform GPT-4o by 2-4% initially and considerably more at longer sequence lengths, as shown Supplementary materials. This highlights the substantial contribution of LLM reasoning capabilities to retaining long preceding contexts and deriving correct results.
4.2
TASK ABLATIONS
3.0
1.50
0.25
0.8
0.50
0.25
0.6
DC NIAH
DC vs. NIAH
3.4
FF FX)
vs.
0.25
Sequance Langth
Sequence Length
(a) Pearson correlation on NIAH tasks.
(b) Correlation under 5% perceptual noise.
Figure 4: Analysis of model performance: (a) Self-consistency vs. DC tasks and (b) Performance under noise conditions.
7

---

Published as a conference paper at ICLR 2026
Isolating DC reasoning. Figure 4a illustrates the breakdown in correlation between performance on retrieval-based (NIAH) and DC reasoning tasks as context length increases. We calculate Pear- son correlation of model scores averaged for every task within selected task groups. In the first experiment, we select DC and NIAH groups; in the second experiment, we select FA plus FI and FX groups. While Pearson correlation between different NIAH subsets remains consistently high (around 0.9), the correlation between NIAH and DC performances notably declines to approximately 0.5-0.7 at 32 frames and beyond.
Our findings closely align with those reported in BABILong (Kuratov et al., 2024); correlation between the models' ranking on their long-context benchmark and short-context one drops from 0.9 to 0.6 when context length increases. These findings suggest that models successful in NIAH-style retrieval not necessarily generalize to the proposed DC reasoning scenarios. And thus MMReD allows assessing a conceptually different capability, which we call dense context reasoning.
Perceptual complexity. Perceptual ambiguity, e.g.. occlusion, visual noise, recognition errors, plays a major role in real-world multimodal reasoning. However, MMReD is specifically designed to isolate reasoning capabilities by minimizing perceptual complexity.
To test whether this abstraction limits generalizability, we conducted an additional ablation where we introduced controlled perception noise into MMReD. Specifically, we modified the environment state by adding synthetic occlusion-like errors (c.g.. randomly relocating entity to incorrect room in one frame) at a 5% rate. We then evaluated LLM performance on both NIAH and DC questions; see Figure 4b
The results show a uniform drop in performance proportional to the error rate, but crucially, the relative gap between LC and NIAH questions, the key trend in Figure 4a| remain intact. While conducted this test on LLMs,, the insight generalizes to LVLMs as well. In our setting, LVLMs and LLMs show consistent delta, likely due to (i) the token budget shift toward visual inputs, leaving fewer tokens for reasoning, and (ii) catastrophic forgetting from vision-domain fine-tuning. Thus, solving dense context reasoning LLMs transfers naturally to LVLMs.
Symbolic environment. We evaluated MMReD under an abstract symbolic projection (5 loca- tions coded as L1-L5, entities coded as E1-E6) test whether results depend on the visual and structural variations. Model rankings and overall performance trends are preserved under the sym- bolic representation: Pearson correlations between standard and symbolic environments remain high across sequence lengths (see Appendix Table indicating that observed DC vs NIAH gaps are not surface- specific to the visual "rooms and characters" instantiation. We therefore interpret MMRel) as probing structural DC reasoning challenge rather than an artifact of the chosen visual motif.
In summary, we show that adding controlled perceptual noise nor structural variations do not alter the key conclusions of MMReD, Our benchmark allows studying dense reasoning in isolation, and can be extended to real-world scenarios by progressively incorporating perceptual challenges.
4.3 MODEL ABLATIONS
The core motivation for our experiments stemmed from the question: What components or ap- proaches enable LVLMs to understand long visual sequences more effectively? To address this, we formulated a series of research questions focusing on architectural components, dataset characteris- ties, and training procedures. These questions are systematically explored in this section.
We computed the relative perfor- Text-only (LLM) vs Multimodal (LVLM) performance LLM-LVLM x 100%. The gap varies with mance gap between text-only and multimodal variants as LVLM
model scale and context length: text representations generally outperform multimodal ones at mid context lengths, while smaller models sometimes invert at extreme N (Table This pattern suggests that text encodings reduce perceptual token-budget pressure, while for smaller models compressed visual encodings can be advantagcous.
Fine-tuning is not enough for generalization. Both fine-tuned Transformer and Mamba archi- tectures demonstrate the same decline in performance as benchmarked zero-shot models. GRPO, being promising way to bootstrap reasoning, is performing even worse than fine-tuned Trans- former. Overall, fine-tuning generalizes a bit better than best regular LLM and worse than reasoning L.MMs, in particular, on tasks involving heavier arithmetic.
8

---

Published as a conference paper at ICLR 2026
Table 2: Relative performance gap (%) between text-only and multimodal models across sequence lengths. Positive values indicate text-only advantage.
Model
=
N
8
= 16
= 32
N =
128
Qwen2.5-3B Instruct
6.7
4.1
-2.3
-5.7
-12.0
-23.2
-32.2
-37.5
Owen2.5-7B-Instmuu
2.3
-4.0
-11.3
-9.1
-1.8
-9.6
-3.9
-19.1
Owen2.5-72B-Insee
0.6
1.2
2.9
5.7
6.2
-3.8
GPT-4o
9.8
7.1
7.2
GPT-4o-mini
7.2
12.6
14.8
14.7
21.0
27.6
32.5
27.6
Mean
11.0
16.8
35.1
35.3
34.1
12.5
0.7
-8.2
Additionally, we performed a controlled In-Context Learning (ICL) ablation by appending five solved examples (from N < 16) each prompt. Across LLMs (4B-32B) observe mean relative improvement of 13.8%; mid-range models gain up to 23% (Table 3). Although ICL yields consistent gains, it is substantially smaller than SFT the same mid-range models (57.7%) and GRPO (27.8%), and none of these approaches fully resolve dense-context failures at large N.
Table 3: Relative mean improvement (%) over baseline across sequence lengths for ICL, GRPO, and SFT. The last column reports the relative improvement on the full dataset.
Method
N=1
N=2
N=4
N=8
N=16
N=32
N=64
N=128
Full dataset
ICL
13.4
16.9
17.9
17.2
13.8
7.5
7.7
12.5
13.8
GRPO
14.6
16.9
23.2
36.6
44.9
37.1
26.5
72.0
27.8
SFT
8.0
24.2
49.1
76.9
89.8
109.2
99.8
178.5
57.7
Model Parameter Count. To investigate the effect of model size on MMReD response quality, we compared the evaluation results of both LVLMs and LLMs, focusing on three representative model families (Figure 5) The results from the Qwen2.5-VE and Intern V1.2.5 series reveal a clear trend: accuracy in question answering improves as the number of parameters increases. However, for the InternVL2.5-38B and 78B models, the performance gap narrows significantly, becoming much smaller compared to the differences observed among smaller models. A similar pattern is evident the results of DeepSeek-RI distilled to LLaMA-70B and Qwen-32B. These findings suggest that, for both visual and text-only models, the number of parameters strongly influences the ability to process long contexts. However, this dependence diminishes as model size increases, with visual reasoning quality eventually plateauing beyond certain threshold.
SeOVIte Lenges
Seaur Leigo
Figure 5: Impact of the number of model parameters on the ability to successfully infer a response using a long visual or text context from MMReD.
Multimodal Adapter Type. We evaluated the efficiency of LVLMs using different adapter types: multi-linear layer adapter (Qwen2.5-VL-7B-Innte 2.6 (Yao et al.) 22024)), and cross-attention (Llama-3.2-111 - results in Figure II show that LLaMA does not provide high performance. The results Qwen2.5- VL and MiniCPM questions, but MiniCPM performs better on FI-FA-R ones. Moreover, Qwen2.5 has low scores for short sequences in certain types of questions.
(Bai et al. 2025a)), QFormer (MiniCPM-o- (Chu et al., 2024)). The
roughly the same: both models perform poorly DC-CC-I and DC-SR-I

---

Published as a conference paper at ICLR 2026
Frame Pooling Methods in Video-Specific Models. We also compare performance of the different video-specific models: IntermVL-2.5 (Wang et al., (024c). Aria (Li et al.) 2024), VideoLLaMA- 3 (Zhang et al. 2025a), and LLaVAViddo-
(Zhang al. (024b). As shown in Figure [2] Videol.LaMA- provides the best performance. Aria performs better than Intern -1.2... Moreover, IntemVVL-2255 has considerable drawdown for all sequence lengths in certain types of questions.
Training Data Volume and Composition. We compare two types of VideoLLaMA3 models: VideoLLaMA3-Image, fine-tuned on both image-text and video data, and VideoLLaMA3, which undergoes video-centric fine-tuning, Both variants use alignment (media paired with captions) and instruction (question answering) data. As can be seen Figure [3] the VideoLLaMA33 3-aae fam- ily performs better on straightforward questions with short sequences. However, its performance declines on longer sequences, particularly for question types such as DC-SR-I and DC-SA-C.
We argue that incorporating video pretraining without textual instructional data on the last training stage negatively impacts the model's performance on our benchmark. As shown in the Figure Ba models from the LLaVA-Video family, even 72B ones, perform worse than image-focused or general purpose LVLMs.
Possible directions of solving the challenge of dense context reasoning include, but are not limited to, examining the potential of architectural innovations specifically designed for long-context pro- cessing, overcoming problems of the attention sinks and similar known expressivity issues of RNNs,
and incorporating uncertainty quantification (Fadeeva et al. 2024 2023: Vashurin et al. 2025) to make reasoning more steerable and not materialize false information in the decoded logits; building on existing approaches of test-time compute scaling alongside rigorous RL goal-based pre-training.
5 CONCLUSION
In this paper, introduced MMReD, novel benchmark designed to systematically assess long- context reasoning capabilities of LLMs and LVLMs. By focusing on both scene-refcrenced (NIAH) and dense context tasks, MMReD provides a comprehensive evaluation framework that challenges models to process extended multimodal sequences and construct complex reasoning chains.
Our results demonstrate that current LLMs and LVLMs struggle significantly with long-context un- derstanding, with performance degrading rapidly as context length increases beyond 32 images. Notably, sassoiing-speciilizzedd LLMSs exhibit superior retention of long contexts compared to stan- dard models, highlighting the importance of targeted fine-tuning and architectural innovations in enhancing long-context comprehension.
Model size positively correlates with long-context performance, but the benefits diminish at larger scales, suggesting architectural limitations rather than parameter count as a key bottleneck. Addi- tionally, our analysis of multimodal adapter types and frame pooling strategies revealed that the ar- chitectural components choice critically affects model's ability reason over long visual sequences.
Unexpectedly, video-oriented LVLMs and multimodal instruction tuning are not beneficial for long- context reasoning. The observed performance decline for these models on extended sequences sug- gests that current video pretraining does not sufficiently address the challenges of long-context tasks.
Our ablation studies confirmed that MMReD's conclusions are robust to potential confounding fac- tors such as the post-in-the-midee phenomenon. By maintaining unique and randomly evolving environments, the benchmark ensures that performance degradation reflects genuine limitations in long-context reasoning rather than artifacts of data structure or task formulation.
In conclusion, our findings underscore the need for more effective architectural modifications, train- ing paradigms, and evaluation benchmarks tailored specifically for long-context reasoning. MMReD serves as a critical step forward in this direction, providing a rigorous, scalable, and unbiased frame- work for assessing and advancing long-contes understanding in both LLMs and LVLMs. Future work may explore integrating hierarchical memory mechanisms, improved multimodal fusion tech- niques, and more diverse pretraining data further enhance long-context reasoning capabilities.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We release our training and evaluation codebase and dataset in the anonymous repositories.
USE OF LARGE LANGUAGE MODELS.
In preparing this manuscript, we made use large language models (LLMs) to aid in polishing the writing. Specifically, used an LLM to (i) suggest alternative phrasings for certain sections (Related Work and Conclusion), (ii) merge fragmented paragraphs into a more coherent narrative, and (iii) check stylistic clarity and consistency across sections. All scientific contributions, including the design of the benchmark, experimental setup, implementation, analysis, and conclusions, were developed entirely by the authors. The LLM did generate new content beyond language-level editing and restructuring, and its usage did not rise to the level of contributing author.
ACKNOWLEDGMENTS
Innopolis University authors were supported by the Research Center of the Artificial Intelligence Institute at Innopolis University. Financial support was provided by the Ministry of Economic De- velopment of the Russian Federation (No. 11119666677-1-00333
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: /tte:///rxxii www.aaiivvv..oo http///awvvv..araae////e absr///////// 2108. .088807733 07732
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 2025a.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Longbench v2: Towards deeper understanding and reasoning realistic long-context multitasks, 2025b. URL httttt:s//axxiiii .rg. abs/
2412. 15204
Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, and Chaim Baskin. HERBench: A benchmark for multi-evidence integration in video question answering, arXiv preprint 1iXi:22112.1887000 2025.
URL /tttps://axxii. wwwp:iiivrrrroooooooooeo /ttp://///......//..5222
Aydar Bulatov, Yuri Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL BtLhhtts::// hattts::/openrevieew net forummiddd Uynr3iPhksal
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open- multimodal models with model, data, and test-time scaling. arXiv preprint arXiv.2412.05271, 2024.
Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. Visionllama: unified laaa backbone for vision tasks. In European Conference on Computer Vision, 2024.

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. URL BRLhtt::: otl.:tttsss//pp....eaatttccocooo
www.nttttttotoooaat... org/
ID:
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. 2021.
LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving Ilm. https: cww.tttt.occoooaaaaoooomm www.:oomm/niiiiee..e Int.nnntennnn.. Imdeploy) 2023.
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao. Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q Wang, Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao. Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou. Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforce- ment learning, 2025. httpps:/://
org/ abs/ 2501 12948
Aaron Defazio, Xingyu Alice Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. | https: https///pppevvieeeeee net forum? ?id=
OXeNkkENui
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of
12

---

Published as a conference paper at ICLR 2026
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 2368- 2338--378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 1111186//1//N1112200 URL http:::: //aclantholocn www.aoogoooooooo..ooooooo wtw.ohoooloooggg/goooooo 9-
M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal, and Roland Memisevic. On the "induction bias" in sequence models. arXiv preprint arXiv:2602.18333, 2026. URL https://arxiv:
org/abs/260r 200.ab///22..... 18333
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. LM-Polygraph: Uncertainty estimation for language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 4464461, 2023. URL stttp:://ai..oog10...666 2ttp://0.48555//ax00
2311. 07383
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the output of large language models via token- level uncertainty quantification. Findings of the Association for Computational Linguisties: ACL 2024, pp. 9367 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/1/2024. findings-acl.558. URL
Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.),
/ttps//////llaathooccooooo ogy. wwwprhhoooogo.oor//222222 fin.iiiiniiiii www.iinnnnnnnnneeeeeenone 558/
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.2107 2024a.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference Computer Vision, pp. 148-166. Springer, 2024b.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling. 2024. URL httpttts///peeeeeeee..eeee www.::enttffootteeee fortmmirrrm
tFYakw VY2
Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scar- lett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 166211 PMLR, 21-27 Jul 2024. URL
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: /tlthttps://ppoeeee http::///peeeeeeeeeeem net. htt///ooummmidic 2mQsCe
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's 's the real context size of your long-context language models? In First Conference on Language Modeling, 2024, Ut//:ttpps//ggennnvvieeeeee////s id Td=IIoBb7768y
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
13

---

Published as a conference paper at ICLR 2026
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.1 12186, 2024.
Najoung Kim and Sebastian Schuster. Entity tracking in language models. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 67 Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 3835-3855, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 10.18653/v www.2222...lloonnn..... URL 213/
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Igorevich Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong: Testing the limits of LLMs with long context reasoning-in-a- in-a-haystackk. In The eight Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2024. URL https: httt:s:://opereevieee net forum?
dd=u7mmcG84BQ
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. htt.https::///// /arxiv..//avvvaa//aa//ee rxiia.v..... org/ abs/ 2309. 061 80
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024.
Elon Litman and Gabe Guo. You need better attention priors. arXiv preprint arX1v:2601.15880 2026. utLpttps: http:////wwwwv..ooo///b///////// 15380
MAA. American invitational mathematics examination, 2024. URL http:://ma..rrg
math- comppt.tiinns/mmmrr..oooom http:///ooocnccoooooooooooooaoaa ie aime
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnos-
tic benchmark for very long-form video language understanding. In Thirty-sevnnth Confer-
ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
https://spenneviee net htt.:///ww......oooooooooooeeeoe
OpenAl. Gpt-4o system
"ttps://openaa..cm///nee/ Sy-ssttttt--stteeeeeeeeeee card/ 2024a. Accessed: 7 March 2025.
OpenAl. Gpt-4o mini: advancing cost- intelligenn.. https://opemai http:://oopaaaa.oo....o /dem//
ottp::a..-..cccccoooooooooooeaa- hww-fficiee--nnteellggeeeeeeeeee 2024b. Accessed:
March 2025.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yar:: Efficient context window extension of large language models, 2023. URL
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URLhttts: httttps:///ppnreevit
net//oorum2o=711675555
Ivan Rodkin, Yuri Kuratov, Aydar Bulatov, and Mikhail Burtsev. Associative recurrent memory transformer, 2024. URL http::tttss//cot. /wwpvv///vvv//////////...//.aa org/
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024. URL/tttps://axxii wwwpxiivoor//aae. abspg//bs/242. 03300
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether ofain-of-thoughh ation for Computational Linguistics, 2022. URL https: //l:://alaaa www.aaattooooogggooooooaa
solve them. In Annual Meeting of the Associ-
findin findings-aci .824
14

---

Published as a conference paper at ICLR 2026
Falcon-LLM Team. The falcon 3 family of open models, December 2024.
Qwen Team. 3wb- Embracing the power of reinforcement learning, March 2025. URL wwt.://m...thu....m http/////////////////////
Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing. Abdelrahman Sadallah, Kirill Grishchenkov, et al. Benchmarking uncertainty quantification methods for large language models with Im-polygraph. Transactions of the Association for Computational Linguistics, 13:220-248, 2025. URL http::///aaiiioo......./////////
00737/ 20037/228113/enchnaee / 1r00://55...aaaaaaaa/oaaaaaaaar0
Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. Softmax not enough (for sharp size generalisation). In Proceedings of the 42nd International Conference Machine Learning, 2025. URL
org. abs/ 2410 01104
Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu. Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, Rohan Anil, Ethan Dyer, Siamak Shakeri, Roopali Vij. Harsh Mehta, Vinay Ramasesh, Quoc Le, Ed Chi, Yifeng Lu, Orhan Firat, Angeliki Lazaridou, rean-Baptiste Lespiau, Nithya Attaluri, and Kate Olszewska. Michelangelo: Long context evaluations beyond haystacks via latent structure queries, 2024. URL https:
7/arxiv. abs/ /2499. 12640
Fei Wang, Xingyu Fu, James Y. Huang. Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang. Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprehensive benchmark for robust multi-image understanding. In The Thirleenth International Conference on Learning Representations, 2025. URL https:
//openzeview.coo net htt.://ww...tttommaaaaaaEEEEE.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing Assionllanuage model's ' perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024a.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi. Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.080. 2024b.
Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:241 1.1422, 2024c.
Yubo Wang. Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro. A more robust and challenging multi- task language understanding benchmark. In The -hirtt-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024d.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In Yoshua Bengio and Yann LeCun (eds.), 4th Inter- national Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016. URL utLhttp: /ttt::://arxiiii wwt.vv/.rga//b/a/oeaoeooooo
05698
Brandon T Willard and Rémi Louf. Efficient guided generation for llms. arXiv preprint arXiv:2307.09702, 2023.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interlcaved video-language understanding. Advances in Neural Information Processing Systems, 01128828-288555 2025.
Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Dar- rell, and David M Chan. Visual haystacks: A vision-centric needle-in- in-a-haystack benchmark. arXiv preprint 207. 2024.
15

---

Published as a conference paper at ICLR 2026
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang. Ru Peng, Rui Men, Ruize Gao. Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint ArXi 10671, 2024a.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin. Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 1.Niv:24/2.15111 15, 2024b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httpt::t//// pestts://aate /ttpp::///xxixa org/ abs/ 09388
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: gpt-4v level milm on your phone. arXiv preprint 0rXiv:2408.01800, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint 2rXiv:2409.028133 2024b.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13100 2025a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. /6852, 2024a.
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token, 2025b. URL https::/aaxiv..o www.xii.org/aaa.. /bs/ 2501. 03895]
Yuanhan Zhang, Jinming WL, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02711 2024b.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, arXiv preprint 4rX2v:2406.0426. 2024.
16

---

Published as a conference paper at ICLR 2026
A DETAILED SYMBOLIC ENVIRONMENT VALIDATION
To test whether MMReD results depend the specific visual "rooms & characters" projection, we introduced an abstract symbolic variant (5 locations x 6 entities). We computed the Pearson correlation coefficient between model performance in the standard environment and the symbolic environment across sequence lengths.
Table 4: Correlation of model rankings between the original projection and novel symbolic pro- jection (5 locations x 6 entities). High correlations indicate ranking stability across projections, supporting generality of observed trends.
Seq len N
8
16
32
64
128
Pearson 0.909 0.9510.16 0.80.1.30
0.72.10.35
0.69.10.36
0.8910.23
0.8610.25
0.83.10.28
We observe consistently high correlation (r >> 0.7 for most context lengths), peaking at > 0.98 for short contexts. This indicates that model rankings remain largely stable across projections. The results suggest that MMReD captures structural dense-context reasoning challenge rather than overfitting to surface-level visual semantics.
We further analyze the relative performance shift when moving from the standard to the symbolic environment. We compute the relative change: A
Original 100%.
Table 5: Relative performance change (A, in %) when shifting from the standard to the symbolic environment across sequence lengths, Positive values indicate improvement in the symbolic domain.
N
16
N
Qwen3-30B-A3B
-16.0
-6.3
-1.3
0.9
-6.7
-5.7
-5.0
-3.5
Dwon3-Noxtt808-A38
18.8
-4.0
1.4
4.2
-1.8
-7.0
3.0
2.7
Owen3-4B
12.3
5.9
8.1
9.7
8.7
-2.6
-5.1
7.1
Qwen3.8R
-10.5
-7.9
Qwen3-14B
-15.5
-3.2
3.8
10.3
12.0
8.9
6.3
18.3
Qwen3-32R
11711
4.6
14.3 1: 4.2
4.2 H:
5.9
8.3
0.9
4.9
13.4
4.4
7.1
3.7 4:
7.4
4.9 4: 8.6
The results reveal distinct dichotomy between initial grounding and long-context tracking:
Semantic Grounding Effect. At sequence length N = 1, we observe substantial performance drop in the symbolic environment (-14.3). This suggests that natural language semantics (e.g.. labels such as "Kitchen") facilitate initial grounding compared to abstract tokens. The performance decrease indicates that strly-stage reasoning benefits from semantic priors embedded in natural language representations.
Convergence Dense Contexts. As sequence length increases, the performance gap narrows considerably. At extreme context lengths (N 128), the mean gap becomes positive (+4.9), indicating that symbolic representations may even improve performance dense-context regimes. This suggests that long-context difficulty primarily stems from structural tracking complexity rather than semantic interpretation. In certain models (e.g., Qwen3-1423 abstract symbols may reduce distraction from semantic priors, allowing the model to focus more directly state transitions.
B LOST-IN-THE-MIDDLLE PHENOMENON
This phenomenon refers to the tendency of transformer-bast based models to struggle with retrieving formation from the middle of long contexts. Since our benchmark involves reasoning over extended sequences, we evaluated whether such biases affect our results.
To investigate this, we focus frame-referenced questions (FX, see Table 1 where model must extract information from specific step the sequence. This includes five question types, where we can calculate the referred image depth by extracting the "step X" number and dividing it by the sequence length.
17

---

Published as a conference paper at ICLR 2026
By averaging performance across all LVLMs and plotting accuracy as a function of the frame position (Figure (7), observe clear pat- tern indicative lost-in-the-middle degrada- signifi-
643
031
4.4
-06
406
-M2
415
4.15
614
tion. If the latter phenomenon cantly impacting our benchmark, we would ex- pect accuracy consistently drop for middle- positioned frames compared to the beginning and end, which is the case. When ana- lyzing individual model performances, we ob- served similar trends- no systematic drop in accuracy for middle-positione frames.
0.21
0.11
331
A37
2.11
0.1
1.04
"Neede" depth
Figure 7: Analysis of the lostiie-the-midlle effect on MMReD frame-referrnced questions. The x- axis represents the relative position of the target frame within sequence; the y-axis shows the nor- malized accuracy difference from the mean.
This suggests that our benchmark and its con- clusions remain independent of lost-in-the- middle biases, at least within the tested context lengths. And we believe that longer context lengths may be required before this effect becomes major factor in setting. The results suggest that longer context lengths may be required before the LITM effect becomes a major factor in our setting.
C MODELS' PERFORMANCE
Below, we present detailed heatmaps of model performance across all MMReD task types and quence lengths. Figure focuses on top-perforning reasoning models. A clear trend emerges: dense reasoning (DC) tasks are consistently more challenging than their NIAH-based counterparts, this gap is particularly evident in the QwQ model. For proprictary models (Figure 9 the contrast is less visually evident but persists when examining average performance across tasks.
arr oe 102222 16
151 MOOSSe DE resnie eri 100 031 90110
10328
pescess
40
3412005
S40 in
100
(a) DeepSeek-Be -eekk-RI RI-Distill-144 14B
(b) Qwen Qw Q-3B
1010ar 300se 10383
100000 000000000000 00 20000000000000000000000000000000 2000000000000000000
24 06 501303000
100000
0420004
d8 52 20
2003 0000000033000000
00
10392
20 300000 39
(c) DeepSeck-R--Distill.
(d) DeepSeek-RI-Dieee -1-
Figure 8: Performance of the best reasoning models.
18

---

Published as a conference paper at ICLR 2026
Secondly, found that including diverse set of question types is essential, as model performance varies across conceptually distinct tasks. For example, despite the near-identical formulation of first appearance (FA) and last appearance (FI) questions, models consistently perform worse average on FI tasks (e.g., Figure [d). This suggests that models struggle to distinguish between similar temporal trajectories in task formulation, especially when concrete step (FX), which are phrased in a similar way, being solved with notably higher accuracy. In addition, our benchmark includes tasks with gradually increasing difficulty, with DC-WHS-C (Who spent the most time in the [R]?) and DC-RE-R (Which room was empry for the most steps?) among the most challenging. The latter highlights recurring weakness in count- based reasoning.
questions referring
0rrees no (ncvn
20:2212
2204
53.55
10:00
co
Same copeen
24 ce 60 cose
20 50
63
13:0 12:22
20 105660 54 70
54
96446
2-4230004
50:001
10204
#:46 To 30
23
Wybe questio
Tepe auessto
Apt--o-miii/iage
gpt-4o- mini/text
boo 20 WOOL FR FE
2033
2600056
0002 P250CUC DF
50300
12 MoTero 875S
(700000
60000
37 14 04000000 rats 42 01
42
5-5000055 00:0004
52.03 020220003
qusston
Tope avestin
gpt-mo/
(d) 4ottex
Figure 9: Performance the OpenAl proprietary models.
As discussed in Section 4.3, various fine-tuning strategies fail to improve performance on our bench- mark with their results detailed in Figure 10 Supplementing Section 4.3 further, studies on different pooling methods are provided in Figure Finally, we report results for video-specific architectures Figures and
D EVALUATION DETAILS
Inference format and parameters The MMReD questions were formulated open-ended question-answer tasks. We provided generic problem description and answer format the system prompt (Table
To standardize output generation, we applied a structured generation approach to produce consistent answer structures in JSON format, { ("answer": <answer_typer
where <answer hanswer_type> was an enum or integer number depending on the expected answer type (C, R, I in Table 1). We utilized outlines library (Willard & Louf 2023) for structured generation, which involves converting the JSON response structure into a regular expression, which further used construct finite state machine to guide LLM generation by adding bias to logits. We used
19

---

Published as a conference paper at ICLR 2026
20100032 100000000000000
1530014
2002200002000 2000000000000000000 COST 34300 04
12.9 35 X3 s6
04
50
50
60
00
7310
24
+33 1013050 001615711100 d2 ovaccuo 1o000
44100000 60:300020000
100000000 1400300
92 32
50
20.00
50
te to 08000000
53 3000000
1000005
54
10200
23000000
10 30.0550
48
2000
te
3-33 PEXXX
00
5450
4450
(a) Qwen2.5-7B Been2..-7B-Inttruee
(b) Qwen2.5-7B --ree.ss-BBl Instruct SFT
16
66
12 72
LA 16 00-00
12
16 46 60
0.42
48
66
0000
(c) Cembba-BBInstrucc SFT
Biaebeek-RR--iiiil--ee-eee Bi1-Distill-Qwen-7B - GRPO
Figure 10: Performance on the fine-tuned models and base model before fine-tuning.
(a) Qween2.5VLL-71-Imstruu
(b) 1etmaa-.--1I--Viiioo--eeeee
(c) MiniCPM-o-2.6
Figure 11: Performance of the models with different types of image pooling.
the Imdeploy (Contributors. 2023) and VLLM (won al. 2023 packages for efficient serving of models.
We preserve the default configuration parameters of all selected LVLMs, except extend the maxi- mum sequence length and increase the positional embeddings to avoid truncating the input sequence. The generation parameters were also consistent across all models: the generation temperature was set to 0, the number of beams was set to 1, and the number of tokens was limited to 50. For LLMs with reasoning, we increase the generation temperature 0.7 and token limit to 2048.
We used at most 4 x A100 80GB in our local evaluations, and if model exceeded time limit of 600 seconds or returned out-of -etooo- error, we dropped evaluation of the current and larger lengths (e.g., in cases of Aria and Llama-3.2- V.iisonn..
Fine- Tine-Tuning
We also ablated whether fine-tuning can help generalize to unseen context lengths. Both SFT and GRPO (Shao et al. 2024 training performed on the same dataset spanning
20

---

Published as a conference paper at ICLR 2026
12 12.46:44
50
46 00 53
30 72
90.0 10
30
66
64
30.06
55 20 28
42
30
30
50
80
32
56
48
99 04
2424
2010
5050
24.04
43
29
10
(a) Intern VL2.5-8B
(b) LaMA337
2414
+.56 42
53
66
0:37
14110
3-32
(c) Aria
(d) -LLVAA-iiee-7B-QUHA
Figure 12: Performance of the different types of video models.
= 11,2,4, 4,8, 16) sequence lengths, 200 samples per each task of the benchmark per sequence
YYang et al. (0224) and Falcon3-Mambee Caaumba-----nnttuuee I--Inutt (Team 2024) models for SFT and DeepSeek-RI-
length. We used Qwen2.55-7B-Instee --ai-------itsi---------
(DeepSeck-Al al. (025) for GRPO to help bootstrap reasoning abilities. In SFT, we used a global batch size of 128, learning rate of x 10-4 and five training epochs: GRPO
trained with 0.2, 3 0.1, 12 generations per sample, global batch size of 168, and one epoch of training. We used linear warmup of 50 steps with constant schedule after warmup and ScheduleFree AdamW (Defazio et al. 2024 optimizer. The gradient norm was clipped to 0.25.
For GRPO rewards, we used a combination of answer correctness and format rewards, enforcing the model to follow an XML template <think> ( (tthinknng (eriingn//ttitt </think> <ar.swer> { ""answer": <answer_typen </apr_ttey>>><</serr>>>> and rewarding for exact matches ground truth answers.
We applied parameter-efficien fine-tuning via LoRA (Hu al.) 202T) with rank 32, alpha of 32, and dropout of 0.05 in both scenarios. Flash Attention (Dao al.) 20222 Dao (024] was enabled for
attention models, and mixed precision training was conducted with b6loat Mamba training was
performed with gradients accumulating in float32 and using efficient CUDA kernels for selective
scan operation from the Mamba codebase (Gu & Dao 2024 Dao
Gu 2024).
As evaluation metric. utilized the accuracy of the exact match between the generated response and the target value. Model predictions that did not follow the specified response format or from which the response field could not be extracted were marked incorrect.
21

---

Published as a conference paper at ICLR 2026
+.52
73
32 24
52
50
20
2200
96
40
14
14-20
20
54
2000
2010
10:00
11
5.44
(a) LaM3-2L
(b) VadeoLLLMMA37B
24-4
20 56
46 30
20
42
00
50
39
46
24
10.90 26 46
50
00
40
10000
40 28 26 60
40
34:
40 38
quettn
(c) VideoLLaMA3-2 2B-agee
(d) VideoLLaMA3-77 -B--A-------meeee
Figure 13: Results across different models in the LideoLLaMMA3 family.
22

---

Published as a conference paper at ICLR 2026
System Prompt
You are an assistant that analyzes sequences of human agents moving in an environment. Format your response as a following json:
{ ""nswer"; <value> ,, where <value> is:
A **single room name** (e.g., 'Kitchen') for location answers.
**number** (e.g., '3') for counting answers.
A **single person name** (e.g., 'Michael') for people answers or 'Nobody' if no person satisfies given conditions.
System Prompt for reasoning- models
You are a helpful AI Assistant, designed to provided well-reasoned and detailed responses. First think about the reasoning process and then provide the user with the answer. Format your final answer with ("answer": <value>), where <value> is:
**single room name** (e.g., 'Kitchen') for location answers.
**umbbr** (e.g., '3') for counting answers.
**single person name** (e.g., 'Michael') for people answers or 'Nobody' no person satisfies given conditions.
Table 2: System prompts used during inference.
E REASONING EXAMPLES
In general, we find that reasoning improves performance: when given sufficient token space, it often guides the model to the correct answer by helping locate the relevant step and correct errors that non-rcasoning models typically make (see example in Table 3).
However, in dense context (DC) tasks, starting at around 32-frame sequences, reasoning models begin to shift their output toward step-by-step. JSON-like listing of observations. This introduces two key issues. First, a model may make a mistake when processing one step, and then proceed to build on that false assumption, effectively compounding and grounding its error, as in example in Table Second, enumerating all steps consumes a large portion the token budget, increasing the risk that the model fails produce final answer within the predefined 768-token generation limit (a threshold, commonly sufficient for solving math problems (Cobbe et al. 2021), and thus adequate for our simpler reasoning tasks).
23

---

Published as a conference paper at ICLR 2026
DegpSeek-RR-Diiiilllaamm77
Q: Who was in the Hallway when Michael first appeared in the Garden?
A: Mary
<think>
Okay, so I need figure out who was in the Hallway when Michael first appeared in the Garden. Let me break this down step by step.
First, I'll look through each step to find when Michael first shows up in the Garden. The data is structured from step to step 64, with each step showing the rooms and the people inside them. My goal is to scan these steps sequentially until I find the earliest occurrence of Michael being in the Garden.
Starting from step 1, check each room in every step. In the early steps like 1-5, Mi chahl isn't t. in the Garden yet. He's s in other places like the Office or Bedroom. each step, I keep an eye on the Garden room.
go through
Continuing this way, at some point, I notice that in step 44, the Garden has ['Michael']. That seems to his first appearance there because prior steps don't show him in the Garden. So, Michael first appears in the Garden at step 44.
Now, once I've identified step 44 as the moment Michael enters the Garden, I need to check who was present in the Hallway at that exact step. Looking back at step 44's s data:
('Kitchen': : II, 'Bathroom: [), 'Garden's ['Michael'), 'Office' S'narr'', 'Daniel), 'Bedroom's ['John'], HHallway': ['Mary']].
From this, it's clear that in the Hallway during step 44, Mary is present. Therefore, when Michael first appeared in the Garden, Mary was in the Hallway.
should double-check my findings to ensure accuracy, Scanning earlier steps confirms that Michael wasn't t in the Garden before step 44, and at step 44, only Mary is in the Hallway. No one else is there except her.
So, putting it all together, the person in the Hallway when Michael first entered the Garden was Mary. </think>
l"answer:: "Mary"
Table A generic reasoning example of the largest used DeepSeek model on the FA-RCFA-C task.
24

---

Published as a conference paper at ICLR 2026
DeepSeek-RL-DDsttl-LLmmaa-
Q: Who spent the least amount of time alone in the Hallway?
A: John
<think>
Alright, let's try to figure out who spent the least amount of time alone in the Hallway. So, have data from step 1 to step 128, each showing which rooms people are in at that step. My task is to go through each step, check who's in the Hallway, and times each person was alone in the Hallway and find out who had the smallest number.
if they're alone there. Then, count how many
First, need understand what "alone" means. means that during a particular step, only that person is present in the Hallway; nobody else is there. So, for each step, I'll look at the Hallway entry in the 'rooms' dictionary. exactly one person, that person was alone that step.
If it has
Otherwi se, even if multiple people are there, as long as someone else is present, it doesn't count as being alone.
So, my plan is:
1. Iterate through each step from 1 to 128.
2. For each step, get the list of people in the Hallway.
Step 1:
Hallway [Sandra] Sandra Sandra=1
Step 2:
Hallway [Sandra] Sandra +1 Sandra=2
Step 3: Hallway [Sandra] Sandra=3
Step 4: Hallway [] - No one
Step 5: Hallway [] No one
Step 6: Hallway = [] + No one
Step 7:
Hallway [Mary] Mary=1
Step 8:
Hallway [Mary, Michael] + Both present Not alone
Step 9: Hallway [Mary, Mi chahl]] + Not alone
Step 10: Hallway [] No
So, from this, Michael was alone the Hallway during step
36, and John was alone during steps 72 and 73. Therefore, since the question asks for who spent the least amount of time alone in the Hallway, the answer is Michael. </think>
("answer": "Michael")
Table 4: Failure case of reasoning model on dense context task, DC-SA-C. Although the model correctly outlines a valid general reasoning strategy, decides to list all observations (orange), making mistake outlining several steps (red), and ultimately produces the wrong final answer.
25


Reviews:
[{"review_id": "rc56X5tIDA", "reviewer": "Reviewer_URmZ", "summary": "The paper presents MMReD, a benchmark for evaluating long-context and cross-modal reasoning in large language and vision-language models (LLMs and LVLMs). Unlike traditional “Needle-in-a-Haystack” (NIAH) setups focused on sparse retrieval, MMReD emphasizes dense context reasoning, where all observations are informative and models must integrate global patterns rather than locating single facts.", "questions": "1. How do you envision MMReD transferring to real-world multimodal reasoning, such as video QA? Could introducing controlled perceptual noise or natural imagery affect the benchmark’s conclusions?\n2. Since all contexts are synthetic, do you plan to integrate text–image mixtures or human-annotated long-context tasks to bridge synthetic and natural settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. MMReD effectively bridges the gap between retrieval-based and integrative reasoning benchmarks by introducing dense, information-rich multi-modal contexts that better reflect real-world reasoning challenges.\n2. The evaluation suite is extensive and well-designed, covering a wide range of strong baseline and state-of-the-art models across relevant modalities and tasks.\n3. Comprehensive ablation studies offer deep insights into model behavior, highlighting the limitations of current approaches in handling dense contextual reasoning.", "weaknesses": "1. The benchmark is constructed in an artificial and controlled environment, which may limit the model's ability to generalize the the real world modalities and with more noises.\n2. The close-ended question with exact-match accuracy may limit the insight into different models' performance in dense context reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JdlNmdu38e", "reviewer": "Reviewer_MPFi", "summary": "The paper introduces a new multimodal (vision + text) benchmark for dense context reasoning. Compared to existing benchmarks, it challenges models to identify and interpret global patterns across entire contexts. The tasks vary in difficulty, with a good number being challenging for existing models. Investigation also shows conventional fine-tuning techniques do not help improve the performance and so more advanced strategies will be needed to address the challenge.", "questions": "* How do the performances of models for text-only vs multimodal versions of the benchmark compare?\n* Does in-context learning help for these sorts of tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The benchmark is well motivated and studies a relatively valuable point about dense context reasoning in multimodal tasks.\n* A large number of models is evaluated, including many recent ones.\n* The benchmark has setups that are doable for existing models, but also setups that are too challenging for current models, so the benchmark is likely to be relevant for at least some time.\n* Various related questions are studied, and these help probe the performance of the models on the task further.\n* The paper is clearly written and easy to read.", "weaknesses": "* All tasks come from one environment, so one could say that the benchmark is somewhat limited in this sense. A benchmark with e.g. three more-distinct types of setups would enable more rigorous investigation of dense context reasoning.\n* There could be a short discussion where the performances of models for text-only vs multimodal versions of the benchmark are compared. Which one is easier / to what extent is the text-only version easier? (one can infer this from the tables though)\n* In-context learning may potentially help so could be interesting to investigate too given its popularity.\n* Minor: the plots would benefit from larger font sizes where possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iPyDZaUDuI", "reviewer": "Reviewer_6qaP", "summary": "This paper introduces a new benchmark, MMReD (Multi-Modal Reasoning in Dense context), to evaluate the ability of large LLMs and LVLMs to perform complex reasoning over long sequences of information. The authors argue that existing benchmarks do not adequately test a model's ability to reason in \"dense context\" scenarios where all information is potentially relevant. MMReD gives models a series of images depicting characters in different rooms and asks questions that require tracking entities, understanding spatial relationships, and identifying global patterns across the entire sequence. The study evaluates a wide range of frontier models and reveals a consistent drop in performance as the context length increases. The results show that even the most advanced models struggle with these dense context tasks. Also, common fine-tuning techniques offer little improvement.", "questions": "* The poor performance of video-oriented LVLMs was quite surprising. Do you have any hypotheses for this? Could it be related to their training strategies or a bias in their training data?\n\n* Do you believe the long-context reasoning is an inherent limitation of the Transformer architecture, or is it a problem that can be addressed with different training data and methodologies?\n\n* The paper focuses on tracking and counting tasks. Have you considered expanding the benchmark to include more abstract forms of reasoning, such as inferring intentions or predicting future states based on observed patterns?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper argues that \"Needle-in-a-Haystack\" tasks (i.e., testing retrieval) are fundamentally different from \"dense context\" tasks, which require integration and reasoning. Figure 4 provides evidence that performance on these two task types does not correlate well at longer context lengths. This shows that dense context is fundamentally different from retrieval tasks, such as the needle-in-a-haystack problem.\n\n* The authors simplify the visual and linguistic elements and isolate the  reasoning capabilities they want to measure. This avoids confounding variables and ensures that the results reflect its capacity for dense context reasoning. Furthermore, the measure to ensure unique sequences and balanced answer distributions prevents models from relying on memorization or statistical shortcuts, making the evaluation more robust.\n\n* The paper also conducts a series of ablation studies. By testing the effects of model size, fine-tuning methodologies (SFT and GRPO), multimodal adapter types, and video-specific pooling methods, the authors provide multiple views on the problem. The finding that reasoning-specialized LLMs outperform complex LVLMs on the textual version of the tasks is particularly interesting. Specifically, multimodal instruction tuning and video-centric pretraining may be detrimental to this type of reasoning. This suggests that the bottleneck is not multimodal perception but a limitation in long-sequence reasoning.", "weaknesses": "* The benchmark's controlled & minimalist design is a potential weakness. The environment is a highly structured, symbolic \"toy world\" with fixed rules (a 2x3 grid, one character moves per step). This does not reflect the ambiguity and chaotic nature of real-world scenarios, which involve complex simultaneous interactions, occlusions, and less predictable patterns. While an ablation with synthetic \"perceptual noise\" is included, it doesn't capture the true complexity of real-world visual and narrative understanding.\n\n* Because the benchmark is generated in a way that follows clear logical rules, there is a risk that future models could add an ad-hoc module specifically to solve MMReD-like tasks without truly developing generalized dense context reasoning. For example, a model could develop a specialized internal module for tracking entities in a grid.\n\n* The paper categorizes questions into two main buckets: Needle-in-a-Haystack (NIAH) and Dense Context (DC). However, the degree of \"density\" is somewhat ambiguous. An analysis of how performance degrades based on the degree of context density required to solve the question could yield deeper insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nhArzmwQxO", "reviewer": "Reviewer_mQFA", "summary": "This paper introduces MMReD (Multi-Modal REasoning in Dense context), a new benchmark designed to evaluate the ability of large language models (LLMs) and large vision-language models (LVLMs) to reason over long, information-rich sequences. The authors argue that existing benchmarks, which primarily use a \"Needle-in-a-Haystack\" (NIAH) setup, are insufficient as they only test a model's ability to retrieve a specific fact from a large, mostly irrelevant context. MMReD, in contrast, creates scenarios where every piece of the context is important, forcing models to identify and interpret global patterns rather than just locating sparse information.\n\nThe benchmark comprises 24 tasks that test models on tracking entities, spatial relationships, and event-based reasoning over sequences of varying lengths, up to 128 observations.\nThe evaluation revealed a consistent and significant drop in performance across all tested models—including advanced LLMs, LVLMs, and reasoning-specialized architectures—as the sequence length increased. \nThe study demonstrates that state-of-the-art models fail to generalize to dense context reasoning challenges. Furthermore, standard fine-tuning techniques like Supervised Fine-Tuning (SFT) and GRPO were found to be insufficient for enabling this capability.\n\nThe paper concludes that there is an inherent limitation in current model architectures when it comes to reasoning over dense, multi-modal contexts. The MMReD benchmark effectively highlights this gap and underscores the need for new architectural innovations and training paradigms to advance long-context understanding in AI systems.", "questions": "1. Could the authors comment on this distinction between reasoning \"length\" and \"depth\"? Do you believe the architectural limitations observed are primarily related to failures in long-term memory and information aggregation, or do they also impact the ability to construct complex, multi-step logical inferences? How might the models you tested perform on tasks requiring deeper causal or counterfactual reasoning over the same long contexts?\nThis is not a criticism of the current design but an opportunity to add valuable nuance. A brief discussion on this \"length vs. depth\" axis of complexity in the conclusion could help frame the current results more precisely and outline a clear roadmap for the next generation of dense context benchmarks, which might incorporate tasks with deeper logical requirements.\n\n2. Could you provide more specific insights into the nature of this limitation? For instance, did your preliminary analysis reveal specific failure modes in the self-attention mechanism, such as attention becoming overly diffuse and uniform on Dense Context (DC) tasks compared to Needle-in-a-Haystack (NIAH) tasks? Or is the failure more related to information decay and representational corruption as information is passed through successive layers?\nThe paper's impact could be significantly strengthened by including a qualitative analysis, perhaps in the appendix, that delves into these potential causes. For example, providing attention visualizations for a successful short-context DC task versus a failed long-context one could offer powerful visual evidence to support your central claim and diagnose the failure mode more precisely.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Novel Problem Formulation: The paper introduces \"dense context reasoning\" as a distinct capability from standard \"Needle-in-a-Haystack\" (NIAH) retrieval, a significant conceptual shift.\nCreative Benchmark Design: MMReD uses a minimalist visual and linguistic environment to effectively isolate reasoning capabilities from perceptual or language complexity.\nClear Writing and Structure: The paper is well-written and logically organized, with an effective abstract and introduction that clearly motivate the work .\nChallenges Evaluation Paradigms: The work significantly challenges the sufficiency of the dominant NIAH paradigm, showing it is not a reliable indicator for complex reasoning.", "weaknesses": "The paper's central conclusion is that the observed performance degradation reveals an \"inherent limitation in current model architectures\". While the extensive experiments show that models fail, the paper provides limited insight into why they fail at a mechanistic level.\n\nThe claim is a high-level one, and the analysis stops short of diagnosing the specific architectural components that are failing. It is unclear if the bottleneck lies in the attention mechanism's ability to handle uniform information density, the decay of information in positional embeddings over long distances, or how representations are processed through successive layers.\n\nThe work could be substantially improved by including more targeted analysis to diagnose the failure modes. An investigation of the models attention patterns on long dense-context (DC) tasks would be highly valuable. Visualizing attention maps could reveal whether attention becomes overly diffuse, incorrectly focuses on recent tokens (recency bias), or fails to integrate information from distant parts of the sequence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)