Cambrian-S: Towards Spatial Supersensing in Video
Authors: Shusheng Yang
Jihan Yang
Pinzhi Huang
Ellis L Brown II
Zihao Yang
Yue Yu
Shengbang Tong
Zihan Zheng
Yifan Xu
Muhan Wang
Rob Fergus
Yann LeCun
Li Fei-Fei
Saining Xie
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Langauge Model
Super Sensing Model
Spatial Understanding
Video Understanding
Memory
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We argue that progress in true multimodal intelligence calls for a shift from reactive, task-driven systems and brute-force long context towards a broader paradigm of supersensing. We frame spatial supersensing as four stages beyond linguistic-only understanding: semantic perception (naming what is seen), streaming event cognition (maintaining memory across continuous experiences), implicit 3D spatial cognition (inferring the world behind pixels), and predictive world modeling (creating internal models that filter and organize information). Current benchmarks largely test only the early stages, offering narrow coverage of spatial cognition and rarely challenging models in ways that require true world modeling. To drive progress in spatial supersensing, we present VSI-SUPER, a two-part benchmark: VSR (long-horizon visual spatial recall) and VSC (continual visual spatial counting). These tasks require arbitrarily long video inputs yet are resistant to brute-force context expansion. We then test data scaling limits by curating VSI-590K and training Cambrian-S, achieving +30% absolute improvement on VSI-Bench without sacrificing general capabilities. Yet performance on VSI-SUPER remains limited, indicating that scale alone is insufficient for spatial supersensing. We propose predictive sensing as a path forward, presenting a proof-of-concept in which a self-supervised next-latent-frame predictor leverages surprise (prediction error) to drive memory and event segmentation. On VSI-SUPER, this approach substantially outperforms leading proprietary baselines, showing that spatial supersensing requires models that not only see but also anticipate, select, and organize experience.

Published as a conference paper at ICLR 2026
CAMBRIAN-S: TOWARDS SPATIAL SUPERSENSING IN VIDEO
Shusheng Yang' Jihan Yang'.', Pinzhi Huang".,, Ellis Brown'-1, Zihao Yang', Yue Yu', Shengbang Tong', Zihan Zheng', Yifan Xu', Muhan Wang', Rob Fergus', Yann LeCun', Li Fei-Fei", Saining Xie Equal contribution, 1 CCore contributor, 'New York University, 2Stanford University
ABSTRACT
We argue that progress in true multimodal intelligence calls for a shift from reactive, task-driven systems and brute-force long context towards broader paradigm of supersensing. We frame spatial supersensing as four stages beyond linguistic-only understanding: semantic perception (naming what seen), streaming event cog- nition (maintaining memory across continuous experiences), implicit 3D spatial cognition (inferring the world behind pixels), and predictive world modeling (cre- ating internal models that filter and organize information). Current benchmarks largely test only the early stages, offering narrow coverage of spatial cognition and rarely challenging models in ways that require true world modeling. To drive progress in spatial supersensing, we present VSI-SUPER, a two-par -aropttt benchmark: VSR (long-horizon yisual spatial recall) and VSC (continual yisual spatial counting). These tasks require arbitrarily long video inputs yet are resistant to frute-force context expansion. We then test data scaling limits by curating VSI-590K and train- ing Cambrian-S, achieving +30% absolute improvement on VSI-Bench without sacrificing general capabilities. Yet performance on VSI-SUPER remains limited, indicating that scale alone is insufficient for spatial supersensing. We propose predictive sensing as a path forward, presenting a proof-of-concep! in which a self- supervised next-latent-frant frame predictor leverages surprise (prediction error) to drive memory and event segmentation. On VSI-SUPER, this approach substantially our- performs leading proprictary baselines, showing that spatial supersensing requires models that not only see but also anticipate, select, and organize experience.
INTRODUCTION
video is not just a sequence of frames in isolation. It is a continual, high-bandwidth projection of a hidden, evolving 3D world onto pixels (Gibson, 2014; Marr, 2010). Although multimodal large language models (MLLMs) have advanced rapidly by pairing strong image encoders with language models (Achiam et al., 2023; Team et al., 2024; Anthropic, 2024; Liu et al., 2023; Tong et al., 2024a), most video extensions (Wang et al., 2024d; Li et al., 2025a; Bai et al., 2025a) remain fundamentally constrained. They still treat video as sparse frames, underrepresent spatial structure and dynamics (Yang et al., 2024d), and lean heavily on textual recall (Zohar et al., 2025), thus overlooking what makes the video modality uniquely powerful.
In this paper, we argue that advancing toward true multimodal intelligence requires a shift from language-centric perception toward spatial supersensing: the capacity not only to see, but also to construct, update and predict with an implicit model of the 3D world from continual sensory experience. We do not claim to realize supersensing here; rather, we take an initial step toward it by articulating the potential path and by demonstrating early prototypes along that path:
0. (Linguistic-only understanding): sensory capabilities; reasoning confined text and symbols. Current MLLMs have progressed beyond this stage, yet still retain traces of its bias.
1. Semantic perception: parsing pixels into objects, attributes, and relations. This corresponds to the strong multimodal "show and tell" capabilities present in MLLMs.
2. Streaming event cognition: processing live, unbounded streams while proactively interpreting and responding ongoing events. This aligns with efforts make MLLMs real-time assistants.
3. Implicit 3D spatial cognition: understanding video as projections of 3D world. Agents must know where and how things relate, and how configurations change over time. Today's 's video models remain limited here.
4. Predictive world modeling: the brain makes unconscious inferences (Von Helmholtz, 1867) by predicting latent world states based on prior expectations. When these predictions are violated, surprise guides attention, memory, and learning (Friston, 2010; Stahl & Feigenson, 2015; Kennedy et al., 2024). However, current.

---

Published as a conference paper at ICLR 2026
Rd Semantic Porception
(
Linguisti Only Understanding
Streaming Event Cognition
Spatial Cogniton
Predictive World Modeling
Tergie poanple
cont chhertos mny aad
wales inpiict
Fradiciva del wentdis aucid mucien
TASK-ORIVEN
WORLD
VIDEO AS MEDIUM SPATIAL PRETACTIVE SENGING
Figure 1: From pixels to predictive mind. We look beyond linguistic-only understanding to envision multimodal intelligence that sees, remembers. and reasons as part of a continuous, lived world. It begins with semantic perception: naming and describing what is seen, Streaming always-on sensing across continuous input streams. Spatial cognition captures the implicit 3D structure of vidco. Finally, predictive world model emerges, one that learns passively from experience, updates through prediction and surprise, and retains information for future use. Lower illustration: Video serves as ideal experimental domain. Models must advance from frame-level Q&A to constructing implicit world models that enable deeper spatial reasoning, scale to unbounded horizons, and achieve supersensing.
cognition goes further, enabling
multimodal systems lack an internal model that anticipates future states and uses surprise to organize perception for memory and decision making.
Our paper unfolds in three parts. First ($ 2), we re-examine existing benchmarks through the lens of our supersensing hierarchy. We find that most benchmarks map to the first few stages, while some, such VSI-Bench (Yang al., 2024d), begin to probe spatial reasoning. However, none sufficiently address the final crucial stage of predictive world modeling. To make this gap concrete and motivate a shift in approach, we introduce VSI-SUPER (VSI stands for visual-spatial intelligence), two-part benchmark for spatial supersensing: VSI-SUPER Recall (VSR) targets oing-horizon spatial observation and recall, while VSI-SUPER Count (VSC) tests continual counting across changing viewpoints and scenes. Built from arbitrarily long spatiotemporal videos, these tasks are deliberately resistant to the predominant multimodal recipe; they require perception to be selective and structured rather than indiscriminately accumulated. Even the best long-context commercial models fail on it.
Second ($ 3), we investigate whether spatial supersensing is simply a data problem. We curate VSI-590K, spatially focused instruction-tumin corpus over images and videos, which use to train Cambrian-S, a family of spatially-grounded video MILMs. Under the current paradigm, careful data design and training push Cambrian-S state-of-the-art spatial cognition on VSI-Bench (>30% absolute gain) without sacrificing general capabilities. Nevertheless, Cambrian-S still falls short on VSI-SUPER, indicating that scaling alone is not sufficient for spatial supersensing.
This motivates the third and final part ($ 4), where we propose predictive sensing as a first step toward new paradigm. We present a proof-of-concept solution built upon self-supervised next-latent-fran prediction. Here, we leverage the model's 's prediction error, or "surprise," for two key functions: (1) managing memory by allocating resources to unexpected events, and (2) event segmentation, breaking unbounded streams into meaningful chunks. We demonstrate that this approach, though simple, significantly outperforms strong long-context baselines such as Gemini-2.5 on our two new tasks. Although not final solution, this result provides compelling evidence that the path to true supersensing requires models that not only see but actively predict and learn from the world.
Our work makes the following contributions. (1) We define hierarchy for spatial supersensing and introduce VSI-SUPER, a supersensing benchmark that reveals the limitations of the current paradigm. (2) We develop Cambrian-S, Cambrian-S S serves as a powerful new baseline, and, by delimiting the boundaries of current methods new benchmark, paves the path for new paradigm. (3) We propose predictive sensing
ofte-of-the-ar model that pushes the limits of spatial cognition.
2

---

Published as a conference paper at ICLR 2026
owW.uue Farmes
AITarappe FOSBAL Clack
Caplort
stanee
MME: VideoMMIE
EguSchema
VM:
VideoMMMU
LV:
Long VideoBench
TM:
Tomato
001 Captooe
(0 dhance
MV: MVBench
Perception Test
HV: HourVidea
VSIRench
VSR: VSI-SUPER Rec.
VSC:
VSI-SUPER Cut.
1
Figure 2: Benchmark diagnostic results reveal varying dependence on visual input. We evaluate model
under three conditions: (a) multiple frames, (b) single frame, and frame captions, benchmarked against chance-level and blind test results. VSR and VSC are new supersensing benchmarks in Section 2.2.
promising new direction for MLLMs, showing that leveraging model surprise is more effective for long-horizon spatial reasoning than passive context expansion.
2 BENCHMARKING SPATIAL SUPERSENSING
To ground our pursuit of spatial supersensing, we first establish how to measure it. This section undertakes two-part investigation into benchmarking this capability. We begin by auditing suite of popular video MLLM benchmarks, where our analysis (Figs. 2 and 12) reveals that they overwhelmingly focus on linguistic understanding and semantic perception while neglecting the more advanced spatial and temporal reasoning required for supersensing (Section 2.1). To address this critical gap. then introduce VSI-SUPER, new benchmark specifically designed to probe these harder, continual aspects of spatial intelligence in arbitrarily long streaming scenarios (Section 2.2). the limits of the current MLLM paradigm.
We this benchmark
2.1 DECONSTRUCTING EXISTING VIDEO BENCHMARKS
To assess if existing benchmarks evaluate true visual sensing or simply rely on language priors, we conduct a series of diagnostic tests. We use our base Cambrian-1 (Tong et al., 2024a) model to probe a suite of representative video benchmarks under varied input conditions, allowing us to disentangle the underlying task demands from the capabilities of more complex video-specific architectures. We establish several experimental conditions for feeding video inputs: Multiple Frames (32 uniformly sampled frames), Single Frame (the middle frame), and Frame Captions. We compare these against two baselines: Blind Test (all visual input is ignored) and Chance Ace (randomly guessing). We conduct fine-grained analysis of each benchmark's 's characteristics by comparing performance across these conditions and baselines (diff (A, B) = A-B).
Results presented in Fig. (a-c) demonstrate that, an image-based MLLM without any video post- training, can attain reasonable performance across many benchmarks, in some instances surpassing chance-level accuracy by 10-30% (see Fig. 2-g.h). This suggests that much of the knowledge these benchmarks target is accessible via standard single-image instruction-tune -uning pipelines. Nevertheless, on two existing datasets, VSI-Bench (Yang al., 2024d) and Tomato (Shangguan et al., 2025), the model's performance falls below chance-level. Employing textual captions in place of visual inputs also yields notable performance improvements, surpassing chance accuracy by more than 20% on benchmarks such as EgoSchema (Mangalam et al., 2023). VideoMME (Fu et al., 2025), Bide (Wu al., 2024b), VideoMMMU (Hu et al., 2025a), Perception Test (Patraucean et al., 2023), and MVBench (Li et al., 2024d) (Fig. 2-i). Similar conclusions can be drawn when comparing benchmark performance against blind test results (Fig. 2-d.f). Such performance implies that these benchmarks primarily probe abilities inferable from textual summaries of video content. The performance difference between using "multiple frames" and "frame captions" (Fig. 2-j) indicates benchmark's demand for nuanced visual sensing language-centrr nature. Our analysis places VidcoMMMU, EgoSchema, VideoMME, Perception Test, and Long VideoBench in this latter category, indicating their potential reliance linguistic understanding rather than visual cues.
3

---

Published as a conference paper at ICLR 2026
Existing benchmarks overwhelmingly focus linguistic understanding and semantic perception while neglecting spatial and temporal reasoning required for supersensing.
We hope to emphasize the inherent challenges in benchmarking and the impracticality of creating a single, all-encompassing benchmark to evaluate every capability. For example, reliance language priors should not be viewed merely as a drawback, as access to rich world knowledge and its effective retrieval is undoubtedly beneficial in many scenarios. We argue that video benchmarks should not be treated as measuring a single, uniform notion of "video understanding." Instead, their design and evaluation should be grounded in the specifie capabilities they aim to assess. The preceding analyses are therefore intended to guide the development of tasks that more effectively drive progress towards spatial supersensing, which will be the central focus of the rest of the paper.
2.2 VSI-SUPER: TOWARDS BENCIMARKING SPATIAL SUPERSENSING IN MLLMS
Referring to Fig. 1, spatial supersensing requires MLLMs to have four key capabilities. However, as outlined by our analysis in Fig. 2, most existing video QA benchmarks mainly evaluate the linguistic understanding and semantic perception aspects, which are more reactive and driven by specific tasks (Fu et al., 2025; Mangalam et al., 2023; Hu et al., 2025a). While recent research has begun to address streaming event cognition through continual sensing, memory architectures, and proactive answering (Chen et al., 2024c; Qian et al., 2025; Niu et al., 2025; Wu et al., 2024a; Song et al., 2024; Zhang et al., 2025a), this capability is often engineered at test time rather than being native model skill. Additionally, although VSI-Bench (Yang et al., 2024d) takes an initial step toward examining spatial cognition, its videos remain short-form and single-scene, and it neither formalizes the problem nor evaluates the essential capability of predictive modeling of the world.
To illuminate the gap between current MLLMs and spatial supersensing, we introduce VSI-SUPER, two-part benchmark for continual spatial sensing. The tasks are intuitive and generally easy for humans, where one simply watches and keeps track of what happens, but they remain surprisingly challenging for machines. They demand selective filtering and structured accumulation of visual information across unbounded spatial videos to maintain coherent understanding and answer questions. Importantly, they are resistant to brute-force context expansion, exposing the need for true spatial reasoning. We detail the two components below.
VSI-SUPER Recall: Long-horizon spatial observation and recall. The VSR benchmark requires MLLMs to observe long-horizon spatiotemporal videos, and sequentially recall the locations of an unusual object. As shown in Fig. 3, to construct this benchmark, human annotators use an image editing model (i.e., Gemini (Co- manici et al., 2025)) to insert surpris- ing or out-of-place objects (e.g., a Teddy Bear) into four distinct frames (and spatial location) of a video capturing walkthrough of an indoor environment (Dai et al., 2017; Yeshwanth al., 2023; Baruch et al., 2021). This edited video then concatenated with other similar room-tour videos to create an arbitrarily long and continuous visual stream. This task parallels the reelle-i-aa-hassted (NIAH) test commonly used in the language domain to stress test the long-context capabilities of LLMs (Liu al., 2024b). Similar NIAH setups have also been proposed for long-video evalua- tion (Zhao et al., 2024; Wei et al., 2025; Hu et al., 2025b). However, unlike benchmarks that insert unrelated text segments or frames, VSR preserves the realism of the "needle" through in -r-frame editing. It further extends the challenge by requiring sequential recall, effectively a multi-hop reasoning task, and remains arbitrarily scalable in video length. To thoroughly evaluate model performance across different time scales, the benchmark is provided in five durations: 10, 30. 60, 120, and 240 minutes. Further details on the VSR benchmark construction are provided in Appendix D.
Il Fremre Edting
Random Video Corcenenatin
Whish of the following corecclly Torlet, Bathoub, šink, Floor
onder which the Teddy Beer appeared the video?
Berntub, Tollet, Sii. Floor Tollet, Bathoub, Sink
Sink, Floss Batioub
Figure 3: VSR Task: recall the placements of novel objects in the
correct order of their appearance across arbitrarily long videos.
VSI-SUPER Count: Continual counting under changing viewpoints and scenes. Here we test the capacity of MLLMs to continuously accumulate information in long-form spatial videos. To build VSC, concatenate multiple room-tour video clips from VSI-Bench (Yang et al., 2024d) and task models with counting the total number of target objects across all rooms (see Fig. 4). This setting is challenging because the model must handle viewpoint shifts, repeat sightings, and scene
4

---

Published as a conference paper at ICLR 2026
transitions, all while maintaining a consistent cumulative count. For humans, counting is intuitive and generalizable process. Once the concept of "one" is understood, extending it to larger quantities is natural. In contrast, as we later demonstrate, current MLLMs lack true spatial cognition and depend excessively on learned statistical patterns.
In addition to standard evaluations (i.e., ask question at the end of video), we query the model multiple times- tamps to assess its performance in streaming settings, where the correct answer in VSC evolves dynamically over time. To examine long-term con- sistency, VSC includes four video durations: 10, 30, 60, and 120 minutes. For this quantitative task, we report results using the mean relative accuracy (MR.. metric, consistent with the VSI-Bench evaluation protocol (Yang et al., 2024d).
Strtaninng How many oifieent chartt am there this vide?
A-2 A-3 A Ar
A 20
Figure 4: Overview of the VSC: count the total number of target objects across all rooms in long-horizon, multi-room videos.
State-of-the-art models struggle on VSI-SUPER. To test whether VSI-SUPER poses real challenge for frontier MLLMs, we evaluate the latest Gemini-2.5-Flash (Team et al., 2024). As shown in Table 1, the model reaches its context limit when handling two-hour videos, despite a context length of 1,048.576 tokens. This highlights the open-ended nature of video understanding, where continuous streams effectively require an "infinite-in, infinite-out" context and can grow arbitrarily long, suggesting that simply scaling up tokens, context length, or model size may not suffice. Though synthetic, our benchmark reflects a real challenge in spatial supersensing: humans effortlessly integrate and retain information from ongoing sensory experiences that unfold over hours or years, yet current models lack comparable mechanisms for sustained perception and memory. Gemini-2.5-Flash demonstrates strong performance on semantic- perceettiiooooooonnnnno and linguistic-une undrriunnnnrrrrnnnneooooo focusse video benchmarks such as VideoMME (Fu et al.. 2025) and VideoMMMU (Hu et al., 2025a), achieving around 80% accuracy. However, even for 60-minute videos in VSI-SUPER that fall well within its context window, performance on VSR and VSC remains limited- 41.5 and 10.9, respectively. As shown in Fig. 13, the model's predicted object counts fail to scale with video length or the true number of objects, instead saturating at a small constant value, suggesting a lack of generalization in counting ability and a reliance on training distribution priors.
Model
VideoMMEIPi et al. 2025)
YideMMMMUUH dt, 2015aa
VSI-Benchi Yang al. 20240)
60 min
VSR
120 mis
60 minn
VSC
120 min
Gesnits-2.5 2.5. Flash
$1.5
45.7
10.9
Out
Table 1: Gemini-2.5- FlasFl demonstrates strong performance on general video benchmarks but shows clear limitations towards spatial supersensing
VSI-SUPER challenges current paradigm. Although the task setup is simple, the challenge posed by VSI-SUPER goes beyond spatial reasoning and reveals fundamental limitations of current paradigm.
VSI-SUPER tasks challenge the belief that scaling alone guarantees progress.
By allowing arbitrarily long video inputs that emulate the dynamics of streaming cognition, VSI- SUPER is intentionally constructed to exceed any fixed context window. This design suggests that frame-by-frame tokenization and processing are unlikely to be computationally viable as a long- term solution. Humans address such problems efficiently and adaptively by selectively attending to and retaining only small fraction of sensory input', often unconsciously (Fei-Fei et al., 2007; Von Helmholtz, 1867). This predictive and selective mechanism, core human cognition, remains absent in current MLLMs but is fundamental to a predictive world model.
VSI-SUPER tasks demand generalization to new temporal and spatial scales at test time.
For example, VSC requires counting in arbitrarily long videos, similar to how humans, who understand the concept of counting, can extend it to any number. The key is not maintaining extremely long context window, humans do not retain every visual detail from extended visual experiences, but rather learning the process of counting itself. Together, these challenges call for paradigm shift. Rather than relying solely on scaling data, parameters, or context length, future models should learn internal world models, perceiving and predicting within an endless visual world across space and time.
Each eye's 6 million cone photoreceptors can send about 1.6 Ghits/s, yet the brain uses only 10 bits/s to guide behavior (Koch al., 2006; Zheng& & Meister, 2025).
5

---

Published as a conference paper at ICLR 2026
3 SPATIAL SENSING UNDER THE CURRENT PARADIGM
In this section, we try to answer the question: Is limited spatial sensing simply a data issue? We begin by enhancing Cambrian-1 (Tong et al., 2024a) with series of architectural and training improvements to establish a stronger image MLLM as our base model (Section 3.1). Then, we construct large-scale, spatial- foccued instruction-tuni tonnnngttnnnngggn dataset. VSI-590K (Section 3.2) to provide a strong data foundation for spatial sensing. Finally, with a refined training recipe (Section 3.3), we introduce the spatially-grounal Cambrian-S model family (Section 3.4).
3.1 BASE MODEL TRAINING: UPGRADED CAMBRIAN-1
We begin by developing an image-based MILMM base model, as robust semantic perception forms the foundation for higher-level spatial cognition. We follow the two-stage training pipeline of Cambrian-1 (Tong et al., 2024a). We upgrade the visual encoder to SigLIP2-SO400m (Tschannen et al.. 2025) and the language model to the instruction-tuned Qwen2.5 (Yang et al., 2024a). For the vision-language connector, we adopt simple two-layer MLP primarily for its computational efficiency. Full implementation details are provided in Appendix F.
3.2 SPATIAL VIDEO DATA CURATION: VSI-590K
is well recognized that data qual- ity and diversity play a critical role in the training of MLLMs (Tong et al., 2024a; McKinzie et al., 2024). We hypothesize that the performance gap on VSI-Bench (Yang
Nore
Dhart
Dipect
Devetpiion
23 Antuedation Vileon
Awertaee Wwloodheeeee and koo
Figure 5: VSI-590K data curation pipeline.
al., 2024d) comes mainly from the lack of high-quality, spatially grounded data in current inttructioontuuinng datasets (Zhang et al., 2025b; Cui et al., 2024). To fill this gap, we build VSI-590K, large-scale instruction sotlictionttnniing tuning dataset designed to improve visual-spatial understanding. We define 12 question types to construct comprehensive and diverse set of questions for instruction tuning, please refer to Appendix E.1 for details about the question type definition and taxonomy. Our dataset is constructed from a diverse span of data sources and types (i.e., simulated and real), please refer to Appendix E.2 for details about the data curation, processing, and QA construction pipeline.
VSI-590K data source ablation. To evaluate the effectiveness of VSI-590K, we perform an ablation study by finetuning the improved C MLLM described in Section 3.1 with part of the video instruction tuning samples from V..VVAVVideo, 178K (Zhang et al., 2025b). This model serves as the baseline in Table 7. The contribution of each data source is evaluated by fine-tuning the model on individual datasets as well as their combination. The VSI-590K Full Mix achieves the highest overall performance on video spatial reasoning tasks, outperforming both the baseline and all single-source counterparts. All sources contribute positively after fine-tuning, though their effectiveness varies.
Data effectiveness: annotated real vidcos > simulated data > ascudo-annotatie images.
This indicates that videos are inherently more informative than static images for spatial reasoning. as training exclusively on video data yields superior performance on both video- and image- based spatial reasoning benchmarks. These findings support the intuition that the temporal continuity and multi-view diversity of videos are key to developing robust spatial representations.
3.3 ROST-TRAINING RECIPE FOR SPATIAL SENSING
We further analyze and ablate our video instruction-tunm tuninng pipeline, focusing on the roles of the pretrained base video model and the instruction-tuning dataset mixture. As shown in Table 11, we begin with four base models that represent progressive increase in video understanding capability: A1 is trained only with image-text alignment on Cambrian-1 alignment data. The language model is identical to base QwenLM it is frozen during training. A2 is finetuned with image instruction tuning on top of AI, essentially our improved Cambrian-1. A3 is initialized from A2 and finetuned on 429K video instruction tuning data. A4 is initialized from A2 and finetuned on 3M video data.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison Cambrian-S with other leading MLLMs. Cambrian- S outperforms both proprietary and open-source models across benchmarks and model sizes. Detailed evaluation setups are listed in Appendix G.
Valoo
bage
Longs
Pereage Tea
Marsy
Moai
Boae LM
CPT-4
UNK
3401
55.7
37.2
71.9
01.3
66.7
66.8
442
INNK
654
36.1
37.3
75.
72.2
53.
445
Spaee Model
22.5
26.5
57.5
Qnan N2-25-1
50.5
28.6
65.1
05.3
47.4
50.
UMA
5,77
45.4
S6.3
66.9
41.2
25.8
62.1
72.
560
Qran28-78
67.5
20.3
27/1
65-4
76.
9144
66-5
01.9
Geasssos
20-4
22.7
61.
96.5
S-18
564
36.
73.5
25.3
91.1
60.3
91.
ReafVi 37
Seailahh 1.66
2710
73%
911
06.
$1.1
1.33
Gead-1775
51.8
46.1
564
30.
57.4
65.9
4-0
23.5
Qawi'l
184
11.
518
41.
330
64.0
17.0
Qea2.54L38
20.6
234
44.
31.
36.0
260
We then finetune these models using two different data recipes: (1) VSI-590K only, and (2) VSI-590K mixed with similar amount of general video instruction tuning data.
Stronger hase model with greater exposure to general video data leads to improved spatial sensing.
As shown in Table 11, SFT with stronger base model, one that performs well on general video benchmarks such as VideoMME and EgoSchema, leads to enhanced spatial understanding. This highlights the importance of broad exposure to general video data during base model training.
Mixing general video data prevents the gencralization loss caused by in-domain SFT.
Furthermore, while in-domain SFT solely on VSI-590K achieves the highest performance on VSI- Bench, it results in noticeable decline on general video benchmarks. However, this performance drop can be effectively mitigated by training on a data mix that includes general videos.
3.4 CAMBRIAN-S: PPATIALLY-GROUNDER MLLMS
Building on all the previous insights, we develop Cambrian-S, family of spatially-grounder models with varying LLM scales: 0.5B, 1.5B, 3B, and 7B parameters. These models are built Figure 6: Overall Cambrian-S training pipeline.
Cembrar-1 Taising
Cambrin- Trahing
LanoueLanguse
Image remrctice
Ganaral Vicko
Spesill Vidko
through a four-stage training pipeline specifically designed to first establish general semantic percep- tion and then develop specialized spatial sensing skills, as illustrated Fig. 6. The first two stages adhere to the Cambrian-1 framework to develop strong image understanding capabilities. In stage 3, we extend the models to video by conducting general video instruction tuning on Cambrian-S-3M. S-MM,, a curated dataset composed of 3 million samples (see detailed composition in Fig. 15). In the final and crucial stage 4, the models are trained for spatial sensing. Here, we finetune the models on a blended corpus combining our specialized VSI-590K with proportional subset of the general video data, following the setup described in Table 11. Complete training details are provided in Appendix F.3.
3.5 EMPIRICAL RESULTS: IMPROVED SPATIAL COGNITION
Improved spatial cognition. As shown in Table 2, our models achieve state-of-the-ane art performance
in visual-spatial understanding in video. Cambrian-S-7D achieves 67.5% on VSI-Bench, significantly outperforming all open-source models and surpassing the proprietary Gemini-2.5-Pro by over 16 absolute points. Since our work in this section can be viewed as a data scaling effort, a natural question is: are the performance improvements simply due to broader data coverage (including more diverse visual configurations and question answer pairs). or has the model actually developed stronger spatial cognition? ? First, we emphasize that there is no data overlap between VSI-590K and the benchmark datasets. Although some datasets originate from the same sources (e.g., from ScanNet). we only use the training split, while the benchmarks use validation and test splits. Moreover, we observe clear signs of generalization in spatial reasoning. For example, in "Route Planning" subtask (absents from VSI-590K because of the high annotation cost), Cambrian-S-77 still performs strongly, showing pronounced scaling behavior with increasing model size too (see Table 12). Furthermore. our training approach proves highly effective even with smaller model sizes: our smallest 0.5B model achieves performance comparable to Gemini-1.5 Pro on VSI-Bench. Importantly, this emphasis on
7

---

Published as a conference paper at ICLR 2026
Table Cambrian-S-7B results on VSI-SUPER. Despite strong performance on VSI-Bench, accuracy on VSR drops sharply from 38.3% (10 min) to 0.0% (>60 min), and VSC completely fails.
VSR
Eval Setup
min
30 min
60 min
120 man
240 min
10 mins
30 min
60 min
min
Sampling, 1386-
2617
$0.0
28.2
16.0
0.
FPS Sampling. LFPS
38.3
35.0
6.0
0.0
0.0
0.6
0.0
0.0
0.
spatial reasoning does not come at the expense of general capabilities: Cambrian-S continues to
deliver competitive results on standard video benchmarks (see Table 14 for complete results).
Cambrian-S achieves statc-of-the-ar spatial sensing with robust generalization to unseen spatial question types, while staying competitive in general video understanding.
Robust spatial reasoning on YSl-Bench-Debiassed. To investigate whether Cambrian-S learns to reason visually, we evaluate it VSI-Bench-Debiasee (Brown et al., 2025), a benchmark specif- ically designed to eliminate language shorteuts through debiasing. As shown Table 2, although performance decreases by about 8% compared to standard VSI-Bench, our models still outperform proprictary counterparts, demonstrating robust visual-spatial reasoning capabilities and confirming that our training extends beyond language- anguage-bbsed learning.
Results on VSI-Super: limitations in continual spatial sensing. Despite its strong performance on spatial reasoning tasks in short, pre-segmented videos from VSI-Bench, Cambrian-S isn't sn't well- equipped for continual spatial sensing. This limitation is evident in two ways. First, its performance deteriorates significantly on long videos. As shown in Table 3, when evaluated on VSI-SUPER with 1 FPS sampling in a streaming-style setup, scores drop steadily from 38.3% to 6.0% as video length increases from 10 60 minutes, and the model fails completely on videos longer than 60 minutes. Second, the model has difficulty generalizing to new test scenarios. Although trained on multi-room house tour videos, it fails to handle unseen examples with just few additional rooms. This issue isn't simply about context length: performance drops even on short 10-minute videos that fit comfortably within model's context window. These results highlight that a purely data-driven approach, no matter how much data engineering effort is invested, faces fundamental limits. Addressing these limitations calls for paradigm shift toward systems that can actively model and anticipate the world while organizing their experiences more efficiently, which we explore next.
Scaling data and models is essential, but alone it cannot unlock true spatial supersensing.
4 PREDICTIVE SENSING AS A NEW PARADIGM
Performance of both Seemini-2.5-Flat (Table 1) and Cambrian-S (Table 3) reveals fundamental paradigm gap: scaling data and context alone is insufficient for supersensing. We propose predictive sensing as path forward, where models learn to anticipate their sensory input and construct internal world models handle unbounded visual streams. We prototype this concept via self- supervised next-latent-frane prediction approach (Section 4)... The resulting prediction error serves as a control signal for two key capabilities: memory management to selectively retain important information (Section 4.2), and event segmentation to partition unbounded streams into meaningful chunks (Section 4.3). We demonstrate through two case studies on VSI-SUPER that this approach substantially outperforms strong long-context and streaming video model baselines.
4.1 PREDICTIVE SENSING VIA LATENT FRAME PREDICTION
We implement our predictive sensing paradigm through a lightweight, self-supervised mod- ule called the Latent Frame Prediction (LFP) head, which is trained jointly with the primary instruction- -tuningtnnnggng objective. This is achieved by modifying the stage 4 training recipe as follows: Latent frame prediction head: We introduce an LFP Head, a two-layer MLP that operates in parallel with the language head, to predict the latent representation of the subsequent video frame. This architecture is illustrated in the top left of Fig. 7. Learning objectives: To optimize the I.FP head, we introduce auxiliary losses,
Figure 7: Training and inference pipeline for the latent frame prediction (LFP) approach.
mean squared error (MSE) and cosine distance,
8

---

Published as a conference paper at ICLR 2026
which measure the discrepancy between the predicted latent feature and the ground truth feature of the next frame. A weighting coefficient balances the LFP loss against the primary instruction-tuning next token prediction objective. Data for LFP training: We augment stage 4 data with a 290K video subset from VSI-590K used exclusively for the LFP objective. Unlike instruction tuning, these videos are sampled at FPS to ensure uniform temporal spacing for latent frame prediction.
During this modified stage 4 finetuning, we train the connectors, language model, and both the language and LFP heads jointly in an end-to-end manner, while keeping the SigLIP vision encoder frozen. All other training settings remain unchanged. For brevity, we still denote the model jointly optimized with the LFP objective as Cambrian-S in subsequent experiments.
Estimating surprise via prediction error. During inference, we leverage the trained LFP head to evaluate the "surprise" (or, fiolation-o-EEpeecatiii (Burgoon & Hale, 1988)) for every incoming visual sensory input. Specifically, during inference, video frames are fed into Cambrian-S at constant sampling rate. Unless otherwise noted, the videos in the following experiments are sampled at I FPS before being input into the model. As the model receives incoming video frames, continuously predicts the latent features the next frame. We then measure the cosine distance between the model's 's prediction and the actual ground truth feature of that incoming frame. This distance serves as a quantitative measure of surprise: a larger value indicating a greater deviation from the model's 's learned expectations. This surprise score acts as a powerful, self seff-supervised guidance signal for the downstream tasks explored next.
Inference:
4.2 CASE STUDY I: SURPRISF-DRIVEN MEMORY MANAGEMENT SYSTEM FOR VSR.
Surprise-driven memory management system. We build one memory management system that dynamically compresses and consolidates visual streams based on the estimate of "surprise" As shown in Fig. 8-(a), we encode incoming frames using sliding window attention with fixed window size. The latent frame prediction module then measures a "surprise level" and assigns it to each frame's KV caches. Frames with surprise level below predefined threshold undergo 2x compression before being pushed into long-term mem- ory. To maintain stable GPU memory footprint, this long-term memory is constrained to a fixed size by a con- solidation function that, once again, operates based on surprise: dropping or merging frames according to their surprise scores (see Fig. 8(b)). Finally, upon receiving user query, the system retrieves the top- most relevant frames from the long-term memory by caleulating the co- sine similarity between the query and the stored frame features (see Fig. 8-(c)). See Appendix H.2 for more design details. While prior works have explored memory system designs for long videos (Song et al., 2024; Zhang et al., 2025a), our focus is on exploring prediction errors (i.e., surprise) as guiding signals.
Visuel Streams Encoding
(bi Long term Memory
Windows MindorcAvartter Avsurtiion
Cother
MLLM
Supprio
Compuex
Memory Retrieval
Juer Churr
Rersiewe
ILM Decocier
Comackdate
Figure 8: Surprise-driven memory man- agement framework for VSR.
(tarmta)))
Dardee Mine
Vodee Ma0))
(a) VSR results
(b) GPU memory usage
(c) Surprise comparison
Figure 9: Surprise-driven memory allows Cambrian-S to maintain strong performance (a) and stable GPU memory footprint (b). LFP prediction error is more robust than adjacent-frand difference as surprise.
Results. We compare Cambrian-S with and without the surprise-based memory system against two advanced proprietary models, Gemini-1.5- 1.5-lash (Team et al., 2024) and Gemini-2.5-Fiaan (Comanici et al., 2025), on the VSR benchmark. As shown in Fig. 9a, Cambrian-S (w/ Mem.) outperforms both Gemini-1.5- 1.5-lahh and Cambrian-S (w/o Mem.) at all video lengths, demonstrating consistent spatial sensing performance across video durations. Although Gemini-2.5-Flas yields strong results for videos within an hour, it fails to process longer inputs. In addition to maintaining high accuracy, Cambrian-S (w/ Mem.) also maintains stable GPU memory usage across different video lengths
9

---

Published as a conference paper at ICLR 2026
(Fig. 9b). This demonstrates that surprise-based memory effectively compresses redundant data without losing critical information. We include two long-video baselines, MovieChat (Song et al., 2024) and Flash-VStream VStreamm(Zhang et al., 2025a), for comparison in Table 17.
Ablation on surprise measurement. Here, we compare our design, prediction error as surprise, to another baseline: SigL.PP feature difference surprise. As shown in Fig. 9c, using prediction error consistently outperforms adjacent- faraccet--fameeee similarity across different video durations.
Predictive sensing provides a more principled approach to modeling the spatiotemporal dynamics of video data than static similarity measures based on per-frame features.
4.3 CASE STUDY II: SURPRISF-DRIVEN CONTINUAL VIDEO SEGMENT FOR VSC.
Surprise-driven event segmentation. In the VSI-SUPER Count (VSC) benchmark, we examine simple setting where surprise is used to segment continuous visual input, identifying scene changes as natural breakpoints that di- vide the video stream into spatially coherent segments. This approach also parallels human problem-solving: when counting objects across large area, people typi- cally focus on one section at time before combining the results. This behavior also related to the "doorway effect" (Radvansky et al., 2011) 1) (i.e., passing through a doorway creates natural memory boundary).
Continus' Visual Streams
Event
Caloulate tuppose
Buffer
surprise
load
LLM Decoder
Query
dkal
As illustrated in Fig. 10, the model continuously accu- mulates frame features in event buffer. When a high- surprise frame is detected, the buffered features are sum- marized produce segment-level answer, and the buffer is cleared to start a new segment. This cycle repeats until the end of the video, after which all segment answers are aggregated form the final output.
Per-segment
answer
supprise
surprise
Figure 10: Surprise-driven event segmen-
tation framework for VSC.
Wide Mone)
(a) VSC results
(b) Different surprise measurement
(c) Streaming
Figure 11: Cambrian-S with surprise-driven event segmentation achieves consistently better results both offline (a) and streaming setups (c). (b) Pred. error as surprise outperforms adjacent- framm difference.
Results. Gemini- 1.5-Flash attains pear-zero performance on VSC (Fig. 11a), showing the task's difficulty. Although Gemini-2.5-Flaast yields much better results on 10-minute videos, its performance declines rapidly on longer videos. In contrast, the surprise-driven event segmentation approach used by Cambrian-S (w/ Surprise Seg.) achieves higher and more stable performance across all video lengths. When the video is segmented using ground-truth scene transitions (i.e., Cambrian-S w/ GT Seg.). performance improves further, representing an approximate upper bound. A deeper analysis in Fig. 17 reveals that Gemini-2.5. 2.5-Faass predictions are confined to limited range and not scale as more objects appear in the video. In contrast, Cambrian-S (w/ Surprise Seg.) produces counts that exhibits a stronger correlation with the true object numbers, indicating better generalization.
Ablation on surprise measurement. We compare surprise-driven approach to adjacent-frane feature similarity (Fig. 1lb). For both methods, we report the best results after hyperparameter tuning.
Evaluation in streaming setup. As the correct answer in VSC evolves throughout the video, we create a streaming QA setup where the same question is asked at 10 different timestamps. The final performance is averaged across all queries. We benchmark against commercial MLLMs marketed for live visual input. As shown in Fig. 1c, although Gemini-Live and GPT-Realtime are intended for streaming scenarios, they achieve under 15% MRA on 10-minute videos and their performance declines to near zero on 120-minute streams, Cambrian-S, however, shows stronger performance, reaching 38% MRA 10-minute streams and maintaining around 28% at 120 minutes.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos. In NeurIPS, 2023.
Anthropic. Introducing claude 3.5 sonnet.
https:: http:://ww.athhrppi..... news//
claude-3----- 3-rode-3-5--o 3-5-nnne 2024.
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. CVPR, 2016.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with predictive architecture. In CVPR, 2023.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506. 099900 2025.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 00008./2900 /2966, 2023b.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.1392. 2025a.
Yutong Bai, Danny Tran, Amir Bar, Yann LeCun, Trevor Darrell, and Jitendra Malik. Whole-body conditioned egocentric video prediction. arXiv preprint arXiv:2506.2 21552, 2025b.
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In CVPR, 2025.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes a diverse real- wolld dataset for 3d indoor scene understanding using mobile RGB-d data. In NeurIPS, 2021.
Ellis Brown, Jihan Yang, Shusheng Yang, Rob Fergus, and Saining Xie. Benchmark designers should "train on the test set" to expose exploitable non- vos-visual shortcuts. arXiv preprint arXiv:2511.04655, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Xindong He, Xu Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. In IROS, 2025.
Judee K Burgoon and Jerold L Hale. Nonverbal expectancy violations: Model elaboration and application to immediacy behaviors. Communications Monographs, 55(1):58-79 79, 1988.
Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models. In ICRA, 2025.
Wenhao Chai, Enxin Song, Yilum Du, Chenlin Meng, Vashisht Madhavan, Omer Bar- Bar-Tal, -eng-eng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and benchmark. In ICLR, 2025.

---

Published as a conference paper at ICLR 2026
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain lchter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision- -ainugla models with spatial reasoning capabilities. In CVPR, 2024a.
Chang Chen, Fei Deng. Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In ICLR, 2024b.
Dongping Chen, Yue Huang. Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. Gui-world: A video benchmark and dataset for multimodal gui-oriented understanding. In ICLR, 2025a.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, fei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videolln-online: Online video large language model for streaming video. In CVPR, 2024c.
Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, et al. Scaling rl long videos. In NeurlPS, 2025b.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. In ICLR, 2025c.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, 2024d.
An-Chich Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang. and Sifei Liu. Spatialrgpt: Grounded spatial reasoning vision-language models. In NeurIPS, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Kenneth James Williams Craik. The nature of explanation. CUP Archive, 1967.
Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang. Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang. Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. URL https: // //sharegpt44. wii.iptt...iitut.ii...
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated reconstructions of indoor scenes. CVPR, 2017.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In NeurIPS, 2022.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, 2017.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. In NeurlPS, 2022.
Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, and Zhongyu Wei. Embspatial-benchh Bench- marking spatial understanding for embodied tasks with large vision-language models. In ACL, 2024.
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language visual representation learning. In ICCV, 2025.
12

---

Published as a conference paper at ICLR 2026
Li Fei-Fei, Asha Iyer, Christof Koch, and Pietro Perona. What do perceive in glance of real-world scene? Journal of vision, 2007.
Karl Friston. The free-energy principle: unified brain theory? Nature reviews neuroscience, 2010.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. In CVPR, 2025.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and practice- survey. Automatica, 25(3):335- 1989.
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, and Yann LeCun. Intuitive physies understanding emerges from self-supervised pretraining on natural videos. arXiv preprint arXiv:2502.11831, 2025.
James J Gibson. The ecological approach visual perception: classic edition. Psychology press, 2014.
Kristen Grauman, Andrew Westhury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In COLM, 2024.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 183. 2018.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
D Hendrycks. Gaussian error linear units (gelus). arXiv preprint 1:X1::16006.0441,, 2016.
Jakob Hohwy. The predictive mind. OUP Oxford, 2013.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-MMMMU Evaluating knowledge acquisition from nulti-discipline professional videos. arXiv preprint arXiv:2501.13822 2025a.
Zi- Yuan Hu, Shuo Liang, Duo Zheng, Yanyang Li, Yeyao Tao, Shijia Huang, Wei Feng, Jia Qin, Jianguang Yu, Jing Huang, et al. Nemo: Needle in montage for video-language understanding. arXiv preprint arXiv:2509.2455 24563, 2025b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276 2024.
Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, et al. Token-efficien long video understanding for multimodal 1lms. arXiv preprint arXiv:2503.04133 2025.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mas: Fast autoregressive transformers with linear attention. In ICML, 2020.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016.
Nicholas GW Kennedy, Jessica C Lee, Simon Killcross, R Fred Westbrook. and Nathan M Holmes. Prediction enror determines how memories organized in the brain. Elife, 2024.
13

---

Published as a conference paper at ICLR 2026
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision- Assion-anguaae--ctinn model. arXiv preprint arXiv:2406. 00200009926,... 2024.
Kristin Koch, Judith McLean, Ronen Segev, Michael A Freed, Michael J Berry, Vijay Balasubrama- nian, and Peter Sterling. How much the eye tells the brain. Current biology, 2006.
Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zisserman, and Federico Tombari. Text- conditioned resampler for long form video understanding. In ECCV, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. TMLR, 2025a.
Bohao Li, Yuying Ge, Yixiao Ge. Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Sced-bench: Benchmarking multimodal large language models. In CVPR, 2024a.
Chengzu Li, Caiqi Zhang, Han Zhou, Nigel Collier, Anna Korhonen, and Ivan Vulić. Topviewrs: Vision-language models as top-view spatial reasoners. In EMNLP, 2024b.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICMI., 2023a.
KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint urXiv:2305.0633. 2023b.
Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In ECCV, 2024c.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. MVbench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024d.
Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nic. Lion-fs: Fast & slow video-language thinker online video assistant. In CVPR, 2025b.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501. 00574, 2024e.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024f.
Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu, Wenxiao Cai, Zheng Liu, and Bo Zhao. Sti-bench: Are milms ready for precise spatial-temporal world understanding? In ICCV, 2025c.
Benlin Liu, Yuhao Dong, Yiqin Wang, Zixian Ma. Yansong Tang, Luming Tang, Yongming Rao, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondences boost temporal reasoning in multimodal language model. In CVPR, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. In ACL, 2024b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024d.
14

---

Published as a conference paper at ICLR 2026
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu- Ku-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. SCIS, 2024c.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024.
Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso de Melo, Jianwen Xie, and Alan Yuille. Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning. In NeurlPS, 2025.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Vidcogpt+: Integrating image and video encoders for enhanced video understanding. arXiv preprint arXiv:2406.09411 2024.
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In CVPR, 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In NeurlPS, 2023.
Maninis, Kaifeng Chen, Soham Ghosh, Arjun Karpur, Koert Chen, Ye Xia, Bingyi Cao, Daniel Salz, Guangxing Han, Jan Dlabal, et al. Tips: Text-image pretraining with spatial awareness. In ICLR, 2024.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvim: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299 2025.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In WACV, 2021.
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mml: methods, analysis and insights from multimodal I1m pre-training. In ECCV, 2024.
Antoine Miech, Dimitri Zhukov, -ean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari. Sile: Improving vision language pretraining with dittiliiillation. In ECCV, 2024.
Junbo NIu, Yifei Li, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, et al. Ovo-bench: How far is your video-Ilms from real-world online video understanding? In CVPR, 2025.
Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi Liu. Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. Spacer: Reinforcing milms in video spatial reasoning. arXiv preprint arXiv:2504.01800 2025.
Abby O'Nell, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In ICRA, 2024.
Junwen Pan, Rui Zhang, Xin Wan, Yuan Zhang, Ming Lu, and Qi She. Timesearch: Hierarchical video search with spotlight and reflection for human long video understanding. arXiv preprint arXiv:2504.01407 2025.
15

---

Published as a conference paper at ICLR 2026
Xiaqing Pan, Nicholas Charron, Yonggian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In ICCV, 2023.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. In NeurlPS, 2023.
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video lims with active real-time interaction via disentangled perception, decision, and reaction. In CVPR, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Gabriel A Radvansky, Sabine Krawietz, and Andrea K Tamplin. Walking through doorways causes forgetting: Further explorations. Quarterly journal of experimental psychology, 2011.
Santhosh Kumar Ramakrishnan, Erik Wijmans. Philipp Krachenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models? In ICLR, 2025.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 1999.
Nikhila Ravi, Valentin Gabeur, Yuan-Time LangTing Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. In ICLR, 2025.
Arijit Ray, Jiafei Duan, Ellis Brown, Reuben Tan, Dina Bashkirova, Rose Hendrix, Kiana Ehsani, Aniruddha Kembhavi, Bryan A. Plummer, Ranjay Krishna, Kuo-Hao Zeng, and Kate Saenko. SAT: Spatial Aptitude Training for Multimodal Language Models. In COLM, 2025.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In CVPR, 2024.
Weiming Ren, Wentao Ma, Huan Yang, Cong Wei, Ge Zhang, and Wenhu Chen. Vamba: Understand- ing hour-long videos with hybrid hamba-transformers. In ICCV, 2025.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In ICCV, 2021.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. IJDL, 2022.
Ziyao Shangguan, Chuhan Li, Yuxuan Ding, Yanan Zheng, Yilun Zhao, Tesca Fitzgerald, and Arman Cohan. Tomato: Assessing visual temporal reasoning capabilities multimodal foundation models. In ICLR, 2025.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longyu: Spatiotemporal adaptive compression for long video-language understanding. In ICML, 2025.
Enxin Song. Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, 2024.
Aimee Stahl and Lisa Feigenson. Observing the unexpected enhances infants' learning and exploration. Science, 2015.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. /8805 2023.
16

---

Published as a conference paper at ICLR 2026
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean- GannBaatistt Alayrac, Montser- rat Gonzalez. Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing into the physical world. arXiv preprini wrXiv:2503.20020, 2025.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri Iyer, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-I: 1: Fully Open, Vision-Centric Exploration of Multimodal LLMs. In NeurlPS, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092: 299.09988, 2023b.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprini arXiv:2502. 14786, 2025.
Paul Voigtlaender, Soravit Changpinyo, Jordi Pont- Juste, Radu Soricut, and Vittorio Ferrari. Con- necting vision and language with video localized narratives. In CVPR, 2023.
Hermann Von Helmholtz. Handbbuch der physiologischen Optik. L. Voss, 1867.
Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic, Ibrahim Alabdulmohsin, Xiao Wang, André Susano Pinto, Andreas Steiner, Lucas Beyer, and Xiaohua Zhai. Locca: Visual pretraining with location-aware captioners. NeurlPS, 2024.
Alex Jinpeng Wang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. Cosmo: Contrastive streamlined multimodal model with interleaved pre-training. arXiv preprint :41v::2001000449, 2024a.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In CVPR, 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing llaguage model's perception of the world at any resolution. arXiv preprint arXiv: /2///, 2024b.
Sinong Wang, Belinda 7 Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In ECCV, 2024c.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang. Yansong Shi, et al. Internvide02:: Scaling foundation models for multimodal video understanding. In ECCV, 2024d.
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding? In ICML, 2025.
17

---

Published as a conference paper at ICLR 2026
Cheng-Kuang Wu, Zhi Rui Tam, Chich- Yen Lin, Yun-Nung Vivian Chen, and Hung-yi Lee. Stream- bench: Towards benchmarking continuous improvement of language agents. In NeurlPS, 2024a.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In NeurIPS, 2024b.
XAI. Grok-1.5 Vision Preview, https: : //x. ai ai/blog/grok 1- 5v, April 2024. RealworldQA, Blog post, Announced on April 12, 2024.
Runsen Xu, Weiyao Wang. Hao Tang, Xingyu Chen, Xiaodong Wang, Fu-Jen Chu, Dahua Lin, Matt Feiszli, and Kevin Liang. Multi-spatialmllm: Multi-frame spatial understanding with multi-modal large language models. arXiv preprint arXiv:2505.17015. 2025.
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly. Marcello Maggioni, et al. Gspmd: general and scalable parallelization for ml computation graphs. arXiv preprint 100.::200..04663, 2021.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high- reshluteoo transcriptions. CVPR, 2022.
representation with large-scale video
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report. arXiv preprint 14215:241..15115 2024a.
Dongjee Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. In NeurlPS, 2024b.
Jihan Yang, Runyu Ding, Ellis Brown, Xiaojuan Qi, and Saining Xie. V-IRL: Grounding virtual intelligence in real life. In ECCV, 2024c.
Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. In CVPR. 2024d.
Jingkang Yang. Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. In CVPR, 2025a.
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In ICLR, 2024e.
Yuncong Yang, Jiageng Liu, Zheyuan Zhang, Siyuan Zhou, Reuben Tan, Jianwei Yang. Yilun Du, and Chuang Gan. Mindjourne:: Test-time scaling with world models for spatial reasoning. arXiv preprint arXiv:2507. /2508, 2025b.
Chun-Hsiao Yeh, Chenyu Wang, Shenghang Tong, Ta- Ying Cheng, Ruoyu Wang, Tianzhe Chu, Yuex- iang Zhai, Yubei Chen, Shenghua Gao, and Yi Ma. Seeing from another perspective: Evaluating multi-view understanding in milms. arXiv preprint arXiv:2504.15280 2025.
Chandan Yeshwanth, Yuch- Mheh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In ICCV, 2023.
Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. Spatial mental modeling from limited views. arXiv preprint 2rXiv:2506..1455 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruogi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
18

---

Published as a conference paper at ICLR 2026
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In EMNLP. 2023.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory- real-time understanding for long video streams. In ICCV, 2025a.
Peiyuan Zhang, Kaichen Zhang, Bo Li. Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406./688. 2024.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. TMLR, 2025b.
Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu, Needle in video haystack: scalable synthetic evaluator for video milms. arXiv preprint 2rXx::2406009367. 2024.
Jieyu Zheng and Markus Meister. The unbearable slowness of being: Why do live at 10 bits/s? Neuron, 2025.
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. ICML, 2025.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
Chemming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering Imms with 3d-awareness. ICCV, 2025a.
Jinguo Zhu Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.1047,, 2025b.
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. In CVPR, 2025.
19

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We grateful Cambrian-1 (Tong et al., 2024a) for the excellent codebase, which served the launching point for our research. Thanks to the TorchXLA team for helpful discussions on TPU, TorchXL.A, and JAX distributed training infrastructure. We also thank Anjali Gupta, Sihyun Yu, Oscar Michel, Boyang Zheng, Xichen Pan, Weiyang Jin, and Arijit Ray for reviewing this manuscript and providing constructive feedback. This work was primarily supported by the Google TPU Research Cloud (TRC) program and the Google Cloud Research Credits program (GCP19980904) E.B. is supported by the DoD NDSEG Fellowship Program. S.X. acknowledges support from the MSIT ITTP grant RKS-2024-00457885 and the NSF award IIS- ISS2444400
APPENDIX
This appendix provides comprehensive implementation details, experimental results, and supplemen- tary analyses supporting the main paper:
$ provides a summarization and limitations of our current work and outlines potential directions
for future explorations.
$ B provides a summarization of related work. including recent advancements in video MLLMs, streaming video understanding, visual spatial intelligence, and predictive modeling.
$ C presents detailed diagnostic test results for video MLLM benchmarks under different evalua- tion setups.
$ D describes the VSI-SUPER benchmark, including implementation details, visualizations, and streaming setups for both Recall and Count tasks.
$ E provides comprehensive documentation of the VSI-590K dataset, including question type laxonomy, QA-pair construction pipeline, ablation studies, and qualitative examples.
$ F details the Cambrian-S model architecture, training data mixture, training recipe across all four stages, and infrastructure setup.
$ G presents additional experimental results including detailed evaluation setups, performance on image and video benchmarks across all model scales, ablations on image-video data contributions, and analysis of the trade-off between spatial sensing and general video understanding.
$ H describes predictive sensing components, including latent frame prediction implementation details, memory framework design for VSI-SUPER Recall, agentic framework design for VII- SUPER Count, and comparisons with existing long-video methods.
A DISCUSSION
In this paper, highlight the importance of and propose a hierarchy for spatial supersensing capabilities in videos, arguing that achieving superintelligence requires AI systems to move beyond text-based knowledge and semantic perception, the current focus of most MLLMs, to also develop spatial cognition and predictive world models. To measure progress, we introduce VSI-SUPER and find that current MLLMs struggle with it. To test whether current progress is limited by data, we curate VSI-590K and train our spatially grounded MLLM, Cambrian-S, on it. Although Cambrian-S performs well on standard benchmarks, results on VSI-SUPER reveal the limitations of the current MI.LM paradigm. We prototype predictive sensing, using latent frame prediction and surprise estimation to handle unbounded visual streams. It improves Cambrian-S performance on VSI-SUPER and marks an early step toward spatial supersensing.
Limitations and Future Work. Our goal is to present a conceptual framework that encourages the community to reconsider the importance of developing spatial supersensing. As a long-term research direction, our current benchmark, dataset, and model design remain limited in quality, scale, and generalizability, and the prototype serves only proof of concept. Future work should explore more diverse and embodied scenarios and build stronger connections with recent advances in vision, language, and world modeling.
20

---

Published as a conference paper at ICLR 2026
B RELATED WORK
Video Multimodal Large Language Models The strong linguistic understanding capabilities of pretrained LLMs (Brown et al., 2020; Touvron et al., 2023a; Bai et al., 2023a; Touvron et al., 2023b), combined with the representational power of vision foundation models used as feature extractors (Radford et al., 2021; Zhai et al., 2023; Tschannen et al., 2025; He et al., 2022; Fan et al., 2025), have driven significant advances in extending these models beyond text to achieve semantic perception of visual content, primarily in the image domain (Hurst et al., 2024; Liu et al., 2023; Li et al., 2025a; Bai et al., 2023b; Tong et al., 2024a; Team et al., 2023; Chen al., 2024d: Wang et al., 2024b; Li et al., 2023a). This momentum has spurred growing research into video-based MLLMs (Li et al., 2024f; 2025a; Zhang et al., 2025b; Song et al., 2024; Bai et al., 2025a; Zhu et al., 2025b; Zhang et al., 2023; Li et al., 2023b; Zohar et al., 2025; Marafioti et al., 2025), which are seen as key step toward connecting multimodal intelligence with real-world applications such as embodied agents (Kim et al., 2024; Yang et al., 2024c). As emphasized throughout this paper, developing truly capable supersensing system requires rethinking several core aspects, including how progress is benchmarked, what constitutes the right data, which architectural designs are most effective, and what modeling objectives best align with the system's 's goals.
Streaming Video Understanding Video is a continuous and potentially infinite stream of visual signals. While humans process it effortlessly, its unbounded nature challenges video MILMS because token lengths increase with duration, causing rising computational and storage costs. Recent work has explored several approaches to address this problem: Efficient architectural design. The quadratic cost of self-attention makes it hard to handle long videos. Recent methods (Li et al., 2024c; Ren et al., 2025) use simpler, faster architectures (Wang et al., 2020; Gu Dao, 2024; Katharopoulos et al., 2020) that reduce computation and work better with longer inputs. Context window expansion. The fixed context length in pre-trained LLMs limits their understanding of long-term content. Recent work (Chen et al., 2025e; Zhang al., 2024; Chen et al., 2025b) extends this window by careful system design, enabling models to handle and reason over longer video sequences. Retrieval- augmented video understanding. To process long videos, some approaches retrieve only the most relevant segments from a larger collection (Korbar et al., 2024; Pan et al., 2025; Wang et al., 2024c) and use them as context for further analysis. Visual token reduction or compression. Other methods shorten the input by reducing visual tokens across or within frames (Shen et al., 2025; Li et al., 2024e; Jiang et al., 2025; Li et al., 2025b; Chai et al., 2025), making it easier to handle long video sequences. While these methods improve performance, they largely treat continuous videos as standard sequence modeling problems, similar to text. We believe future MI.LMs should build internal predictive models to efficiently process continuous visual streams, as humans do.
Visual Spatial Intelligence Understanding spatial relationships from visual inputs is crucial for perceiving and interacting with the physical world. As multimodal models become more physically grounded, interest in spatial intelligence has surged, leading to new benchmarks (Yang et al., 2024d; Ramakrishnan et al., 2025; Yin et al., 2025; Majumdar et al., 2024; Yeh et al., 2025; Li et al., 2025c; Xu al., 2025; Team al., 2025) and research focused on enhancing models' spatial reasoning capabilities (Yang et al., 2025b; Ma et al., 2025; Ouyang et al., 2025; Du et al., 2024; Chen et al., 2024a; Cheng et al., 2024; Cai et al., 2025; Liu et al., 2025; Li et al., 2024b; Zhu et al., 2025a; Ray et al., 2025). In this paper, we study visual spatial intelligence through the concept of spatial supersensing in videos and explore ways to strengthen MLLMs' spatial reasoning by refining data curation, optimizing training strategies, and introducing new paradigms.
Predictive Modeling A learned internal predictive model (Craik, 1967; Ha & Schmidhuber, 2018) allows an intelligent agent to represent and simulate aspects of its environment, enabling more effective planning and decision-making. Model predictive control (MPC) (Garcia et al., 1989) applies similar principles in control theory, leveraging internal forward models to anticipate future trajectories and select optimal actions real time. This concept draws inspiration from how humans form mental models of the world (Rao & Ballard, 1999; Hohwy, 2013; Friston, 2010) and how these internal representations influence behavior (e.g.. unconscious inference (Von Helmholtz, 1867)), serving as simplified abstractions of reality that enable prediction and efficient action. A growing body of work has explored the idea of predictive modeling through self-supervised representation learning (Assran et al., 2023; 2025), and text- action-conditioned video generation (Zhou et al.,
21

---

Published as a conference paper at ICLR 2026
2025; Yang et al., 2024e; Bar et al., 2025; Chen et al., 2024b; Bai et al., 2025b; Garrido et al., 2025). In this paper, motivated by how humans leverage internal world models to process unbounded sensory input efficiently and effectively, we explore how to equip MLLMs with similar predictive sensing capability.
C BENCHMARK DIAGNOSTIC TEST RESULTS
C.I DETAILED RESULTS OF CAMBRIAN- 1-M 1-B ON VIDEO BENCHMARKS
Table Detailed results of our improved Cambrian-1-79 on video MILMM benchmarks under different evaluation setups.
SUPER Recall
VSI-SUPER Count
Long' Long VideoBennc
VideoMMMU
Percegtion Test
VidcoMME
SehSeh
MVBench
HourViden
VSI-Bench
Tomato
Evaluation Sctups
Chance-
25.0
20.0
14.0
25.0
22.0
27.3
33.3
20.0
34.0
25.0
0.0
Cambrian- 1-7B (Our upgraded)
Blind Test
31.2
31.9
25.0
42.5
7.8
19.6
40.7
24.3
17.4
20.0
0.0
Single Frame
41.6
44.0
29.0
46.9
15.8
52.1
27.7
20.4
19.7
Multiple (32) Frames
53.7
48.1
31.9
51.4
18.9
55.4
55.6
31.6
25.8
22.7
(32) Frame Captions
55.3
52.4
40.1
52.2
16.8
47.7
55.6
29.5
21.8
9.6
C.2 BENCIMARK EXAMPLES
VideoMME
VSI-Bench
feature the indicates independently spoce?
standing ng refrigerator lucing washer atove my right,
Why the objects Bynn??
many
Figure 12: Illustrations of how spatial sensing is conceptualized in current video benchmarks. The left panel features examples from the "spatial reasoning" subcategory of VideoMME (Fu al., 2025), including a question regarding gravity from Shutter Authority's "What if the Moon Crashed into the Earth?" and question regarding astronaut gear from NASA's "Astronaut Bruce McCandless II Floats Free in Space. contrast, right panel shows samples from VSI-Bench (Yang et al., 2024d), which highlight visual-spatial reasoning tasks such as object counting, identifying relative directions, route planning, and more.
D VSI-SUPER BENCHMARK
D. VSI-SUPER RECALL.
Implementation details. To construct this benchmark, we begin with videos from the VSI-Bench collection (Yang et al., 2024d). Annotators select videos and manually insert unusual object from a curated pool into four distinct frames using 2emiini-2.0-Flssh focusing on placing the objects in plausible locations. For each insertion, the annotators record the object's location and its order of appearance. We then combine these edited clips with randomly sampled unedited videos to produce final videos with lengths of 10, 30, 60, 120, and 240 minutes. For each duration, create 60 videos, each with one corresponding question. We downsample videos to 1 frame per second to ensure the model always the edited frames during inference.
22

---

Published as a conference paper at ICLR 2026
Visualization. We present qualitative examples of edited frames of our VSR video dataset in Fig. 18. The inserted objects appear visually plausible at their locations, which is a direct result of our high-quality annotations.
D.2 VSI-SUPER COUNT
Implementation Details. To build VSI-SUPER Count, we concatenate videos from VSI- Bench (Yang al., 2024d) and sum their object counts create a new ground truth. This process requires two additional normalization steps. First, we unify the object category labels from the different source datasets (i.e., ScanNet (Dai et al., 2017), ScanNet++ (Yeshwanth et al., 2023), and ARKitScenes (Baruch et al., 2021)) )). Second, we address a data bias towards small object quantities by rebalancing the question-ansee -nnweerr pairs to create a more uniform distribution of counts. The final benchmark includes videos with lengths of 10, 30, 60, and 120 minutes, each accompanied by 50 corresponding questions. Different from VSR, all videos VSC are downsampled to 24 FPS.
Streaming setups. For the streaming setup, repeatedly query the total number of objects in video at distinct timestamps. To construct the ground truth at these query timestamps. we need to determine the first appearance time of each unique object in the video. To find these appearance times, we use the method proposed by the VSI-Bench (Yang et al., 2024d). This allows for the direct calculation of the ground truth object count at any given timestamp.
D.3 VISUALIZATION GEMINI-2.5-FLASSH PREDICTIONS ON VSC.
Gamini-2.5--Faae Flash
Perfect Prediction
mins
30 mins
mins
150
250
200
Prediction
Podicioon
100
Predicton
100
50
25
50
75
50
100
150
50
100
150
200
250
Ground Truth
Ground
Ground Truth
Figure 13: Visualization of Gemini-2.5-Flash'ss predictions ground truth VSC. The model's predicted object counts saturate at small constant values and fail to scale with video length or true object counts, indicating limited generalization in counting and reliance training distribution priors.
VSI-590K DATASET
In this section, provide more details for our VSI-590K dataset, including the question type definition, question-answer pair construction pipeline, and some examples for each data source.
E. DETAILS OF QUESTION TYPE DEFINITION
Taxonomy. When curating visual-spatial intelligence supervised fine-tuning datasets, an important perspective is how to define the question type. Inspired by VSI-Bench (Yang et al., 2024d), we expand its task definition in a more systematic manner. As shown Table 5, we distinguish these question types in four perspectives:
attributes: We categorize questions into five distinct spatial- temporal attribute types: size (comparing or measuring object/space dimensions), direction (orientation in space), count (enumeration objects), distance (proximity between objects), and appearance order (temporal sequence of objects appearing in videos).
Soaatilltempar
23

---

Published as a conference paper at ICLR 2026
Relative versus absolute: Questions classified as relative when they involve comparison between multiple objects (e.g. "which is larger?"), or absolute when they require specific measurements quantities (e.g., "what is the height in meters?"). This distinction applies across most attribute types.
Perspective taking: This dimension captures the viewpoint from which spatial relationships are evaluated. Questions may be posed from the camera's perspective (e-g. "from the camera's perspective, is the object on the left or right?") or from the perspective of specific objects in the scene (e.g., "facing the object; from objectz...")
Questions are categorized based on whether they can be answered using static images only, or require dynamic video information. Some attribute types, like appearance order, are only applicable to videos, while others like size can be questioned either modality.
Modality:
Additionally, following VSI-Bench, also categorize question types into three different groups (i.e., Configuration, Measurement, or Spatiotemporal) according to their different spatiotemporal characteristics.
Table 5: Taxonomy of spatiotemporal question types in VSI-590K. Questions are stratified along five axes: attribute type, relative vs. absolute (Rel./Abs),, perspective, modality (V: video, I: image), and group. An example question template is provided for each type.
Types
ReI/Abs.
Perspective
Modality
Group
Example template
Rel.
V &
Configuration
J. which larger?" "What the height the (objerr] in (wnir]?"
"Between [object: | and [objects),
Size
Abs.
Measurement
Abs.
V
V
Measurement
"What the roo's size (umit)?"
Rel.
Camera
Configuration
"From the camera's perspective, is the (object) on the left or the right?"
Direction
Rel.
Object
V
Configuration
"Facing the [object, from the Lobjecty), would the [objecty] be placed left, right, or back?"
Abs.
Object
V
Measurement
"Standing [object: }. facing toward (objects). ). how far clockwise do motate (in degrees) to see the (abjects) )
Rel. Ahs.
V
Configuration
"Are there fewer (object,) | than (objecta] 7"
Count
V&I
Measurement
"How many (abjecz) are present!"
Rel.
Camera
Configuration
"Which object is closer the camera, the (abject_ the [object_2]]"
Rel.
Object
V
Configuration
"Which neater the (object_ 3), the [object_/) []] or the [objecr_]] 2
Distance
Abs.
Object
V&I
Measurement
"What is the distance between the [object_/] /| and the [object_2] [wnit]?
"Determine how (object /) [object_21, [object J), and [object_4 4) are ordered by their initial appearances in the video"
Appr. Order
Spatiotemporal
E.2 DETAILED QA-PAIR CONSTRUCTION PIPELINE
We introduce the concrete pipeline used for curating VSI-590K here.
3D-annotated real videos. For the 3D-annotated real videos, we follow the practice established by Thinking in Space (Yang et al., 2024d). We begin by researching all publicly available datasets containing both 3D instance-level annotations and video or panorama images. From these datasets, we extract key information including object counts, object bounding boxes, and room size measurements, which we then standardize into unified format. Afterward, this structured information is incorporated into augmented question templates to create paired question-answer sets.
3D-annotated simulated videos and images. For simulated data, which inherently contains rich annotations, we followed a procedure similar to that used for 3D-annotated real videos. As for ProcTHOR (Deitke et al., 2022), our primary effort is generating 3D scenes with randomly placed agents to render traverse videos. For Hypersim (Roberts et al., 2021), which provides image-level rather than scene-level 3D annotations, we utilize individual images with their corresponding 3D annotations. In both cases, extract the necessary information, convert it to our designed unified format, and incorporate it into augmented question templates, following the same approach used for 3ID-nnottted real videos.
24

---

Published as a conference paper at ICLR 2026
Unannotated web-crawled real videos. For unannotated web-crawled real videos, shown in Algorithm 1, we implement stage processing pipeline. We begin by sampling frames at regular intervals and filtering out blurry images. For each valid frame, employ the vecabuocyyl object detector Grounding-DDie DDNN (Liu et al., 2024c) with predefined categories interest. When a frame SAM2 (Ravi et al., 2025) to extract instance-wise semantic masks. Besides, to transform 2D image content into 3D representations, we employ VGGT (Wang al., 2025) extract 3D point for each image and integrate them with the previously generated instance masks. Notably, we apply erosion algorithm to refine the instance masks, which mitigates inaccurate point cloud estimations object boundaries. This pipeline has enabled us to create pseudo-annotations from approximately 19,000 room tour videos from YouTube and robotic learning datasets, yielding diverse spatial question-answe pairs across various room types and layouts without manual 3D annotations. By processing individual frames rather than complete videos, our pipeline ensures higher quality semantic extraction and more reliable reconstruction results, avoiding the noise and inconsistent issues typically encountered when applying reconstruction and semantic extraction techniques to entire video sequences.
contains sufficient valid objects,
E.3 DATA STATISTICS
We provide detailed data statistics for VSI-590K in Table and Fig. 14.
Table 6: Data statistics for VSI-590K. We collect data from 10 sources with different video types and annotations to improve diversity.
Dataset
# Videos
# Images
# QA Pairs
Amnotated Real Videos
S3DIS (Armeni al., 2016)
199
5,187 60,207
Aria Digital Twin (Pan al., 2023)
183
ScanNet (Dai al., 2017)
1,201
92,145
ScanNett+ V2 (Yeshwanth
2023)
856
138,701
ARKitScenes (Baruch al., 2021)
2,899
57,816
Simulated Data
ProcTHOR (Deitke al., 2022)
625
20,092
Hypersim (Roberts et al., 2021)
5,113
176.774
Unannotated Real Videos
You YouTube Room Tour
20,100
20,100
Open X- E-boodiment (O" Neill et al., 2024)
14,801
14,801
AgiBot World (Bu et al., 2025)
4,844
4.844
Total
5,963
44,858
590,667
Direction (43 m0)
-Mboouue Desccion Object (2250
(heles Devetien Camero and
beayiie Dinaliee Diipe (R oig
"belase Oestrees Ubeer
Abbouut Desunce Obeee (0
Seadiv Siae Oipest (8A5)
Seachs Direcioo Camara
Sie h6 vi)
VSL-S90K
Seorive Couet 2181g
-haar Objer (Lied
Mexsa smon Os-00
Ooject (00000
-Abeakns Sis (-H) Aod
Abcckr Bisckk
Abvekiee Too Ciject 3.00)
-Mbooun Coure Nued
meeles Court (225)
Spadoteemene (27x))
Aspourence 12:00
eacanca (1750
Figure 14: VSI-590K statistics. QAs are grouped by: question types (left) and task groups (right).
E.4 ADDITIONAL ABLATION STUDY
Table 7 ablates the contributions of different data sources VSI-590K. Table presents an ablation study how different task groups affect the model's spatial sensing capability. Our results show that all three task groups-oee confinuratn.... measurement, and spatiotemporal are integral, as removing any one of them degrades performance. We further assess spatial reasoning using the held-out Route
25

---

Published as a conference paper at ICLR 2026
Table Contributions of Different Data Sources in the VSI-590K Mixture. This table illustrates the impact of different data sources on VSI-Bench performance. The combined dataset, VSI-590K Full Mix, achieves the best overall results. Among individual sources, annotated real video datasets contribute the most significant improvements, followed by simulated videos, and then pseudo- annotated images.
Image
VSI-Bench (Video)
MMVP
3DSR
CV-B
Obj Ct
Abs Dst
Obj Sz
Rm Sz
Red DM
Dir
Ric Pin
Ap Ord
Avg
Baseline
52.7
54.5
73.5
28.5
18.1
20.0
36.0
22.2
42.9
31.3
24.6
33.0
Real Videos
S3DIS
54.0
54.9
75.3
41.6
63.8
21.0
44.9
37.0
43.8
47.4
34.0
41.1
ADT
50.6
56.5
77.5
41.0
51.0
29.8
52.5
40.2
42.3
38.8
34.0
39.8
ARKiiScenes
50.0
77.3
51.0
32.7
55.1
43.5
ScanNet
54.7
57.7
77.5
56.3
70.9
37,9
67.5
59.3
57.0
46.7
35.1
76.1
ScanNett+ V2
52.7
57.3
77.5
56.3
72.5
40.7
65.7
56.9
59.7
47.1
31.4
76.2
Simulated Videos
ProcThor
53.3
55.7
74.9
36.4
21.0
29.7
49.3
3.8
52.3
45.7
30.4
58.7
HyperSim
52.0
56.0
79.7
45.6
67.8
32.0
59.3
36.4
53.2
47.0
32.5
36.6
Preudo-Annotated Images
YTB Room Tour
55.3
52.6
75.0
32.5
43.4
25.8
24.2
27.3
38.7
31.4
28.4
40.9
OXE & AGIBot
56.0
54.4
72.5
30.6
40.3
23.1
27.9
26.6
38.0
22.8
32.0
33.8
Full Mix
54.7
54.0
77.9
63.2
73.5
49.4
71.4
70.1
66.9
61.5
36.6
76.6
Table 8: Ablation study on VSI-590K task groups. We study models' performance change when one certain task group are omitted from the training data.
VSI-Bench
Obj Cr
Abs Det
Obj S2
Rm Sz
Rel Dst
Rel Dir
Rie Pin
Ord
Avg
VSI-590K Mixture
All
63.2
73.5
49.4
714
70.1
66.9
61.5
36.6
76.4
Configuration
51.9
46.2
43.0
70.4
66.0
48.0
36.8
27.3
77.3
w/o.
Measurement
49.7
74.5
19.1
38.5
63.9
55.6
35.1
79.5
Spatiotemporal
73.7
65.2
32.5
47.6
Plan subtask and find that the configuration group is the most influential, whereas the measurement group is the least. We attribute this outcome to the fact that route planning requires holistic understanding of the spatial layout, which is more explicitly provided by configuration QA pairs compared to measurement and spatiotemporal tasks.
E.5 EXAMPLES OF VSI-590K
To better illustrate VSI-590K, provide qualitative visualization results in Figs. 19 25, These visualizations demonstrate that VSI-590K delivers great diversity and quality for spatial question- answering supervised fine-tuning.
F CAMBRIAN-S IMPLEMENTATION DETAILS
In this section, we provide holistic training details of our Cambrian-S models.
F.1 MODEL ARCHITECTURE
Following the original Cambrian-1 (Tong et al., 2024a) and common practices in most MILMs (Liu et al., 2023; Li et al., 2025a), our model (both our upgraded Cambrian-1 and Cambrian-S) integrates pre-trained vision encoder, pre-trained language model as the decoder, and vision-language connector to bridge these two modalities. Specifically, we employ SigLIP2-So4000 So400M (Tschannen et al., 2025) the vision encoder. This encoder trained using combination of losses: text soatttoke--prdicttn (LocCa (Wan et al., 2024)), image-text contrastive (or sigmoid (Radford et al., 2021; Zhai et al., 2023)), and masked self- seff-prediction (SIL.C (Naeem et al., 2024)/TIPS (Maninis
26

---

Published as a conference paper at ICLR 2026
Algorithm 1: QA generation pipeline for unannotated web-scrawled videos
Input:
Video sequence V, valid category list Canaa invalid category list Cimalid, sampling interval At, blur threshold Taker minimum object count minn minimum point count (sD, erosion kernel Karsson
Output: Selected frame set Question-anssen -antorr pairs @
Initialize + 0, Q + 0:
S SamplcFranes(V. A foreach frame S do BlurDetection f) > Tolur then
// sample frames at interval At
L continue;
O GrroundingDNNO(f,co Cuatia Cimania;; 1ists
1/ Detect objects from both category
30 E O cateegrry()) E Cimatu then
continue:
Oralid + 10 E O category(o) C:ali);;
10
O < Omin then
11
L continue:
M+ 0;
1/ Initialize mask set
13
foreach object o Onlid do
14
b + GerBoundingBox());
<- SAM2(1 b);
// Generate mask using SAM2 // Apply erosion on the masks
16
m + Erode (m, Karssa));
M .M {m'}:
18
Pmop +- VGGT();
11 Generate 3D point map using VGGT // Initialize 3D point set
19
P + 0;
foreach mask
M de t ExtractMaskeedoiitssmy, Pamp):
21
Extract 3D points covered by mask
22
if Pmidd 2 0:D then
23
(P(P):
24
[P] > then - QAGenerator(P): Q - Q [q];
25
// Generate QA pairs from 3D geometry
26
27
F F (}]
28
Return F, Q;
al., 2024)). For the language model, we utilize the instruction-tuned tuundddntttddddd Qwen2.5 LLMs (Yang et al., 2024a). Unlike Cambrian-1, which used SVA for a deeper vision-language fusion, we employ simpler GELU-activated (Dauphin et al., 2017) two-layer MLP as the vision-language connector to maintain balance between performance and efficiency.
F.2 TRAINING DATA MIXTURE
As mentioned in Section 3.4, our Cambrian-S models are trained with four training stages (See Fig. 6). For the first stages (i.e., vision-language alignment stage and image instruction tuning stage), refer readers to Cambrian-1 (Tong et al., 2024a) for the detailed training data mixture. In the third stage, we finetune the image instruction-tunel models Cambrian-S-333 we conduct spatial video instruction tuning by finetuning the model on VSI-590K. Cambrian-S- our curated video instruction tuning dataset with around 3M video QA samples, built upon a set of open-sourced video datasets (e.g., ALLVA-Viden (Zhang et al., 2025b), ShareGPT40 (Cui et al., 2024), VideoChat2 (Li et al., 2024d), MovieChat (Song et al., 2024), EgolT (Yang et al., 2025a), Perception Test (Patraucean al., 2023), Vript (Yang et al., 2024b), VideoChatGPT-Plus (Maaz et al., 2024), Ego4D (Grauman et al., 2022), HowTolooo (Miech et al., 2019), VILA (Xue et al., 2022), (Wang et al., 2024a), GUI-World (Chen et al., 2025a), Video-Localized - (Voigtlaender et al., 2023), and etc.).
and during the last stage,
HTStep (Afouras et al., 2023), TimeIT (Ren et al., 2024),
We detail its composition in Fig. 15.
27

---

Published as a conference paper at ICLR 2026
Gemeral Video QA/ Caption (B1.3%)
WWeV decGA (0.00)
LLaVA--Videe- CTK (44.20)
PeronotionTest (3
Egoconeric QA (3.2%)
-LaVi--ioand (R.40)
YeuCCocc (0.75]
-poo-r-onx (2085)
Stanedemri O
FAVO (UUN)
-EpuQA 00.80
TVGA (4.0%)
ShamGPP (0.16)
(X79)
Vi pe (xoN)
CokeeChur 10 166
CS3M
NexT- BoxTT-QA11 (3.06)
Fine- grained Video Q2A/ Caption (7.75)
Video Ressoning
NTURGRD 0.900
Aethdty (U100
TDI (2.8%)
ActhityNee (LENO
STAR (1.28)
LSHD (25kg
- HTTW (OLPs)
-octvityae--il (9.29)
Viddeo locelmed Narrative (1.k))
Didemo (2.20)
Timall (2,400
Nee-whyhe-FF 8T1, (1.53
Text Recognition (1.3%)
Video Classifiction (0.7%)
-CLEVRIR 12,%)
TextVit n.580
-000 (0 200
Figure 15: General video instruction tuning datasets of Cambrian-S-3M, used in Cambrian-S stage 3 & 4 training.
Table 9: Training configuration for stage and stage 2.
Stage (Vision-Language Alignment)
Stage 2 (Image Instruction Tuning)
Vision Eaceder
SigLIP2-So-00MM
Langunge Decoder
Qwen2.5-0.5B. 1.5B, Instrucc
VL-Connector
2xMLP-GELU
Recipe
Data
Cambrian-Aigamenne Aligmmen-225M 338) 729
Cambrian-7M
Image Resolution
AnyRes (L'p to sub- (o-binaaes)
# of Tokens per Image
Up to 7,290
Training Recipe
Max Sequence I Length
2,048
8,192 x 10-5
Trainbbe Module
VL-Connector 1 x 10-3
1 Learning Rate
Batch Size
Warmup Ratio
0.06
0.03
F.3 TRAINING RECIPE
Stage Vision-language alignment. We freeze most of the model's parameters and train only the vision-language connector on the Aambrian-Allgnmeen-2222 images are padded to a fixed resolution of 384 384, and the maximum sequence length set to 2048.
dataset (Tong et al., 2024a). Input
Stage 2: Image instruction tuning. We unfreeze both the vision-language connector and the LLM decoder, while keeping the vision encoder frozen. The model is then fine-tuned the Cambrian-7M image instruction tuning dataset. Compared to Cambrian-1 (Tong et al., 2024a), we adopt the AnyRes strategy (Liu al., 2024a) enhance the model's image understanding capabilities. Specifically, input images are resized while preserving aspect ratio, then divided into multiple 384 384 sub- images. This enables the model to handle images with higher and more flexible resolutions. To accommodate the increased number of visual tokens introduced by the AnyRes strategy, we extend the sequence length to 8192. Detailed training configurations for stage and 2 are provided in Table 9.
Table 10: Training recipe for Cambrian-S stage and stage 4.
Stage Video Instruction Tuming)
Stage (Sputial
Tuning)
Model
Vision Fnooder
Sigl IP2-S5400M Qwen2.54..B 1.SB, 3B, 7B-Instmer 2xMLP-GELU
Language Decoder
VL-Connector
Kecipe
Dain Souree
Cumbhran-S-3M Pad (384x(384)
590K general Video data (sampled Carsbvian-SSSM
Video Frame Resolution
Pad (384x
Frame Sampling Strategy
Unifoom
Unitarm
e Frames per Video
64
128
Tokens per Video Frame
Training Recipe
Max Sequence Length
$,192
16,384
Traitable Modules
Vi--Conector LLM
Leuriing Rate
1 10-3
Global Butch Size
256
Warmup
28

---

Published as a conference paper at ICLR 2026
Stage 3: General video instruction tuning. To equip the model with general video understanding capabilities, we perform video instruction tuning on a mixture of curated Cambrian-S 3-3M video data and sampled image instruction data from Cambrian-7M: As in previous stages, the vision encoder remains frozen, and the remaining modules are fine-tuned. For image data, we reuse the sampling strategy from stage 2. For video data, we uniformly sample 64 frames per video, resize them to 384 x 384, and further downsample their feature maps to 8 8, i.e., 64 tokens per frame.
Stage 4: Spatial video instruction tuning. The final stage focuses on enhancing the model's spatial reasoning capabilitiess by fine-tuning on our proposed VSI-590K. To preserve general video and image understanding, we mixed 590K video samples from Cambrian-S-3M and 120K image samples from Cambrian- 7M. Training settings are mostly consistent with stage 3, except for two key changes: (1) we increase the number of frames per video to 128, and (2) we extend the sequence length to 16,384, both to support richer temporal modeling. Detailed configurations for stage 3 and 4 are listed in Table 10.
F.4 INFRASTRUCTURE
All models in this paper trained using TPU v4 Pods with the TorchXLA framework. To support large-scale video instruction tuning-wherr long sequence lengths introduce prohibitive computa- tional and memory costs we leverage GSPMD (Xu et al., 2021) and FlashAttention (Dao et al., 2022) implemented by Pallas.
GSPMD is automatic parallelization system designed for flexible and user-friende user-friendly large-scald distributed training. It allows users to write training code as if for a single device, and then scale effortlessly across hundreds of devices with minimal changes. Our training framework is based TorchXLA and GSPMD to shard data, model parameters, activations, and optimizer states across multiple devices. This reduces the peak memory usage and improves training throughput.
To accommodate long sequences, we integrate FlashAttention backed by Pallas, which significantly reduces TPU HBM (V-Mem) usage under long-context inputs. This enables to scale the input sequence length up to 16,384 tokens for the 7B model on TPU v4-512 Pod.
G CAMBRIAN-S ADDITIONAL RESULTS
G.1 DETAILED EVALUATION SETUPS
We describe the evaluation settings used for most image and video benchmarks, excluding VSI- SUPER. For image inputs, following the any-resolution design adopted in our training pipeline, each image is resized while preserving its aspect ratio, and its resolution is maximized that it can be partitioned into at most nine 384 384 sub-images. For video inputs, we apply uniform frame sampling with a fixed number of frames. Specifically, checkpoints from stage and stage are evaluated with 32 uniformly sampled frames, while those from stage 3 and stage 4 use 64 and 128 frames, respectively.
G.2 POST-TRAINING RECIPE FOR SPATIAL SENSING
Table 11 studies the effectiveness of different post-training strategies. We find that stronger base models helps to achieve better SFT performance on spatial sensing tasks after finetuning on VSI-590K.
G.3 VSI-BENCH SUB-TASK BREAKDOWN
Table 12 provides detailed breakdown of Gemini, GPT, and our Cambrian-S models' performance on VSI-Bench.
G.4 DETAILED PERFORMANCE ON IMAGE AND VIDEO BENCHMARKS
Table 13 and Table 14 detail the performance of all our checkpoints (from stage to stage 4 and from 0.5B to 7B) on image-basee based and video-based MLLM benchmarks, respectively. For image bench- marks, report the results MME (Yin al., 2024), MMBench (Liu et al., 2024d), SeedBench (Li
29

---

Published as a conference paper at ICLR 2026
Table 11: Post-training exploration for spatial sensing. We examine four base models with progressively increasing exposure to visual data, from image-only training to extensive video training, and analyze their distinct trends during spatial sensing tuning under two different data recipes. Al: only the connector is trained for image-language alignment; A2: Al w/. Cambrian-7M image instruction-tuning data; A3: A2 further finetuned on 429K video instruction-tuning samples; A4: A2 further finetuned on 3M video instruction-tuning samples. From A1 to A4, the models show monotonic improvement in video understanding ability, I-IT and V-IT denote instruction finetuning on image and video data, respectively. Finally, show that stronger base models yield better SFT performance on spatial sensing tasks.
Model
VSI-Bench
VideoMME
EgoSchema
Perception Test
Different Models
(wio, L-FT, Le. QueaL.M)
21.4
44.2
42.9
44.5
(A1 in. Cambrian-1)
25.8
53.7
48.1
55.4
A3 (A2 V-II, 129K dat)
28.9
61.2
50.3
66.3
A4 (A2 V.r. 2M data)
35.7
62.6
77.0
70.9
SFT w/. VSI-590K
from Al
S7.2
40.3
38.7
52.3
from A2
66.8
46.7
47.2
52.3
from A3
68.8
52.3
48.4
55.8
from A4
69.2
54.1
55.2
59.2
SFT w/. VSI-590K & general V-IT data mixture
from A1
61.3
60.5
52.8
65.0
from A2
63.2
62.6
52.9
65.6
from A3
64.0
61.0
54.9
66.8
from A4
65.1
61.9
77.3
71.2
Table 12: VSI-Bench sub-task breakdown. Best results are bolded. Notably, even without any route planning data in training, Cambrian-S-7B outperforms Gemini-1.5-Prr this task.
Obj. Count
Ahs. Dist.
Obj Size
Rocm Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Methods
Avg.
Numerical Ans
Multiple Chhoiie Answer
Statisties
Chance Level (Random)
25.0
36.1
28.3
25.0 25.2
Chance Level (Frequency)
34.0
62.1
32.0
29.9
33.1
25.1
47.9
28.4
Proprictory Models (API)
GPT-4o
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-1.5 1.5 Flash
42.1
49.8
30.8
53.5
54.4
37.7
41.0
31.5
37.8
Gemini-1 1.5 Pro
45.4
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
Gemini
51.5
42.8
61.1
47.8
45.9
71.3
Open source Models
Cambrian-SS S-77
67.5
73.2
50.5
74,9
72.2
71.1
76.2
41.8
80.1
Cambrian
57.3
64.8
78.8
Cambrian-S S-B55B
54.8
68.4
40.0
61.5
50.1
62.4
48.9
29.9
77.5
Cambriun-5-0.55
50.6
67.9
35.4
52.2
52.5
52.3
46.5
25.8
72.2
et al., 2024a), GQA (Hudson Manning, 2019), ScienceQA (Saikh et al., 2022), MMMU (Yue al., 2024), Math Vista (L.u al., 2024), AI2D (Kembhavi al., 2016), ChartQ (Masry al., 2022), OCRBench (Liu et al., 2024e), TextVQA (Zhou et al., 2018), DocVQA (Mathew et al., 2021), MMVP (Tong et al., 2024b), RealworldQA (xAI, 2024), and CVBench (Tong al., 2024a), following Cambrian-1 I's grouping strategy.
G.5 CONTRIBUTIONS FROM IMAGE-BASER AND VIDEO-BASED INSTRUCTION TUNING
To elaborate on the respective contributions of image-based and video-based instruction tuning to model's 's final video understanding capabilities, conducted series of experiments. These experiments employed varying proportions of image and video data during the finetuning stages, and observed the resulting performance trends across diverse video benchmarks.
More specifically, for the initial image MLLM training, we randomly sampled 1M, 4M, and 7M (QA) pairs from Cambrian- to train distinct models. Subsequently, for video-specific finetuning, we randomly sampled 25%, 50%, 75%, and 100% of video QA pairs from LLAVA-Video. 178K (~~ 1-1.6M data samples in total) perform video-only finetuning each of these
image
30

---

Published as a conference paper at ICLR 2026
Table 13: Detailed results of Cambrian-S checkpoints on image MLLM benchmarks.
Genoral
Kanwlegg
OCW Chan
Vosion- Centic
Moth Wanh
CV-Bondh
MMMMU
CharQA
TonVOA
DieVQA
Avg
MMP
SMHB
GIO4
AVS
Avg
MMVP
Motiad
Models
Min--Gamiss---58
72.7
1606.0
72.7
64.5
62.9
99.1
18.7
LLaVA-NeXT-B
72.5
1603.7
72.1
72.7
65.2
55.5
72,8
41.7
36.3
71.6
63.9
69.5
49.0
64.8
72.6
56.6
36.7
90.1
02.2
65.3
CombrianS-77 78
Singe 1
11.5
209
29.6
5.6
0.1
2.5
3.1
2.2
2.9
7.4
0.9
27.6
0.9
0.1
0.9
0.0
2.7
0.8
0.0
Singe 2
74.9
1804
79.0
76.3
66-00
63.9
83.7
48.
45.3
77.
79
78.9
67.66
79.2
90.6
66.3
53.3
57.7
20.0
74.0
Singe 3
74.4
1588.9
79.7
76-4
662.4
601.4
82.2
46.2
36.1
77.0
35.5
75.3
64.01
77.1
$5.6
67.0
58.0
56.1
71.8
72.3
Sugge 4
74.8
1998.4
$0.4
77.0
61.88
64.5
K2.7
48.0
50.6
76.9
75.2
74.7
64.8
76.6
$4.5
70.5
60.0
54.8
74.5
83.0
Cumbrian--33
Stage
0.1
0.0
1524.6
70.7
1495.7
76.5
46.0
72.4
1485.6
60.8
50.0
76.3
Cambeian 1.59
Stage
11.2
282.1
28.6
3.3
3.8
4.2
1.4
2.6
7.9
1.0
21.8
1.4
1.5
0,0
0.0
0.0
Stage 2
63.5
1417.
71.3
71.2
60.6
50.9
15.5
41.1
20.5
06.1
68.0
54.8
59.9
68.3
78.6
19.3
59.7
60.3
58.3
Stago
63.1
1425-3
70.5
72.1
58.7
52.5
40.8
32.3
64.8
64.2
59,5
66
72.9
40.0
59.9
57.8
Singo
63.0
1984
70.1
58.7
54.7
72.3
42.0
19.7
64.1
63.1
58.0
66.6
74.8
59
43.
$4.5
76.3
Camleimns500558
Stage
10.1
379.6
10.7
9.0
1.8
6.2
8.9
3,0
0.2
7.9
2.0
1.9
0.7
10.6
20.1
12.3
Stago
57.7
1124.3
56.6
61.7
38.5
31.0
10.5
56.0
51.1
58.7
63.1
41.2
23.3
51,8
45.6
46.1
Stago 3
59.6
13000
55.8
63.5
55.3
41.2
62.7
92.6
18.0
$1.4
52
46.5
46.8
56.0
59.
45.5
32.0
52.8
52.2
54.9
Stago:
60.0
8
60.7
66.4
53.5
44.0
63.4
34.0
78.6
90.1
526
48.0
47.1
56.6
58.6
48.7
360
51.1
51.6
66.2
Table 14: Detailed results of Cambrian-S checkpoints on video MILM benchmarks.
VSI-Bendh
Valey
EgoSchema
Video MNNM
Long ViBeanch
MVBench
Percept Test
Tomato
Hour
Model
Base I.I.M
Cambrian-S-7B
Stage
21.4
21.0
27.5
44.3
42,9
11.3
32.3
43.9
44.4
Stage
Qwen2.5-7B
24.6
20.1
31.3
52.3
47.5
28.1
51.1
49.2
53.5
Stage
35.7
30,3
38.9
62.8
76.9
38.3
56.7
66.3
70.8
Stago
67.5
27.9
63.3
76.3
59.4
64.8
Cambrian-S-38
Stago
0.7
16.5
15.9
19.5
23.8
30.6
18.6
Stage
Qwen2.5-3B
22.3
21.3
31.7
49.4
42.2
26.0
48.7
44.5
47.0
Stago
23.3
26.3
35.9
58.9
73.4
27.1
52.0
61.0
Stage
57.3
26.0
36.8
60.1
73.6
26.3
$2.3
60.2
65.9
Cambrian
Stage
21.1
23.5
26.2
40.1
33.0
18.7
38.5
40.8
45.2
Stage
Qwen2.5-1.58
22.6
24.6
34.4
47.8
38.2
46.9
45,3
Stage
23.4
23.1
33.2
56.1
67.8
28.6
49.4
58.2
63.6
Stago
54.8
22.2
56.4
69.0
25.0
50.2
57.1
63.2
Cambrian-S-0.18
Stage
23.6
21.5
25.0
34.3
Stage
Qwen2.5-0,58
19.6
20.0
27.9
37.4
29.7
17.3
39.0
40,2
46.3
Stage
23,9
41.8
63.8
50.7
Stage
50.4
23.8
28.1
44.0
62.4
15.9
43.8
51.8
56.0
pretrained image MLLMs. The hyperparameters for image instruction tuning and video finetuning maintained detailed in Table 9 and Table 10, respectively. The experimental results, presented in Table 15, yield the following observations:
Models trained with more image data do not inherently outperform those trained with less when evaluated on video benchmarks without finetuning, As indicated in the table, direct evaluation on video benchmarks reveals comparable performance across all three models, which were initially trained on 1M, 4M, and 7M image datasets, respectively.
Finetuning on video data can be generally beneficial for models pretrained with larger image datasets, though universally. When all models were finetuned on 100% video data, the model initially trained on 7M images outperformed the other two on 5 out video benchmarks (specifically, Hour Video, VideoMME, EgoSchema, Long VideoBench. and Perception Test).
Incorporating video data into the training process consistently benefits performance across all video benchmarks. We observed that finetuning an image-based MILM with video
31

---

Published as a conference paper at ICLR 2026
Table 15: Video MLLM performance trained with different proportions of image and video data.
A/ St--Bench
Hour Vider
EhgoSchema
VidcoMMMi MMMI
Long VBench
MVHench
Percept. Test
Tomato
Video
Image
Video data
Charce Level
34.0
22.0
20.0
25.0
20.0
14.0
25.0
27.3
33.3
0%
26.0
20.2
32.5
52.1
46.9
32.0
51.4
50.5
54.2
25%
32.4
36.2
60.4
47.0
40.1
53.5
57.0
61.9
IM
50%
33.3
27.2
36.2
61.7
47.1
40.1
53.2
59.2
64.3
75%
32.7
28.8
34.4
60.7
48.7
37.7
53.3
59.5
66.3
100%
34.4
35.1
61.3
48.9
39.6
53.0
60.1
67.5
0%
26.7
20.5
31.8
53.1
44.8
32.0
52.1
51.5
54,9
25%
32.3
26.7
37.0
61.3
45.0
38.6
53.1
57.6
61.9
4M
50%
31.9
27.4
37.2
61.9
45.7
38.1
54,2
59.5
65.2
75%
33.8
27.9
36.2
61.1
47.3
40.9
53.1
60.1
67.0
100%
33.8
28.0
35.5
60.5
50.2
40.2
52.2
60.5
67.7
0%%
25.8
18.9
31.6
53.7
48.1
31.9
52.5
$1.4
55.4
25%
31.5
24.6
36.7
61.3
48.8
37.7
54.7
58.3
62.3
7M
50%
31.4
36.6
61.0
37.9
53.6
65.6
75%
31.8
27.0
35.7
618
50.7
38.0
53.0
60.2
67.9
100%
32.6
27.7
37.3
62.1
52.4
39.4
54.3
60.6
68.8
data, even small portion such as 25%, improved its performance on all evaluated video benchmarks.
Increasing the amount of video data used for finetuning does not guarantee consistent performance improvements across all benchmarks. While video finetuning is generally advantageous, some benchmarks (e.g., VideoMME, VSI-Bench, Tomato) do not show further gains with more video data. For instance, models finetuned with 100% video data exhibited performance
par with those finetuned with only 25% video data on the VideoMME benchmark. Only EgoSchema, MVBench, and Perception Test demonstrated consistent benefits from increased video data, a phenomenon we hypothesize is related to the underlying video distribution the training videos.
G.6 ON THE TRADE-OFF OFF BETWEEN SPATIAL SENSING AND GENERAL VIDEO UNDERSTANDING
67.5
VidaoMME Part, (4)
60
EgoSchema Pert. 050
70
85
62.5
ySi-Bench
60.0
50
46
55.0
VSI-Only
40
40
Mixed
0.5
1.5
3.0
7.0
0.5
1.5
3.0
7/0
0.5
1.5
3.0
7.0
Model (B)
Model
Figure On the trade-off between spatial-sensing and general video understanding.
In Section 3.3, we compare model performance when fine-tuned either on VSI-590K alone or on mixture of VSI-590K and general video data. We observe that fine-tuning on VSI-590K alone consistently yields higher performance on spatial sensing tasks, whereas mixed-data fine-tuning offers better balance between spatial sensing and general video understanding. To further explore this trade-off across model scales, we conduct fine-tuning after stage 3 using either VSI-590K alone or the mixed dataset, under four different model sizes: 0.5B, 1B, 3B, and 7B parameters. We then evaluate these models on both general video understanding and spatial sensing benchmarks, as shown in Fig. 16.
32

---

Published as a conference paper at ICLR 2026
The results confirm that the previous conclusion holds across all scales: VSI-590K-onl fine-tuning excels at spatial sensing, while mixed-data fine-tuning provides a better overall balance. Notably, however, the performance gap VSI-Bench narrows model size increases. We attribute this to the greater capacity of larger models to learn and retain diverse capabilities. This trend suggests that scaling to even larger models may further mitigate the spatial sensing performance drop typically observed when fine-tuning with mixed data.
H PREDICTIVE SENSING
H.1 LATENT FRAME PREDICTION IMPLEMENTATION DETAILS
Latent frame prediction head. As shown Algorithm 2, our next-frame prediction head is simple two-layer MLP with GELU activation (Hendrycks, 2016), running in parallel with the MLLM's original language model head. The output dimension set to 1152, matching the output dimension of our vision encoder (i.e., siglisggggspeeee 00tt--00tnn-aaahhh1-----------
Algorithm 2: Latent frame prediction (IFPP) head architecture (in PyTorch style).
LEPilead
Sequential
(0): Lineaz (in, featufrfeetee
out_featture--556 blas-True)
(1): GELU
(2): Linear (ir_features=3384, out_features=1112,, bias=True)
On the balance between LFP and instruction tuning losses. As mentioned in Section I, to build the model's internal world model, we slightly modify our stage introducing two auxiliary losses (i.e, cosine distance and mean-squared error) to optimize the next frame prediction objectiveness. A coefficient is applied to balance the LFP loss against the instruction tuning loss, which we ablate in Table 16.
Table Evaluation results across different benchmarks with varying LFP loss weights. Our default setup (0.1 loss coefficient) is highlighted in gray
LFP loss coeffcient
VSI-Bench
VideoMME
EgoSchema
Perception Test
0.0 (i.e., No LFP Loss)
67.5
63.4
76.8
69.9
0.1
66.1
63.9
76.9
69.7
0.5
60.8
63.6
77.2
66.4
1.0
56.6
61.0
72.9
65.1
H.2
MEMORY FRAMEWORK DESIGN FOR VSI-SUPER RECALL
As introduced in main paper (and shown in Algorithm 3), our predictive memory mechanism comprises three distinct memory levels (M,, Mj, M.r) and four key transition functions governing their interaction: Sensory Streaming, Memory Compression, Memory Consolidation, and Retrieval. This section details the implementation of these functions.
Basic memory units. For our implementation, we utilize the encoded key-value pairs from each Large Language Model (LLM) layer as the basic memory units. This choice, rather than using output latent features from a vision encoder or vision- language connector. allows us to fully leverage the LLM's internal capabilities for memory construction without requiring external modules. This design decision will be elaborated upon in subsequent sections.
Streaming sensing. Each incoming frame is initially processed independently by the vision encoder and the vision-language connector with window size of W.. Subsequently, it further encoded by
33

---

Published as a conference paper at ICLR 2026
the LLM, referencing selected previous frames. The key-value pairs from these preceding frames, cached in the Sensory memory buffer (M,). provide the necessary context for this encoding step.
Surprise-based memory compression. In the meantime of encoding a single frame, we assess its "surprise" level. This is achieved by calculating the difference between the model's s prediction for the current frame and the actual ground truth observation (both in the latent feature space). When frame of timestamp is moved from the sensory memory buffer M, to the long-tern memory Mt, if it is deemed non-surprising (i.e. its surprise score is below a predefined threshold T,), we will downsample its' key-value pairs by factor of 2 along the spatial (H W) dimension. This surprise-based compression mitigates redundancy in the information stored within Mt.
Surprise-based memory consolidation. Long-term memory Mj is initialized with a predefined budget size Biong (e.g., 32,768 tokens). When the volume of memory tokens surpasses this budget, we apply a surprise-based consolidation function to Mi to ensure it remains within the allocated limit. Our consolidation function is straightforward yet effective: we identify the surprise score associated with each frame in Mj. Then, the frame with the lowest surprise score is removed (or "forgotten"). Then, we merge or drop some of these frames according to their surprise scores (we tried three different strategies here: 1. forget the oldest memory, 2. forget the least surprise memory. and 3. forget the least surprise memory while merging adjacent surprise memories if any adjacent surprise memories exist). This process is iterated until the total size of Mj falls below the budget.
Retrieval. Upon receiving a user query q. we first retrieve the most relevant frames from the long-term memory (M,) construct the working memory (M,,). This Mn, then serves as the context for answering the user's query. To perform this retrieval efficiently without resorting to external modules, we utilize the inherent similarity measurement capabilities of the LLM'' attention mechanism. Specifically, for each transformer layer, the user query q is transformed into the attention mechanism's 's query feature space. We then compute the similarity between this query feature and the key features of each frame stored in M, Similarity is measured using cosine distance, and for simplicity, multi-head features are treated as a single feature. The frames with the highest similarity scores have their key-value pairs selected and utilized by the attention mechanism to further encode the user query.
Algorithm 3: Memory framework design for VSI-SUPER Recall.
Input: Frames (J1,.... Jr). User query q
Input: Encoder ε, Decoder D. Surprise Estimator S, Surprise threshold T
Input:
Compression function C, Consolidation function G. Retrieval function R
Input:
Sensory memory M, +- 0 with budget B., Long- term memory Mi + 0 with budget Bi, Working
memory M... for 1 do
2 + E(f,
M. + MM++M {
// Streaming sensing
< L-S(,, M.):
// Surprise estimation
while [.,] > B. do
Dequeue zua from Ma:
m + 1[s 2 7] Zald + 1:st < (((ca))))
// Selective compression
Mi + Mi U }m}
if [M > B, then
L Mi G(M))
// Memory consolidation
I1 Me Mi);
// Retrieve working memory
12 à + Mw);
1/ Answering query with M.
13 return à
H.3 AGENTIC FRAMEWORK DESIGN FOR VSI-SUPER COUNT
Algorithm 4 presents our agentic framework for the VSI-SUPER Count task. Similar to the memory design in Algorithm 3, encode sensory frames using sliding window approach with window size of W,, The latent frame prediction module continuously estimates the expected next frame and computes the prediction error quantify how "surprise" the actual next frame is. As frame arrivs,
34

---

Published as a conference paper at ICLR 2026
the oldest frames that exceed the sensory memory window are dequeued and stored in the long-teme termtterm memory. If dequeued frame is deemed "surprising" (i... its prediction error exceeds a predefined threshold 7), which may indicate scene spatial boundary, accumulated long-term memory and reset it afterward. The generated response is then stored the answer memory bank. The final answer is computed the aggregation of all intermediate answers stored in this bank.
trigger a query response using the
Algorithm 4: Agentic framework design for VSI-SUPER Count task.
Input: Frames "f.....,fr' fr user query
Input: Encoder E, Decoder D, Surprise Estimator S, threshold
Input: Sensory memory M. +- 0 with budget B.
Input: Long-term memory Mi 0, Answer memory bank Mans
for to To
zi - M()
M. + U
// Streaming sensing
M.):
// Surprise estimat
Mal Br then
Remove oldest 2old from Ms;
Mr+ + Mr U {
11 Store to long-term memory
if st T then à + D(g, Mi):
// Answer query using long-term memory
10
MA t- Mas U {a]:
11
Mic e 0;;
// Reset long-term memory
12 return Sum(Mams
H.4 COMPARISONS WITH EXISTING LONG-VIDEO METHODS
We compare method (both surprise-rriven memory and agentic framework) with existing methods designed for long-video understanding, in Table 17. Specifically, all experiments here are conducted with our L.F--finetueed Cambrian-5-7B, with different strategy handle the ever-expanding visual sensory input. For MovieChat, we follow the official implementation in (Song et al., 2024), maintain fixed-size long-term memory bank, and the long-term and short-term memory budgets to 64 and 16, respectively. For Flash-VStream (Zhang et al., 2025a), as its abstract memory module introduces additional parameters and requires dedicated training process, we only implement the three remaining memory components (i.e., spatial memory. temporal memory, and retrieved memory), and keeping other hyperparameters aligned with the default setup.
Table Compare our framework with existing long-video methods on VSI-SUPER.
VSR (Duration in Mins.)
VSC (Duration in Mins.)
Eval Setups
10
30
60
120
240
10
30
60
120
MovieChat
18.3
21.7
16.7
26.7
25.6
0.0
0.0
0.0
0.0
Flash- VStream
28.3
33.3
23.3
28.3
31.7
0.0
0.0
0.0
0.0
Ours
45.0
40.0
40.0
40.0
40.6
42.0
35.0
34.0
H.5 VISUALIZATION OF CAMBRIAN-S'S PREDICTIONS v.s. GROUND TRUTH ON VSC
Fig. 17 visualizes the predicted counts VSC, revealing that Cambrian-S" predictions exhibit strong linear growth with the ground truth, aligning well with the # perfect-count line (gray dashed). Conversely, Gemini-2.5-Fla. 2.5-aahhs 's's predictions are confined limited range of small values and do not scale as the actual count increases, highlighting its poor extrapolation capabilities.
35

---

Published as a conference paper at ICLR 2026
CambrtnnS
Lin Perfect Prediction
Pediatin
50
Podficionn
100
200
30
150
100
150
200
Ground Truth
Ground Truts
Ground Truth
Ground Truth
Figure 17: Cambrian-S scales to higher ground truth object counts whereas Gemini saturates. Predicted counts plotted against ground-truth counts for videos of different lengths (10, 30, 60, and 120 minutes). Using surprise-driven segmentation, Cambrian-S"s predicted counts grow approximately linearly with the ground-truth, tracking the y x perfect-count line (gray dashed), whereas Semini-2.55-Fasse 's predicted counts remain clustered near small values and fail to increase with grouhd-truth count, indicating early saturation and poor extrapolation larger counts.
36

---

Published as a conference paper at ICLR 2026
Which of the following correctly represents the order in which the Stitch appeared video?
A. Stove, Trash bin, Refrigerator, Counter
B. Trash bin, Refrigerator, Counter, Stove
C. Stove, Counter, Refrigerator, Trash bin
D. Trash bin, Stove, Counter, Refrigerator
Which of the following correctly represents the order in which the Hello Kitty appeared in the video?
A. Nightstand, Bed, Crib, Blue bench Blue bench, Crib, Nightstand, Bed D. Blue bench, Bed, Crib, Nightstand
C. Bed, Nightstand, Blue bench, Crib
Which of the following correctly represents the order in which the Golden Retriever appeared in the video?
A. Bed, Table, Chest of drawers, Floor
B. Table, Chest of drawers, Bed, Floor
C. Chest of drawers, Floor, Table, Bed
D. Floor, Bed, Chest drawers, Table
Which of the following correctly represents the order in which the white Ragdoll appeared in the video? A. Ground, Trash bin, Bench, Table
B. Table, Bench, Ground, Trash bin
C. Ground, Trash bin, Table, Bench
D. Trash bin, Bench, Table, Ground
Figure 18: More examples of our VSI-SUPER Recall benchmark. Note that only edited frames are visualized.
37

---

Published as a conference paper at ICLR 2026
Absolute Direction (Object)
Standing by the backpack, looking toward the table, how far counterclockwise in degrees must I turn the trash bin?
Answer: 334.09
Absolute Distance
Measuring from the closest points of each, how far apart are the chair and the door in meters?
Answer: 2.32
Absolute Distance
Considering the chair and the door, which object's longest edge is the shorter?
A. Door
B. Chair
Figure 19: Examples of VSI-590K (Annotated Real Video).
Object Appearance Order
Determine the initial appearance order of these categories the video: door, chair, lamp, refrigerator.
A. refrigerator, door, lamp, chair
B. refrigerator, chair, door, lamp
C. refrigerator, chair, lamp, door
D. door, chair, lamp, refrigerator
Absolute Size
Provide the longest side's length for the door in inches.
Answer: 72.00
Room Size
Indicate the room's 's dimensions in square feet. If there's more than one room, estimate their total size.
Answer: 232.76
Figure 20: Examples of VSI-590K (Annotated Real Video).
38

---

Published as a conference paper at ICLR 2026
Relative Direction (Object Perspective)
Facing the door while standing near the window, in which of the following positions is the board relative me: front-left, front-right, back-left, back-right? Use Cartesian quadrants, with me at the origin looking toward positive y-axis
A. Back-right
B. Front-right
C. Front-left
D. Back-left
Relative Distance (Object Perspective)
Identify the object among (bookcase, chair, board, door) that closest to the window based on the shortest distance between their closest points. Choose the nearest instance if several exist.
A. Bookcase
B. Chair
C. Board
D. Door
Figure 21: Examples of VSI-590K (Annotated Real Video).
Relative Distance (Object Perspective)
If I am standing by the dresser and facing the chair, is the closet to my left, right, or back?
An object is to my back if I would have turn least 135 degrees in order face it.
A. Left
B. Right
C. Back
Figure 22: Examples of VSI-590K (Annotated Simulated Video).
39

---

Published as a conference paper at ICLR 2026
Relative Direction (Object Perspective)
With the toilet beside me and facing the cabinet, is the lamp positioned front-left, front- right, back-left, or back-right relative to me, based on Cartesian plane quadrants?
A. Back-right
B. Front-right
C. Front-left
D. Back-left
Relative Distance (Object Perspective)
Identify the object among (bookcase, chair, board, door) that is closest to the window based on the shortest distance between their closest points. Choose the nearest instance several exist.
A. Bookcase
B. Chair
C. Board
D. Door
Figure 23: Examples of VSI-590K (Annotated Simulated Video (Frame)).
40

---

Published as a conference paper at ICLR 2026
Object Counting (Relative)
If counted, would chairs be fewer than, more than, equal in number to tables?
A. Fewer
B. More
C. Equal
Relative Direction (Camera Perspective)
Through the camera's lens, is the sink captured on the left or right part of the scene?
A. Right
B. Left
Figure 24: Examples of VSI-590K (Unannotated Real Video (Frame)).
Object Counting (Absolute)
What would be the count you tallied all the chairs?
Answer: 6
Relative Distance (Camera Perspective)
In terms of proximity to the camera, which closer: a table or sofa?
A. Table
B. Sofa
Figure 25: Examples of VSI-590K (Unannotated Real Video (Frame)).
41


Reviews:
[{"review_id": "ARkBhiaNDQ", "reviewer": "Reviewer_jh5E", "summary": "This work presents a new benchmark and a new paradigm for learning multimodal video intelligence incorporated with the capability of predictive sensing using an internal world model. Extensive experiments and analyses verify that previous methods cannot well address this new problem via simply scaling data and compute and the proposed method achieves significant improvement.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.This work discusses an interesting new paradigm for multimodal video modeling called spatial supersensing, which aims to overcome the limitations of previous methods in predictive modeling driven by internal world model.\n\n2.The proposed predictive sensing paradigm seems to be capable of generalizing to various downstream tasks and could be a more advanced version of multimodal intelligence, supported by a lot of experiments and analyses.", "weaknesses": "1.Although the authors claimed that the proposed predictive modeling paradigm better helps downstream video understanding tasks, it seems this argument lacks sufficient experimental evidence. What would be the advantage when it comes to downstream generalizatioin comparing the proposed predictive paradigm and previous paradigms?\n\n2.Another concern is that, from my personal understanding, the proposed framework utilizes the \"error\" between the predicted next frame latent and the ground-truth next frame latent as a predictive signal. However, as we know that not all future frames are predictable, i.e., some errors are high because of those next frames are unpredictable but some errors are high because the model cannot make accurate predictions. How do the method handle these situations? Will the surprise measurement still make sense when the model comes across some unpredictable frames? And how would such error be utilized for specific video understanding goals?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W3OfVqF0Qb", "reviewer": "Reviewer_x9om", "summary": "This paper frames \"spatial supersensing\" as a long-term goal for multimodal AI, proposing a four-level taxonomy of capabilities: 1) semantic perception, 2) streaming event cognition, 3) implicit 3D spatial cognition, and 4) predictive world modeling.  The authors argue that current models and benchmarks are stuck at the first two levels, failing to address the challenges of continual, long-horizon spatial reasoning.\n\nTo demonstrate this gap, the paper makes three primary contributions. First, it introduces VSI-SUPER, a benchmark with two tasks (VSO for long-horizon recall and VSC for continual counting) specifically designed with arbitrarily long video streams to break the current \"brute-force long-context\" paradigm. Second, it curates a large-scale spatial instruction-tuning dataset (VSI-590K) and trains a new SOTA model family (Cambrian-S) that excels on existing spatial benchmarks. Third, it shows that this SOTA model still fails on VSI-SUPER, demonstrating a fundamental paradigm gap. The fix is simple: add a latent frame prediction head and use surprise (prediction error) to drive memory compression and event segmentation, which stabilizes accuracy with sequence length.", "questions": "1. In Table 2, why is VSO@10min even worse than VSO@30min?\n\n2. Is prediction error (surprise) measured by patch-averaged errors or some global tokens?\n\n3. Why is predictive error a better measurement than ground-truth error? I have a hard time in understanding this. To me, an educated guess would be that the next-frame prediction is purely trained on in-domain, short-clip data. But during evaluation, the model takes in out-of-domain observation (uncommon objects inserted by editing models or uncommon shot transition from room-to-room) that results in higher errors. Any insights from the authors regarding this question?\n\n4. Is it possible to provide a detailed and holistic compute requirement for evaluating the proposed benchmarks? e.g., how many tokens will be consumed for 10/30/60/120 mins-long videos? how many effective frames in total because of subsampled FPS? how many tokens per frame? what's the GPU requirement for running the evaluations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality and Significance: The paper’s primary strength is its insightful framing. The 4-level taxonomy is a clear and useful way to structure the field's challenges.  The diagnostic audit of existing benchmarks (Fig. 2), which shows many are solvable with text captions, is a solid contribution that validates the need for VSI-SUPER.\n\n2. The task design of VSO is well-grounded.\n\n2. The experimental structure is very effective at proving the paper's story.", "weaknesses": "Overall, this is a good work, although with some overclaims.\n\n1. **Limited Task Complexity:** While VSI-SUPER is effective at probing long-horizon memory, the tasks themselves are synthetic and narrow. VSO relies on finding artificially inserted objects, and VSC is a simple counting task (more on \"why calling it simple when frontiner models fail later). This is a reasonable first step, but these tasks do not yet capture the full scope of \"spatial supersensing,\" which should arguably involve more complex, emergent reasoning about object interactions, causality, or multi-step agentive plans over time.\n\n2. **Design philosophy for VSC:** I will take an opposite stance on the VSC design and the proposed solution. \n  - The VSC benchmark (continual counting) feels co-designed to justify the proposed \"surprise-driven **segmentation**\" solution. The paper frames this as a failure of long-context models, but simply creating a task that is infeasible for a standard MLLM is not, by itself, evidence of a \"paradigm gap.\" The VSC benchmark seems to be designed for the proposed method rather than the proposed method is designed for the benchmark. \n   - If we have the prior that this task requires divide-and-conquer at first, it is easy to come up with other divide-and-conquer methods. For example, (1) a tool-use baseline that first passes the video through a standard shot-detection algorithm and then counts within each segment. Shot-detection algorithms, which detects shot-transition, are very mature and widely used in data preparation pipelines for video generation models. (2) A non-learning baseline that uniformly clips the video into chunks, counts within each, and sums the results (which would likely perform well, barring minor errors at clip boundaries). Without these comparisons, it is unclear if the LFP head is a necessary innovation or just an overly complex solution for this specific task. **The difficulty stems from video length, which is solvable by simple partitioning, not from cognitive complexity that would necessitate a new paradigm**.  \n    - (more on tool-use) It’s trivial to make some infeasible tasks for current MLLMs. For example, asking an MLLM to count every grain of rice in an image is also infeasible or doing 40-digits arithmetic operations, but this a tool-use problem, not necessarily a fundamental cognitive one.\n\n3. (minor) **Limited Scope of Evaluation**: The entire experimental setup, from the VSI-590K dataset to the VSI-SUPER benchmark, is heavily skewed towards **indoor**, slowly-panned videos (e.g., ScanNet, ProcTHOR).  This leaves the robustness of the \"predictive sensing\" approach as an open question. It is unclear how the \"surprise\" signal, trained at 1 FPS, would generalize to more dynamic, real-world scenarios like egocentric video or outdoor driving scenes, which feature fast motion, variable frame rates, and different types of \"surprising\" events.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hRSESOKXLg", "reviewer": "Reviewer_32fi", "summary": "In this paper, the authors try to investigate the problem of long-stream spatial cognition issue when applying very long spatial video input, which raises difficulties of long-context and memory to handle well with spatial reasoning, planning, or continual comprehensions, etc. This paper then proposes a very hard dataset to comprehensively test the powerful multimodal image/video MLLMs, and finds that these powerful models still face serious bottlenecks to resolve the long-duration streaming video to maintain the useful and informative contexts or memory. To evaluate this ability, this paper proposes two benchmarks, so-called VSI-Super from both Long-horizon Spatial Observation and Recall and Continual Counting under Changing Viewpoints and Scenes, to evaluate MLLMs to observe long spatiotemporal videos and recall the specific locations of an unusual object in the correct order of its appearance, and continuous unique object counting in long-form spatial videos. Due to the limited resources and unbounded consumption, even the very powerful MLLM cannot solve these tasks well. A new dataset, VSI-590K, is then proposed to upgrade the CAMBRIAN-1 to achieve more powerful spatially-grounded models, ranging from 0.5B to 7B scales. This paper utilizes a concept, predictive sensing, to train the model to align with the latent feature of the next frame based on the current frame, which is then used to quantify the frames' context memory so as to handle the very long spatial video input by simulating the human's memory. After that, this paper obtains improved performances across various spatial cognition benchmarks.", "questions": "Hope the author can explain and demonstrate well the question above. I will consider raising my score if the authors rebuttal well.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The huge effort to collect and curate the VSI-Super benchmark and VSI-590K dataset demonstrates the great workload of this paper, which I think this can boost the spatial intelligence community if released with high-quality.\n\n2. The so-called predictive sensing, which is modeled by next frame prediction, sounds like a reasonable way to maintain the history memory context for the scenarios that go smoothly and do not change the scene or even entities drastically.\n\n3. Organizing the next latent frame error as a signal to decide the memory saving cost for the long-horizon context sounds interesting, though this seems very sensitive and vulnerable for the complex scene videos.\n\n4. The experimental attempts are extensive.", "weaknesses": "1. The VSO task, which requires MLLMs to observe long spatiotemporal videos and recall the specific locations of an unusual object in the correct order of its appearance, sounds very similar to Needle In A Video Haystack and the common spatial perception task, which requires object appearance order. Can the authors explain and demonstrate the main differences and also the motivations?\n\n2. Regarding the VSC task, does the model need to recognize and count the objects from different instance levels or just the category level? Is this within the very fine-grained object perception?\n\n3. From Figure 5, it is unclear how the paper lifts the 2D image into the 3D space. Can the authors provide more details?\n\n4. Regarding the video scenario, when the event scene changes into another new one, does the model still need to take efforts to store these old history contexts, since our human may try to forget this or not? I notice this paper sets up a fixed window to take the event memory in Fig. 12; however, what if the user asks for a retrieval or comparison with a very old object within the already cleaned memory pool?\n\n5. Regarding the VSO memory design, what are the differences when compared with the SAM2 memory design, Moviechat, or Flash-vstream?\n\n6. Though promising, the so-call supersensing sounds similar with the world model and long-video understanding, can authors provide with better demonstration about the critical differences?\n\n7. The writing organization looks a bit chaos and there is a typo in Line316-317.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SomtGh2q1c", "reviewer": "Reviewer_Y21j", "summary": "The paper proposes spatial supersensing as a framework for video understanding with four levels: semantic perception, streaming event cognition, implicit 3D spatial cognition, and predictive world modeling. It contributes: (1) VSI-SUPER, a benchmark for continual spatial sensing with two tasks (VSO for long-horizon recall, VSC for continual counting), (2) VSI-590K, a large-scale spatial instruction-tuning dataset combining real, simulated, and pseudo-annotated data, (3) Cambrian-S, a family of video MLLMs achieving SOTA on spatial tasks, and (4) a predictive sensing approach using latent frame prediction for memory management and event segmentation.", "questions": "Q1. Why is cosine distance a valid surprise metric? \n\nQ2. Why the complete failure on longer videos? Table 2 shows 0% on VSC for 120-minute videos. Is this a memory issue, accumulation of errors, or fundamental limitation?\n\nQ3. What about streaming video at higher frame rates? 1 FPS sampling may miss important events.\n\nQ4. Is it possible to get the human accuracy even on some sample of the data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1: It correctly and timely identifies that current video benchmarks focus on semantic perception while neglecting spatial/temporal reasoning.\n\nS2: It creates a Comprehensive dataset, VSI-590K which combines real annotated, simulated, and pseudo-annotated data with clear methodology.\n\nS3: Strong Baseline as Cambrian-S and novel and creative application of prediction error/\"surprise\" for both memory compression and event segmentation.", "weaknesses": "W1: Claims \"data alone isn't enough\" doesn't have proper justification. VSI-590K focuses on spatial cognition, not streaming/continual tasks. So, the failure on VSI-SUPER doesn't prove data is insufficient, it seems to show that they didn't collect data for that problem.\n\nW2: There are a lot of missing justifications for core design choices, for example: \n(a) Why is cosine distance a valid surprise metric? Any intuition or citation? \n(b) Lines 214-215, 319, 364 reference predictive sensing, surprise signals and other neuroscience related terms without proper citations, which might be difficult for the intended audience of the paper to understand.\n(c) Why 2-layer MLP for prediction?\n\nW3: Predictive sensing is only tested on two specialized tasks (VSO, VSC). The generalizability to other spatial reasoning scenarios is not very clear.\n\nW4. The pipeline for generating pseudo-annotations (Fig. 5) could introduce systematic biases. No analysis of annotation quality is provided.\n\nW5. No human baseline on the proposed benchmark.\n\n---\n\nMinor:\n\n1. Writing is not very clear and coherent.\n\n2. Typo: line 317, add space between fixed-context and Cambrian-S.\n\n3. Line 1024: GPT, which version?\n\n4. Missing related works: I see some other works which seems related to this and try to understand spatial and/or temporal reasoning of MLLMs, long video understanding and world models. I think authors should add some of these at least in the related work section [1, 2, 3, 4, 5, 6, ...] .  I know that the field is very dynamic and it's not possible to add all the papers, so they should consider adding these and other papers they can find as a minor suggestion. \n\n---\n\n1. Lu, Yiren, et al. \"Bard-gs: Blur-aware reconstruction of dynamic scenes via gaussian splatting.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n2. Kang, Bingyi, et al. \"How far is video generation from world model: A physical law perspective.\" arXiv preprint arXiv:2411.02385 (2024).\n\n3. Shen, Xiaoqian, et al. \"Longvu: Spatiotemporal adaptive compression for long video-language understanding.\" arXiv preprint arXiv:2410.17434 (2024).\n\n4. Upadhyay, Ujjwal, et al. \"Time Blindness: Why Video-Language Models Can't See What Humans Can?.\" arXiv preprint arXiv:2505.24867 (2025).\n\n5. Song, Chan Hee, et al. \"Robospatial: Teaching spatial understanding to 2d and 3d vision-language models for robotics.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n6. Lin, Yueqian, et al. \"Hippomm: Hippocampal-inspired multimodal memory for long audiovisual event understanding.\" arXiv preprint arXiv:2504.10739 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)