LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding
Authors: Hanyu Zhou
Gim Hee Lee
Venue: ICLR 2026 Poster (2026)
Keywords: 4D scene understanding
large multimodal model
spatiotemporal prompt
multimodal learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a general vision-language large multimodal model for 4D scene understanding.

Abstract: Despite achieving significant progress in 2D image understanding, large multimodal models (LMMs) struggle in the physical world due to the lack of spatial representation. Typically, existing 3D LMMs mainly embed 3D positions as fixed spatial prompts within visual features to represent the scene. However, these methods are limited to understanding the static background and fail to capture temporally varying dynamic objects. In this work, we propose LLaVA-4D, a general LMM framework with a novel spatiotemporal prompt for visual representation in 4D scene understanding. The spatiotemporal prompt is generated by encoding 3D position and 1D time into a dynamic-aware 4D coordinate embedding. Moreover, we demonstrate that spatial and temporal components disentangled from visual features are more effective in distinguishing the background from objects. This motivates embedding the 4D spatiotemporal prompt into these features to enhance the dynamic scene representation. By aligning visual spatiotemporal embeddings with language embeddings, LMMs gain the ability to understand both spatial and temporal characteristics of static background and dynamic objects in the physical world. Additionally, we construct a 4D vision-language dataset with spatiotemporal coordinate annotations for instruction fine-tuning LMMs. Extensive experiments have been conducted to demonstrate the superiority of our method on various tasks of 4D scene understanding. Our code: https://github.com/hyzhouboy/LLaVA-4D.

Published as a conference paper at ICLR 2026
LLAVA-4D: EMBEDDING SPATIOTEMPORAL PROMPT INTO LMMS FOR 4D SCENE UNDERSTANDING
Hanyu Zhou', Gim Hee Lee'
School of Computing, National University of Singapore (hy. zhou, gimhee. lee) 8nus edu.
ABSTRACT
Despite achieving significant progress in 2D image understanding, large multi- modal models (LMMs) struggle in the physical world due to the lack of spatial representation. Typically, existing 3D LMMs mainly embed 3D positions as fixed spatial prompts within visual features to represent the scene. However, these methods are limited to understanding the static background and fail to capture tem- porally varying dynamic objects. In this work, we propose LLa LA-4,, general LMM framework with novel spatiotemporal prompt for visual representation in 4D scene understanding. The spatiotemporal prompt is generated by encoding 3D position and ID time into dynamic-awar 4D coordinate embedding. Moreover, we demonstrate that spatial and temporal components disentangled from visual features are more effective in distinguishing the background from objects. This motivates embedding the 4D spatiotemporal prompt into these features to enhance the dynamic scene representation. By aligning visual spatiotemporal embeddings with language embeddings. LMMs gain the ability to understand both spatial and temporal characteristics of static background and dynamic objects in the physical world. Additionally, we construct 4D vision-language dataset with spatiotemporal coordinate annotations for instruction fine-tuning LMMs. Extensive experiments have been conducted to demonstrate the superiority of our method on various tasks of 4D scene understanding. Our code: https::/githhub.comm// wtw.:i/www...oo//yyoooo//l///aaa
INTRODUCTION
Large multimodal models (LMMs) (Alayrac et al., 2022; Liu et al., 2023) aim to learn the repre- sentation alignment between language and other modalities such as vision and audio. They have been widely applied in multiple scene understanding tasks such as dense caption (Wang et al., 2022), visual QA (Li et al., 2022a; 2023), scene grounding (Kamath et al., 2021), etc. Although recent language-vision LMMs including LLaVA (Liu et al., 2023) and PaLI (Chen et al., 2023b) have achieved great success in 2D image understanding. they still face challenges in the 3D physical world. This is because these LMMs trained solely on 2D images lack the representation of 3D) spatial characteristics to interact with the physical world. In this work, our purpose is to improve the characteristic representation and scene understanding of LMMs for the physical world.
As shown in Fig. ((a), existing 3D language-vision LMM methods (Hong et al., 2023; Zhu et al., 2024a;a; Zheng al., 2024; Chen et al., 2024a) 3D positions as spatial prompts which are then embedded into visual features to represent the whole scene. For example, Hong et al. (2023) extract 2D visual features from multi-view images and use 3D positions transform these features into corresponding 3D inputs for LMMs. However, these 3D LMM methods can only handle the static background with limited ability to understand dynamic objects the scene. Unlike static backgrounds, dynamic objects exhibit temporally varying spatial characteristics such position shifts and deformations. Unfortunately, existing 3D LMM frameworks neglect temporal aspects in using unified spatial representation for the entire scene. As shown in Fig. (c--d), 3D LMMs perform poorly on dynamic understanding tasks. This highlights the need to capture both spatial and temporal information to improve scene understanding in dynamic physical environments.
As shown in Fig. 1, we propose a novel spatiotemporal prompt embedded with the visual repre- sentation model the spatiotemporal characteristies of the scene. We design the spatiotemporal

---

Published as a conference paper at ICLR 2026
la black orr Risat
This a baakk Baxt is slowly moviig slong the rodd Aid.
Large Language Medel
Large Language Model
ScaRAA
Spotial Presep!
Spusio Temporaa Promp
Limbed
Embed
Vision Fnender
3D Positio
Vision Encoder
Cocrdinata
Encoding
Encoding
Cwith (Anigh
vide
dMeig
ondratze
endrunze
Multi-Vine View Imagess
Instraction
MultiNiew Videos
Insruction
INO BILWWAD VYoedDLLM aC
LMM (Ours)
(c) Performance on Multiple Benehmarks
4D Dense Caption
Visual Grounding
4D Question Amswrrin
o Whlthobee prose usky
U.S cieer berled a 12.0) sutS Hone doverie the slgne bortt pount hemootie
fueeliy ylad Lucki s.okay aoee iallakka with dack sajes fout amonn wall
on
iafall object is grac
20 LMM ras sangless bobl prace
1os (en (ct. 1.31
widdr.
Where obget kidyy ponoon
sasgleses gree
[1.5 3.5 ous, 30-
LAME The Mlscdppa lodki gooe
apwont
Viee
riciin over sgaine mooiii grifti..
115 A2ay
scadh the viador art acss
(d) Comparison 4D Understanding Tasks
Figure 1: Illustration of 3D and 4D LMM paradigms for physical world understanding. (a) Existing 3D-LMMs encode 3D positions as spatial prompts but overlook dynamic objects. (b) Our 4D LMM framework embeds 4D coordinates: position and time as spatiotemporal prompts to capture both background and dynamic objects. (c) Performance comparison of LMMs on benchmarks. (d) Comparison on 4D understanding tasks.
prompt based on the observation that dynamic objects and static backgrounds share similar 3D positional encoding but differ significantly in motion patterns. This motivates the extension of 3D positional encoding into a dynamic-aware 4D coordinate encoding to better differentiate objects from the background. Additionally, we observe that visual features of a scene can be disentangled into spatial and temporal components, which effectively distinguish objects from the background. wiwiioeempporl--dinettaanggeee vision embedding for scene representation. The 4D spatiotemporal prompt and visual spatiotemporal features jointly model the fine- grained spatiotemporal characteristics of dynamic objects and static backgrounds, thereby enhancing the 4D scene understanding of the physical world by LMMs.
This inspires the design of
In this paper, propose LLa 4A- general inguage-vision large multimodal model for 4D scene understanding. As illustrated in Fig. 2, our LLaVA-4D includes a dynamic-aware 4D coordinate encoding and a parpoottmmpora-disntnnannggee constructs 4D coordinates for multi-view videos and incorporates optical flow to enhance spatiotem- poral encoding. The vision embedding module disentangles visual features from multi-view videos into spatial and temporal components, and enriches these spatiotemporal features with dynamic-aware 4D coordinate embeddings through cross-attention fusion. We further perform spatiotemporal encod- ing on textual 4D) coordinates within language embeddings, which are then aligned with the fused visual spatiotemporal embeddings. In this unified framework, 4D coordinate encoding and visual spatiotemporal embedding collaboratively enhance the modeling of dynamic and static elements to improve 4D scene understanding LMMs. Additionally, we present Char4D, a 4D vision-language dataset with spatiotemporal coordinate annotations designed to instruction-tune our model for more effective 4D scene understanding. Our main contributions are summarized as follows:
vision embedding. The 4D coordinate encoding module
We are the first to propose a general vasion-language large multimodal model for 4D scene understanding. Our model embeds 4D spatiotemporal prompt into visual representation to enable LMMs to comprehend both dynamic objects and static backgrounds.
We observe that backgrounds and objects share similar 3D spatial position encoding but exhibit distinct motion patterns in the temporal dimension. This motivates the design of a dynamic-aware 4D coordinate encoding spatiotemporal prompt to distinguish objects from backgrounds.
We discover that spatial and temporal components disentangled from visual features more discriminative for background and objects. This inspires the apatiotemporalddisenttee vision embedding for scene representation.
2

---

Published as a conference paper at ICLR 2026
We build 4D
dataset with coordinate annotations for instruction fine-tuning and conduct extensive experiments to verify the effectiveness of our method in 4D scene understanding.
RELATED WORK
2D Vision-Language LMMs. Leveraging the strong reasoning capabilities of large language models (Brown et al., 2020; Touvron et al., 2023a;b), numerous vision-language LMMs (Alayrac et al., 2022; Chen et al., 2023a; Li et al., 2023; Lin et al., 2024; Liu et al., 2023; 2024; Li al., 2025a) have been developed to learn the correspondence between image-based visual and linguistic representations with broad applications in 2D scene understanding tasks. However, these vision-language L.M methods cannot work well when applied to the 3D physical world. This is because these LMMs lack the representation of 3D spatial characteristies and can only learn visual knowledge within the camera plane from 2D images, leading to underperformance of LMMs on 3D scene understanding, We thus aim to enhance the visual representation and improve LMMs on 3D scene understanding.
3D Vision-Language LMMs. The main challenge in understanding the physical world is representing 3D spatial characteristics. Some researchers (Hong et al., 2023; Wang et al., 2023b; Chen al., 2024a; Zhu et al., 2024a; Deng et al., 2025; Zhi et al., 2024; Zheng et al., 2024) address this by embedding 3D positions as spatial prompts within visual features, using point-based or image-based approaches. Point-based methods (Wang et al., 2023b; Chen al., 2024a; Zhi et al., 2024; Deng et al., 2025) reconstruct point clouds from 3D positions and use 3D vision encoder to extract features for LMMs. To reduce reliance reconstruction precision (Kerbl al., 2023), image- based methods (Zhu et al., 2024a; Hong et al., 2023; Zheng et al., 2024) encode multi-view images into 2D visual features concatenated with embeddings of 3D positions. However, these methods unified spatial representations that limit their ability to capture dynamic objects with temporal variations. We thus propose a novel spatiotemporal prompt for dynamic scene representation, embed it into multi-view visual features to enable 4D scene understanding in LMMs.
4D Vision & Language. A related line of research is 4D vision and language (Li al., 2025b; Deng et al., 2024; Sun et al., 2024), which focuses on modeling spatiotemporal characteristics. Li et al. (2025b) use 4D Gaussians (Wu et al., 2024) to represent dynamic scenes for semantic caption queries of different targets. Deng et al. (2024) introduce 4D encoder to directly extract scene visual features for alignment with object recognition texts. However, these 4D models have two limitations. First, they are usually task-specific and can only handle similar cases within the same data distribution of training set. Second, they adopt the same representation strategy for dynamic objects and static background, which has the potential risk of misalignment of heterogeneous features. In this work, present the first general LMM for different tasks of 4D scene understanding by disentangling visual features and embedding 4D spatiotemporal prompts to differentiate objects and background.
3 OUR LLAVA-4D
Overview. Fig. shows the architecture of our VA-44. Given a multi-view video input sequence I, our LLaVA-4D achieves 4D scene understanding progressively through the following three stages:
1) Dynamic-Aware 4D Coordinate Encoding (ef. Sec. 3.1). This is the 4D prompt construction stage where we construct 4D coordinate tensors [x: y, from multi-view videos using visual geometry, and perform spatiotemporal encoding (-). TE(-) on the coordinates. The encoded
position and time are concatenated as spatiotemporal prompt to guide visual fusion, i.e.:
p4n
[PE(æ, =) TE(t) -8).
(1)
where Wp is MLP-based learnable parameter and /3 is optical flow for temporal dynamic awareness. 2) Saapttempoora----Dtannaea Vision Embedding (ef. Sec. 3.2). This is the visual representation stage where extract visual features from multi-view videos using vision encoder, and disentangle these visual features into spatiotemporal components:
fa. fi STD()).
(2)
where fa is spatial feature and ft is temporal feature. We further embed encoded 4D coordinate features into these spatiotemporal features via cross-attention fusion:
f.t CAtt(ff., PaD),
(3)
where Jst denotes the output visual spatiotemporal feature with 4D awareness.
3

---

Published as a conference paper at ICLR 2026
[Re: This tiger is walking in snow, scanning its survunddings. 12, is p1) AL is (x2, =2]... At my, zn].
Pretrained Large Language Model
Vicunl Tokovs
Linguistic Tokens
Projector
Cross-Aliention Fusion
Position Encode
Sparial
Tamporal
MLP
Encode
Encode
Concat
Language Fmbedding
Vision Encode
Disentangle
Position Eneode
Time Encodo
Word
Position
Time
Text Tokemizer
What tiger 27] Please provide the position of the tiger each time.
Fie
View 2 Vision Input
Plaara
Fan
Frew
Flam
Paired
4D Coordinate
Language Instruction
Spatial Token
Tensporal
Liqussti Poskion Tokeu
Lisguistic Time Tikes
TICH
Our LLaVA-4D consists of three stages: I) 4D coordinate encoding. Encode 3D position and ID time with optical flow. 2) Vision embedding. Disentangle visual features into spatiotemporal features and embed the encoded 4D coordinates via cross-attention fusion. 3) Language embedding. Align textual position and time with the fused vision embedding for 4D scene understanding.
Figure 2:
3) Coordinate-Aligned Language Embedding (ef. Sec. 3.3). This the linguistic representation stage where visual spatiotemporal features are projected into the language embedding space using multi-layer perception: -. MLP(fst), where To denotes visual spatiotemporal to- kens. Input instructions tokenized into language space, where textual position and time spatiotemporally encoded into corresponding linguistic tokens T7"
The output r, and 77" denote the visual and linguistic representation with 4D coordinate prompt, respectively. Subsequently. the LLM utilizes these enhanced visual and linguistic tokens to improve 4D scene understanding. Our unified framework embeds 4D coordinates as spatiotemporal prompts into visual representations to enable spatiotemporal understanding of dynamic objects and static background in LMMs. The following sections detail the design cach stage.
Remarks.
3.1 DYNAMIC-AWARD 4D COORDINATE ENCODING
2D LMMs can capture spatial relationship between targets in images by using vision encoder to implicitly or explicitly incorporate 2D position encoding. Similarly, 4D scene understanding can be enabled by the integration of 3D positon and 1D time into LMMs.
4D Coordinate Definition. Given an image from certain view at timestamp t, we use SfM (Schonberger & Frahm, 2016) for camera pose P = [R T] and MVS (Seitz al., 2006) for depth D. Combined with intrinsic parameter K, we transform 2D pixel coordinate 22n to world coordinate system via geometric projection (Zou et al., 2018; Zhou et al., 2017):
EaD = R (D((1D) K -T),
TnD
(4)
where æ3D = [x., y. 2 denotes 3D position. After traversing all videos, we concatenate time and corresponding 3D position to form the 4D coordinate tensor [xr, y. 2,
Spatiotemporal Encoding. We perform spatiotemporal encoding to convert 4D coordinates into learnable feature patterns. It is challenging to directly distinguish objects from the background solely based on spatial dimensions such as multi-view images captured at a specific time. We circumvent this challenge by adopting the same spatial position encoding strategy for objects and background via learnable Fourier feature (Li et al., 2021):
(5) where d denotes the dimension and W, the learnable parameter of the Fourier feature. From the temporal dimension such continuous video sequence certain view, objects and background
Pays = PE(x, 2) = 1/vd [cos[[æ, y, =]W,T sin([2, y, 2 W)),
4

---

Published as a conference paper at ICLR 2026
New
OPject
4D Dypami Scene
Vision Eiscoder
Citdder
Shan
Moomiler
Monffold
Olyvorr
Baokgrandd
Vision
SparioTeemperll Discntangle
Einooder
Clusue
View
Multi View Videos
Figure 3: Feature distribution of static background and dynamic object in a 4D dynamic scene. Visual features of dynamic objects appear scattered while static backgrounds are clustered. In contrast, spatiotemporal features show clear discrimination between objects and background.
have different motion patterns and thus add motion information into the temporal encoding:
(6) where 3 is an estimated optical flow, and ((.) is softmax function. Note that the optical flow is used as an auxiliary motion cue for temporal encoding instead of as the sole source temporal information for dynamic scene understanding. We further concatenate the spatial and temporal encoding outputs to obtain dynamic-aware 4D coordinate embeddings as the spatiotemporal prompt for subsequent visual fusion. Moreover, this spatiotemporal prompt is extensible, where can further add some other spatiotemporal attributes such as semantic and action to guide the alignment between visual and linguistic representations (ef. Sec. 5.2 for details).
= TE(() 3 1/vd [cos( sin(tW,T)) +1++ (B)),
3.2 SPATIOTEMPORALS DILEAAANASEEE VISION EMBEDDING
Unlike 2D image and 3D scene, 4D scene consists of spatial such as color and temporal such as motion components. A unified visual representation for 4D scene usually suffers from misaligned het- erogeneous features, which inspires us to disentangle visual features into spatiotemporal components.
Spatiotemporal Disentanglement. Spatial features mainly reflect the appearance of the entire scene, while temporal features focus more on continuous varying in motion patterns. After obtaining multi-view visual features fu,,, where is the view and is the time, get the correlation of visual features between different views at the same time as spatial features:
f. Aggregate (ffatte(((fmm.eessss. | + j))-
Next, we further calculate the correlation of visual features between adjacent images of continuous time at the same view as temporal features:
ft Afgregatee((f....foo---aa
To illustrate the importance of spatiotemporal features, encode the selected object and background regions of multi-view videos into visual, spatial and temporal features, and cluster these features for visualization in Fig. 3. The visual features of the object region appear scattered, but the spatiotemporal features of both the object and background regions are clustered. This indicates that spatiotemporal features are highly discriminative for the entire scene. Consequently. disentangling visual features into spatiotemporal components is essential for effective 4D scene representation.
Attenion Fusion. Single disentangled spatiotemporal features cannot be localized world coordinate system. We need to further embed 4D coordinates into the spatiotemporal features for localization. We first introduce an MLP to make the dimension of the 4D coordinate embeddings the same as the dimension of the spatiotemporal features: p4n MLP 4D coordinate embeddings with the spatiotemporal features via cross-attention mechanism:
pc)). Next, we fuse the
WWPDD k wh ft), a=a-v, v. a o(MLPo oby (PID)), fst o (1-a) a) fe.
[fa, fe), softmax(gk: /Vd),
(9)
where is learnable weight. As a result, we can obtain the 4D-aware visual spatiotemporal feature.
3.3 COORDINATE- ALLGNED LANGUAGE EMBEDDING
Although vision embeddings can represent scene knowledge, LMMs understanding scenes require the alignment between visual and linguistic representations. Since the input of large language model
5

---

Published as a conference paper at ICLR 2026
Stuge Content Alignment
View
View 2
View
d0 (25
2D DC
2D QA
3D DC
3D QA
DX ChardD
MG
C3 C (P
3D Detection
ChaGPP-4V
Stuge 2: Coordinate Alignment
203.4k
181.K
2D VG
3D VG
Bus
Point
Time
Color
879.12 Fa
Text-Cloly
GFT
Stage 3: Instruction Fine-Tuning Tuning
224.6k
O
Hus renyy
Ar. acaue it cg
Tea Eush slies
4D DC
4D QA
4D VG
beig bunal A..
noled in Malk pis awtregtiec
larg:
12
(a) Overview of Chur4D Dutoset
(b) Truining Pipeline
(c) Instruction Generation
Overview of our dataset and training pipeline. (a) Chat4D dataset includes 2D, 3D, and 4D vision-language training sets for dense captioning, QA, and visual grounding. (b) Three-stage training: stages 1-2 use 2D/3D data for initialization; stage 3 uses 4D data for instruction fine-tuning, (c) Spatiotemporal characteristics are extracted as local descriptions to generate 4D instructions.
Figure 4:
requires text-like tokens, we first build a multi-layer perception to project the fused spatiotemporal features into language embedding space for preliminary alignment with linguistic representations. This projected fused spatiotemporal features is the visual tokens denoted as n,. Subsequently, we tokenize the input instruction into the language space with word tokens Tr, and apply the same spatiotemporal encodings PE(-) and TE(-) to textual position tp and time tt:
Tv = PE(pp), T. = TE(()..
(10)
We further fuse the encoded position and time with corresponding word token: T/" 72 + where w, and Wy denote the learnable weights. We concatenate 4D-aware visual tokens with coordinate-aligned linguistic tokens for the LLM to reason. Particularly, 4D coordinate encoding ensures spatiotemporal localization of the scene within our unified framework. Additionally, dis- entangled vision embedding models spatiotemporal knowledge of the scene, and vision-language alignment further enable the LMM achieve interactive understanding of 4D scenes.
DATASET AND TRAINING PIPELINE
4.1
OUR CHAT4D DATASET
Many vision-language datasets have been proposed to evaluate 2D/3D scene understanding of LMMs. However, there is currently no vision-language dataset specifically designed for 4D scene understanding in LMMs. To address this gap, we introduce the Chat4D dataset. As illustrated in Fig. 4(a-b), our dataset includes 2D, 3D and 4D vision-language data types, where 2D/3D data are used to initialize multimodal spatiotemporal understanding and 4D data is used for instruction fine-tuning.
2D&3D Vision-Language Data. To develop multimodal spatiotemporal understanding, our model requires image- format inputs and a large number of rision-language pairs. We thus integrate existing standard 2D/3D spatiotemporal datasets (Chen et al., 2024b; Wang et al., 2023a; Luo et al., 2023; Azuma et al., 2022; Lyu et al., 2024; Chen et al., 2021; Zhang et al., 2023) and adapt specific text instructions (e.g., 2D/3D position and time) to align with our spatiotemporal encoding strategy. This approach effectively trains the LMM for spatial and temporal understanding. These datasets cover dense captioning (DC), visual QA and visual grounding (VG) tasks with a total of 654.5K samples.
4D Vision- LanguaeL Data. We merge existing 4D dynamic scene reconstruction datasets train the LMM for 4D spatiotemporal understanding: iPhone (Gao et al., 2022), HyperNeRF (Park et al., 2021), N3DV (Li et al., 2022b), PanopticSports (Luiten et al., 2024), DAVIS (Perazzi al., 2016), and Immersive (Broxton et al., 2020)). Note that most videos in these datasets last 6-12 seconds. Additionally, we develop a data generation approach to produce paired 4D lannuan data for instruction fine-tuning. As shown in Fig. 4(c), we utilize the 3D object detection method (Rukhovich et al., 2022) and GPT-4V (Yang et al., 2023) to extract local spatiotemporal information such as category, position, time from multi-view videos. These extracted features are then processed by text-only GPT to generate global 4D descriptions in understanding tasks produce dataset of 224.6K samples. To further improve label quality, apply two rounds of data cleaning, such as automatic filtering and manual inspection. For the automatic filtering, enforce temporal consistency and spatial overlap constraints remove 4.7%
formats for typical

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative results of LMMs for scene understanding tasks differentt 3D and 4D datasets.
3D Benchmark
4D Benchmark
Methods
Scan2
SeanQA
MullsDReter
ScanRet
ChariD (Ours)
C610.57
B-4 8.0.1 Mc0.55
CT
B-4T
MT
FIGO.ST
SAc0010.511
CT
B-4T
SAcci00.51 TAccf
3D-LLM
69.4
12.0
14.5
61.6
11.3
31.4
Chat 3D v2
63.9
31.8
87.6
14.0
41.6
38.4
81.8
13.7
39.5
78.8
92.6
Grounded 3D-LLM
70.6
35.5
72.7
13.4
40.6
44.1
66.3
12.2
43.7
3D
PQ3D
80.3
36.0
29.1
87.8
17.8
50.1
51.2
84.7
14.3
51.5
LLaVA-3D
79.2
41.1
30.2
91.7
14.5
20.7
42.2
87.4
14.8
45.6
Video 3D LLM
83.8
42.4
28.9
102.1
16.2
19.8
52.7
51.7
89.4
16.1
52.8
Spatial-MLI.M
3UR-LLM
GPT-4u w/ Co, Com.
87.0
18.0
4D
LLaVA-ID (Ours)
85,3
45.7
31.3
97.8
17.9
21.2
54.3
53.2
93.5
17.2
58.9
54,6
Table 2: Quantitative results of LMMs for scene understanding tasks on VSI-Bench.
Numerical Answer
Multiple-Choie Choice Answer
Methods
Average
Obj, Count
Abs, Dist.
Obj, Size
Room Size
Rel. Dist
Rel. Dir.
Route Plan
Appe, Order
-LaVA-Vidco-7B
35.6
48.5
14.0
47.8
24.2
43.5
42.4
34,0
30,6
LLaVA-One Visioono
32.4
47.7
20.2
47.4
12.3
42.5
35.2
29.4
24.4
VongVA-7B
29.2
38.0
16.6
38.9
22.2
33.1
43.3
25.4
15.7
VILA-1 5-8B
28.9
17.4
21.8
50.3
32.1
34.8
31,0
IntemVL2--BB
31.3
38.0
33.4
SpatialMMLLM MLMM
48.4
65.3
34.8
63.1
45.1
41.3
46.2
33,5
46.3
LLaVA-4D (Ours)
48.6
68.2
35,3
64.8
49.6
44.5
45.2
33.8
47.5
of the samples with inconsistent motion, impossible object trajectories, or mismatched spatial regions. For the manual inspection, we manually discard 0.8% of the remaining samples with abnormal labels, including timestamp misalignment, large coordinate deviations, and incorrect color or attribute descriptions. The two steps substantially increase the reliability of the final annotations.
4.2 TRAINING PIPELINE
To ensure the stability of the training process and improve the performance of the model, we divide the entire training into three stages in Fig. 4 (b) follows:
Stage 1: Content Alignment. The training sets of the DC and QA tasks in the 2D&3D vision- language data of Chat4D are used initially align the content between visual and linguistic representations. This provides a foundational spatiotemporal understanding for the proposed model. At this stage, only parameters of the pros-aateention fusion and the projector are updated. The 4D spatiotemporal coordinate features PAD are temporarily set as zero padding.
Stage Spatiotemporal Coordinate Alignment. In order further improve the fine-grained understanding capability of our model under the spatiotemporal coordinate prompt, we use the training data of the VG task in the 2D&3D vision-language subset of our Chat4D to refine the spatiotemporal coordinate alignment between visual and linguistic representations. At this stage, update all trainable parameters of 4D coordinate encoding and cross-attention fusion modules while keeping all other modules frozen.
Stage 3: 4D Task Instruction Fine- Fine-Tuning. To further improve our model for 4D scene under- standing, use 4D vision- lisiuanlanguage data of Chat4D to enhance the generalization of our model for fine-gra greiggraiedd spatiotemporal understanding with 4D coordinates through a multi-task instruction fine-tuning strategy. All trainable parameters updated while the vision encoder remains frozen.
EXPERIMENTS
Implements Details. Our LLaVA-4D utilizes the pre-trained weights of LLaVA-1.5-7D (Liu et al., 2024) and the vision encoder of LLLP--VIT--336pp (Radford et al., 2021). aoossnattnntion fusion module is transformer-batt based architecture. The whole model is trained on RTX 4090 GPUs over 86 hours using AdamW as the optimizer. training stages 1-2, we set the learning rate to 1.0e 4 with batch size of 48. We use learning rate of 1.0e 5 with batch size of 16 in training stage 3.

---

Published as a conference paper at ICLR 2026
1,5s
1.7s
1.98
2.18
2.3s
View
View
View
View
User
What does the person do? Please provide its courdimates.
LLaVA-3D
A person is riding hurse, jurnging over obstacle in an ouldoor equestrian arena. II is at [11.26, 2.10, 7,35, 12.48, 0.36, -7.10).
Video-3D LLM
The person horseback riding sandy arena, jumping over high obstacle during show jumping com- petition, surrounded fonces and other jump setups. is at [11.04, 2.05, -7.30, 12.39, 0.31, -7.05.
LLaVA-4D
This person is riding brown horse sandy mid-air, landing smonthly, and continuing toward the next obstacle. At 1.5s, it is at (11.15, 2.08, -7.32, 12.45, 0.30, 77.15). Al ..., [-.]] A12.3s, it is at (12.80, 1884, 66800 14,26, 0.12, 6651].
upproaching the hurdle, jumping high, clearing
Figure 5: Visual comparison of LMMs 4D scene understanding.
Comparison Methods. Since there are currently no public 4D LMMs for comparison, we compare our model with 3D LMMs: 3D-LLM (Hong et al., 2023), Chat-3D v2 (Huang et al., 2023), LL3DA (Chen et al., 2024a). 3D-LLaVA (Deng et al., 2025), Grounded 3D-LLM (Chen al., 2024c), PQ3D (Zhu et al., 2024b), VL-33 (Zhu et al., 2024a), Video-3D LLM (Zheng et al., 2024), Spatial- MLLMM (Wu et al., 2025), 3UR-LLM (Xiong et al., 2025), and GPT-4o with Coarse Correspondences (Liu et al., 2025). For fair comparison, all methods are fine-tuned the same evaluation benchmark.
Evaluation Metric. We compare all competing methods on multiple 3D datasets: Scan2Cap (Chen et al., 2021), ScanQA (Azuma et al., 2022), ScanRef (Chen et al., 2020) and Multi3DRefer (Zhang et al., 2023) and our Chat4D dataset. We evaluate the quality of generated text response for Scan2Cap and ScanQA in terms of CiDEr (C), BLEU-4 (B-4) METEOR (M). We choose the F1 metric of object prediction precision for MRefe and the accuracy of intersection over unions for grounding task from ScanRef. The metrics are also applicable the evaluation our Chat4D) where grounding accuracy is divided into spatial and temporal components: S/TAcc.
5.1 COMPARISON WITH OF-TE-OFTTEE-OOE MODELS
Quantitative Results. In Table 1, we compare the competing methods on 3D and 4D datasets. For method performs better than other methods. This shows that
3D understanding comparison, spatial features disentangled from multi-view images have stronger representation than ordinary visual features for 3D scene. For 4D understanding comparison, our method achieve significant strength due to dynamic-aware 4D coordinate as spatiotemporal prompt.
Comparison on VSI-Bench. In Table 2, we introduce VSI-Bench (Yang et al., 2025) and compare our model with several representative LLMs: LLaVA-Video-7B (Zhang et al., 2024c), LLaVA- VneVinin (Li et al., 2024), LongVA-7B VA-7B (Zhang et al., 2024b), VILA--15-8B (Lin et al., 2024), InternVL2-88 (Chen et al., 2024c), and Long V ILA-BB (Chen et al., 2024d). The results show that our VA-DD achieves the best performance on most metrics, with only a few metries slightly lower than competing methods. This verifies the superiority of our model in multimodal spatial reasoning.
Qualitative Results. In Fig. we select a typical 4D scene from our Chat4D dataset visualize the comparison between our model and 3D LMMs. The results show that 3D LMMs cannot respond to timestamp and corresponding temporal information, while our method can understand the temporal content. This is because 3D LMMs lack the representation of temporal characteristic. In contrast, our method introduces the spatiotemporal prompt to enhance the dynamic representation of 4D scenes.
Comparison on Temporal Understanding. In Table 3, we select a subset of video clips contain- ing typical actions and obtain the corresponding time intervals for these actions to compare our model with several video-based LMMs: LLaVA- ST (Li et al., 2025a) and Grounded- VideoLLM (Wang et al., 2024). The results show that our tloU scores are comparable to those of the baseline models, but our spatiotemporal understanding metrics,
Table 3: Comparison on temporal understanding
Method
SAcr@0.51
TAce?
1ol.
Groanded V.
9.4
5.1
47.0
LLaVA-ST
15.2
7.3
58.7
LLaVA (Ours)
54.6
8

---

Published as a conference paper at ICLR 2026
Object
View
Map
Object
Object Temporal
View
Map
Map
Temporoal
Spotial
Map
Object Spotial
View
Map
Background
Background
Background Spatial
Visual Feature Encoding
Spatiokemppral Discntangling
Coordinate Embedding
Figure 6: Feature visualization at different stages. Spatiotemporal disentanglement improves the discriminability of background and objects, which are further separated by 4D coordinate embedding.
such SAcc and TAcc, are significantly higher. This is because tloU mainly depends on video-level timestamp annotations, whereas SAcc and TAcc rely on 4D spatiotemporal coordinate understanding. Therefore, these results further demonstrate that our LLaVA-4D represents promising paradigm with fine-gre fine-grained multimodal spatiotemporal reasoning capabilities for 4D dynamic scene understanding.
5.2 ABLATION STUDY AND DISCUSSION
Effect of Visual Representation Modules. In Table 4, we verify the effectiveness of coordi- nate embedding, feature disentanglement and feature fusion modules in visual representation. Coordinate embedding is the key improving the overall performance of 4D understanding
Table 4: Effect of visual representation modules.
Coor. embed Fear. disent. Feat. fusion) C1 B-4f SAcc@0.51 TAcef
62.3 11.7
34.8
12.7
85.4 15.1
51.5
47.5
X
89.0 16.5
54.3
51.2
93.5 17.2
58.9
54.6
tasks by a large margin. Feature disentanglement improves the upper limit of 4D scene understanding to certain extent by strengthening the representation of spatial and temporal characteristics. Feature fusion further enhances the spatiotemporal understanding ability of the LMM.
Role of 4D Coordinate Encoding, In Table 5, we analyze the impact of 3D position encoding and ID time encoding on the performance of 4D understanding. When 4D coordinates are not encoded, the spatiotemporal understanding performance of the LMM is negatively affected certain extent. 3D position encoding mainly contributes to the spatial understanding ability of the LMM, and 1D time encoding can further improve the performance temporal understanding.
Table 5: Role of coordinate encoding.
Encenling Larget
Ct
B-47
SAcc600.57
TAccT
w/o Encondig
75.0
12.1
47.2
46.8
w/ 3D position
88.6
15.3
53.4
47.0
time
w/ 4D cocrdinate
93.5
17.2
58.9
54.6
How Spatiotemporal Features Work? We study how spatiotemporal features work within
model in Fig. 6. Initially, the visual features objects and backgrounds appear relatively scattered. After spatiotemporal disentanglement, object features distinctly divided into spatial and temporal components with background features remain clustered. Incorporating coordinate embedding further organizes object features into two distinct sets: spatial and temporal where background features consolidate into a unified spatial set. This demonstrates the strong spatiotemporal representation capability of our method for dynamic objects and static backgrounds.
Choice of Spatiotemporal Fusion Strategy. Table 6 shows attention-bra based fusion outper-
Table 6: Discussion on spatiotemporal fusion.
Fusion straliggy
Ct
B-4
SAccor05t
TAcet
forms concatenation and weighting fusion strate-
w/ Concatenation
89.0
16.5
54.3
51.2
gies. This is because concatenation and weight- ing rely global unified fusion with fixed
Weighting
89.5
16.5
55.1
51.4
w/ Attention
58.9
54.6
weights. In contrast, attention-base fusion can dynamically adjust the fusion weights of spa- tiotemporal features according to 4D coordinate embedding. This allows the LMM effectively focus more on meaningful spatiotemporal features for 4D understanding.
Extensibility of Spatiotemporal Prompt. To verify that the spatiotemporal prompt in model is an extensible feature, we introduce additional spatial semantic and temporal action masks as prompts based on the encoded 4D coordinates train model. As shown in Fig. our model can reason

---

Published as a conference paper at ICLR 2026
Spatial Semantic
Tempomi Action
1.4s
1.9%
2,45
User
How many persons in the scene?
User
What is the person doing this scene?
LLaYA-4D
Four:
LLaYA-4D
This person is performing dince move on street, spinning on one hand, and switching leg positions mid-air.
User
Please segment these persons.
User
Please segment this dunce move.
LLaVA-4D
Sure, [SBG].
LLaVA-4D
1.4, [SEG]. A: 19%, [SEG].A12.k A12.4s, (SEG).
Figure 7: Visualization of spatiotemporal prompt extended other spatiotemporal vision tasks.
about the visual features of semantic and action based on specific text instructions. Incorporating spatiotemporal prompts thus enhances the generality of our model for various vision tasks.
Impact of Textual Coordinate Encoding. Ta- ble ablates the impact of textual coordinate encoding scene understanding by the LMM. We deduce: 1) Textual coordinates as instruc- tions improve the fine-grained spatiotemporal understanding of the LMM. 2) Textual coordi- nate encoding further improves the upper limit of 4D spatiotemporal understanding. This is because coordinate encoding helps minimize the risk of LLM misinterpreting coordinate values.
Table 7: Impact of textual coordinate encoding.
instruction
Ct B.4 SAcc@0.51 TAcef
wlo Courdinate
83.5
13.2
43.2
25.8
w/o Encoding
90.1
16.7
56.3
53.0
Courdinate
w/ Encoding
93.5 17.2
58.9
54.6
Robustness to Annotation Errors. In Table 8, we discuss the impact of annotation errors, where we inject small random perturbations into the 3D coordinates and timestamps using mild Gaussian noise, and fine-tune our model on the perturbed version of Chat4D. The results show that the drops across all metrics are within point, which is small relative to the absolute performance level. This suggests that our LLaVA-4D is robust to minor spatiotemporal noise in the annotations.
Table 8: Discussion impact of annotation errors.
Fine luning stralegy
Ct
B-4f
SAco@0.5t
TAcef
Our buse LLaVA 4D
93.5
17.2
58.9
54.6
Fine-tuned
17.0
53.9
perturhed Chat4D
(0.5L)
(0.25)
(0.8.)
(0.74)
Effeet of Temporal Encoding Strategy. In Ta- ble 9, we analyze the effect of different temporal encoding strategics. We can observe that frame rate-based encoding performs worse than the motion speed-based encoding, especially in fine- grained temporal understanding. The underlying reason is that the frame rate- rate-based approach uses globally fixed-step encoding, which struggles to explicitly capture the temporal characteristics of independently moving objects in dynamic scenes. In contrast, our motion speed-based approach is locally adaptive encoding scheme that is capable of directly representing the magnitude of motion, and thus reflecting the true physical dynamics of the scene.
Table 9: Effect of temporal encoding strategy.
Temporal encoding strategy
C1
B-47
SAcc@0.55
TAcef
Frame rate-hased encoding
91.0
16.7
57.3
49.5
Motion speed-based encoding
93.5
17.2
88.9
54.6
Limitation. Our model has achieved success in end-to-end inference for scene understanding, but the preprocessing stage related 4D coordinate construction relies on external modules, e.g. SfM. This slightly reduces the overall efficiency. Since the preprocessing stage is replaceable, we plan to adopt end-to-end geometric models, e.g. MonST3R (Zhang et al., 2024a) improve practicality.
6 CONCLUSION
In this work, we propose VL-4A- a first general lasion-l LMM for 4D scene understanding. We introduce dynamic-aware 4D coordinate encoding as spatiotemporal prompt for scene content localization. Additionally, we propose a pnaiiotepooralldiientaanggee vision embedding method that integrates 4D spatiotemporal prompts into disentangled spatiotemporal features for effective scene representation. By aligning visual spatiotemporal embeddings with language embeddings, our approach allows LMMs to comprehend the 4D physical world. To support training, construct Chat4D, a comprehensive dataset covering 2D, 3D and 4D vision- language data for multimodal spatiotemporal understanding. Extensive experiments validate the effectiveness of our method.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research work is supported by the Tier 2 grant MOE-T2EP20124-0015 from the Singapore Ministry of Education.
REFERENCES
Bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Arthur Mensch, Katie Millican, David Moore, Michael Needham, al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 2022.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe, Scanqa: 3d question answering for spatial scene understanding. In IEEE Conf. Comput. Vis. Patlern Recog., pp- 19129-19139, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Michael Broxton, John Flynn, Ryan Overbeck, Daniel Frickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with layered mesh representation. ACM Trans. Graph., 39(4))66-1, 2020.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nielner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pp. 202-221. Springer, 2020.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint 19000223, 2023a.
Mandy Chen, Adams Wei Yu, Hamid Palangi, Paul Smolensky, Yinfei Yang, Xiaowei Yuan, Kathy Meier-Hellstern, Jianfeng Gao, Ed Chi, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2003.07822 2023b.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. L13da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26428-26438, 2024a.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang- wei Chao, Byung Eun Jeon, Yuwei Fang, Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13320-13331 2024b.
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprini arXiv:2405. 10370, 2024c.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint ar0i 10188, 2024d.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi gpt-4v? closing the gap to commercial
Hu, Jiapeng Luo, Zheng Ma, al. How far multimodal models with open-source suites. Science China Information Sciences, 67(12)220010,, 2024e.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel Chang. Scan2cap: Context-award dense captioning in rgb-d scans. In IEFE Conf. Comput. Vis. Pattern Recog., pp. 3193-3203, 2021.
11

---

Published as a conference paper at ICLR 2026
Jiajun Deng, Tianyu He, Li Jiang, Tianyu Wang, Feras Dayoub, and lan Reid. 3d-llava: Towards generalist 3d Imms with omni superpoint transformer. arXiv preprint arXiv:2501.01163, 2025.
Zhichao Deng, Xiangtai Li, Xia Li, Yunhai Tong, Shen Zhao, and Mengyuan Liu. Vg4d: Vision- language model goes 4d video recognition. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5014-5020. IEEE, 2024.
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Systems, 35:
17768-33770, 2022.
Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20494, 2023.
Haifeng Huang, Zehan Wang. Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers. arXiv preprint arXiv:2312.08166 2023.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, Nicolas Carion, and Karteek Alahari. Mdetr: Modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1780- 1790. 2021.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 2024.
Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. Llava-st: A multimodal large language model for fine-grained spatial-temporal understanding. arXiv preprint arXiv:2501. 082282, 2025a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-mmag pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 12888-12900. PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5521-5531, 2022b.
Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang, and Hanspeter Pfister. 4d langsplat: 4d language gaussian splatting via multimodal large language models. arXiv preprint arXiv:2503. 10437, 2025b.
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34: 15816-15829, 2021.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi. and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pp. (6689-266699 2024.
12

---

Published as a conference paper at ICLR 2026
Benlin Liu, Yuhao Dong, Yiqin Wang, Zixian Ma, Yansong Tang, Luming Tang, Yongming Rao, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondences boost apatial-empoors reasoning in multimodal language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3783-3792, 2025.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296-26306, 2024.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pp. 800-809. IEEE, 2024.
Ruipu Luo, Ziwang Zhao, Min Yang. Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability, arXiv preprinl urXiv:2306.07207, 2023.
Ruiyuan Lyu, Jingli Lin, Tai Wang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chenming Zhu, Dahua Lin, and Jiangmiao Pang. Mmscan: A multi-modal 3d scene dataset with hierarchical grounded language annotations. Advances in Neural Information Processing Systems, 37:50898- 50924, 2024.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Bartin-Brualle and Steven Seitz. Hypernerf: higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2/06.1
2021.
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Lue Van Gool, Markus Gross, and Alexander Sorkine-Homung. A benchmark dataset and evaluation methodology for video object segmentation. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 232, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pam Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. PMLR, 2021.
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projec- tion for monocular and multi-view general purppose 3d object detection. In Proceedings of the JEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2397-2406, 2022.
Johannes L. Schonberger and Jan-Michael Frahm. Stracturefrrommmotion revisited. In IEEE Conf.
Comput. Vis. Pattern Recog., pp. 4104-4113, 2016.
Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 06), volume 1, pp. 519-528. IEEE, 2006.
Jingtao Sun, Yaonan Wang, Mingtao Feng, Yulan Guo, Ajmal Mian, and Mike Zheng Shou. L4d- track: Language-to-4d modeling towards 6-dof tracking and shape reconstruction in 3d point cloud stream. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 21 1114611115, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.098 09288,, 2023b.
13

---

Published as a conference paper at ICLR 2026
Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videollmm: Sharpening fine- grained temporal grounding in video large language models. arXiv preprint arXiv:2410.03290, 2024.
Jianfeng Wang, Jianwei Yang, Xiaowei Wang. Lu Yuan, Lei Zhang, Yejin Choi, and Jianfeng Gao. Git: A generative image-to-text transformer for vision and language. In Advances in Neural Information Processing Systems, volume 35, pp. 18002-18014, 2022.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint wrXiv:2307.06942 2023a.
Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308.0876, 2023b.
Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-m Boosting milm capabilities in visual-based spatial intelligence. arXiv preprint arXiv:2505.23744 2025.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In IEEE Conf. Comput. Vis. Puttern Recog., pp. 20310-20320, 2024.
Haomiao Xiong, Yunzhi Zhuge, Jiawen Zhu, Lu Zhang, and Huchuan Lu. 3ur-llm: An end-to-end multimodal large language model for 3d scene understanding. IEEE Transactions on Multimedia, 2025.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang. Chung- Chhng Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint NKT::2309.1742. 9(1):1, 2023.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.0382. 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. /6852, 2024b.
Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15225-15236 2023.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.027/1, 2024c.
Duo Zheng, Shijia Huang, and Liwei Wang- Video-3d Ilm: Learning position-aware video represen- tation for 3d scene understanding. arXiv preprint arXiv:2412.0044 2024.
Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, and Chuang Gan. Lscenellm: Enhancing large 3d scene understanding using adaptive visual preferences. arXiv preprint arXiv:2412.01222, 2024.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 1851-1858, 2017.
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering Imms with bd-awerensss. arXiv preprint a7xiv:2409.18125, 2024a.
14

---

Published as a conference paper at ICLR 2026
Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision- laioonlanguuae understanding via promptable queries. In European Conference Computer Vision, pp. 188-206. Springer, 2024b.
Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In Eur. Conf. Comput. Vis., pp. 1-18. Springer, 2018.
15


Reviews:
[{"review_id": "3MSgsTf9a3", "reviewer": "Reviewer_gcu5", "summary": "This paper introduces LLaVA-4D, a framework that enhances MLLM for 4D spatiotemporal scene understanding with a new 4D feature embedding solution. The contribution is a spatiotemporal prompt that encodes 3D position and 1D time into a 4D coordinate embedding to better distinguish static backgrounds from dynamic objects. The model also disentangles visual features from multi-view videos into separate spatial and temporal components, which are then fused with the 4D prompts. Experiments are conducted on several 3D benchmarks and a new Chat4D benchmark.", "questions": "Please refer to my comments above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a new 4D spatial-temporal understanding framework for MLLMs. Some new designs for visual encoding and positional embedding are proposed. A new data recipe and a new benchmark for 4D understanding are constructed. \n\n- The new designs looks reasonable overall. Ablation studies show the effectiveness compared to baseline solutions.", "weaknesses": "- My main concern is the performance of the proposed method. A quite complex solution is proposed in the paper (with a new visual encoding solution and some new designs for embedding), but the results are not that impressive. Many recent methods, like Spatial MLLM [r1], 3UR-LLM [r2], and Coarse Correspondences [r3], that can achieve better performance on ScanQA are not compared or discussed. \n\n[r1] Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence\n\n[r2] 3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding, TMM\n\n[r3] COARSE CORRESPONDENCES Boost Spatial-Temporal Reasoning in Multimodal Language Model, CVPR 2025\n\n- How about the performance of LLaVA-4D on some new and more challenging benchmarks like VSI-Bench, which is specifically designed for 3D MLLM?\n\n- For baselines of the new Chat4D benchmark, I would recommend adding results of SoTA proprietary models like GPT5, Gemini-2.5-pro, and open-source generalist MLLM like Qwen3-VL for better references. It would also provide more comparsions and discussions of the new benchmarks compared recent 3D/4D MLLM benchmarks. It would also be helpful to highlight the value and unqiue properties of the new benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "suqeKFTFvO", "reviewer": "Reviewer_n3mW", "summary": "This paper proposes a novel vision-language multimodal model that supports dynamic scenes. By embedding 4D spatiotemporal cues, LMMs can simultaneously understand the spatial features of static backgrounds and the temporal features of dynamic objects. A 4D spatiotemporal cue fused with optical flow is designed, and a 4D vision-language dataset containing 879.1K samples is constructed. Based on this dataset, a three-stage training pipeline is designed to enhance the model's understanding capabilities.", "questions": "- What is the inital of w_p and beta? \n\n- The paper uses CLIP-ViT-L-336px as the visual encoder, but why not use a more suitable encoder for video (such as the Video Swin Transformer) was not chosen?\n\n- What are the advantages of LMMs frameworks over pure vision models in 4D tasks, and how do they compare to 4D Gaussians and VG4D?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The dataset makes a significant contribution, and the method can significantly improve the ability to understand 4D scenes.\n\n- The writting is clear and motivation is easy to understand.\n\n- The three-stage training avoids the convergence difficulties caused by directly training 4D features, enabling the model to smoothly transition from basic 2D/3D capabilities to 4D capabilities.", "weaknesses": "- Data quality: 4D data relies on GPT-4V to extract spatiotemporal information and GPT to generate instructions, which may contain annotation errors (such as coordinate deviation and timestamp misalignment), affecting the model fine-tuning effect. However, the paper does not evaluate the impact of annotation errors on performance.\n\n- Temporal coding: it relies solely on optical flow to estimate motion information. However, optical flow is prone to inaccurate estimation in fast-moving or occluded scenes, which may lead to deviations in the temporal characteristics of dynamic objects and affect the accuracy of the model's trajectory prediction for high-speed moving objects.\n\n- Lack of experiments: the contribution in TAcc of optical flow and the kind of temporal coding like frame rate-based vs. motion speed-based are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DQOdiKMpPG", "reviewer": "Reviewer_NHKE", "summary": "In this paper, the authors introduce LLaVA-4D, an advanced vision-language large model designed specifically for comprehensive 4D scene understanding. By leveraging the robust LLaVA multimodal framework, the authors’ primary innovation involves the incorporation of a novel spatiotemporal prompt. This innovation blends static 3D spatial perception with dynamic temporal dimension information, enhancing the model's capability to grasp scene dynamics. Specifically, LLaVA-4D encodes both 3D spatial coordinates and time into cohesive 4D coordinate embeddings, subsequently embedding these into visual features that have been temporally and spatially decoupled. This integration aligns these refined visual features with corresponding language embeddings, significantly augmenting the model's comprehension of dynamic scenes. Furthermore, to facilitate training and evaluation, the authors developed Chat4D, a comprehensive 4D vision-language dataset featuring detailed spatiotemporal coordinate annotations. Experimental findings underscore LLaVA-4D's superiority over existing 3D language-multimodal models (LMMs) in tackling 3D tasks and its clear leadership in 4D tasks, marking a significant advancement in 4D scene understanding.", "questions": "1.\tThe 4D benchmark in Chat4D is mainly based on 4D dynamic scene reconstruction datasets. The authors used GPT-4V and text-only GPT to generate a lot of global 4D descriptions. During this process, how did the authors assess and fix the hallucination problem of large models?\n2.\tWill the code and dataset be made open-source later?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe LLaVA-4D model is a groundbreaking innovation, endowing large language models with 4D understanding, transcending previous 3D-focused limitations.\n2.\tThe authors have a clear reason for decoupling spatiotemporal visual features, and ablation experiments also prove that this decoupling module can make the representation of spatiotemporal features stronger.\n3.\tThe Chat4D dataset, developed by the authors, addresses a critical void in 4D scene understanding for multimodal large language models. This contribution promises to enhance future research endeavors positively.", "weaknesses": "1.\tThe paper lacks clarity on LLaVA-4D's capabilities, failing to specify its maximum video frame rate and duration. Additionally, it omits critical details about the 4D data in the Chat4D dataset, such as the average video length. \n2.\tWhen comparing with state-of-the-art models, it seems that other models haven’t been trained or fine-tuned on Chat4D. (As shown in Table 1, they don’t even have the ability to output information related to the temporal dimension.) This makes it hard to directly show that LLaVA-4D is better than other models at understanding the temporal dimension.\n3.\tWhen testing the model’s ability to understand the temporal dimension, the study only uses TAcc as the evaluation metric. We need more evaluation methods—like action interval prediction and the corresponding measurement metrics, for example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)