4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time
Authors: Ziqiao Ma
Xuweiyi Chen
Shoubin Yu
Sai Bi
Kai Zhang
Chen Ziwen
Sihan Xu
Jianing Yang
Zexiang Xu
Kalyan Sunkavalli
Mohit Bansal
Joyce Chai
Hao Tan
Venue: NeurIPS 2025 poster ()
Keywords: Large Reconstruction Models
4D Reconstruction
Gaussian Splatting
Primary area: deep_learning
TL;DR: We introduce 4D-LRM, the first large-scale 4D reconstruction model that takes input from unconstrained views and timestamps and renders arbitrary novel view-time combinations.

Abstract: Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at some times to any view at any time? We provide an affirmative answer with 4D-LRM, the first large-scale 4D reconstruction model that takes input from unconstrained views and timestamps and renders arbitrary novel view-time combinations. Unlike prior 4D approaches, e.g., optimization-based, geometry-based, or generative, that struggle with efficiency, generalization, or faithfulness, 4D-LRM learns a unified space-time representation and directly predicts per-pixel 4D Gaussian primitives from posed image tokens across time, enabling fast, high-quality rendering at, in principle, infinite frame rate. Our results demonstrate that scaling spatiotemporal pretraining enables accurate and efficient 4D reconstruction. We show that 4D-LRM generalizes to novel objects, interpolates across time, and handles diverse camera setups. It reconstructs 24-frame sequences in one forward pass with less than 1.5 seconds on a single A100 GPU.

4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time
Ziqiao Ma
Xuweiyi Chen
Shoubin Yu1.3
Sai Bi
Kai Zhang'
Ziwen Chen 1.5
Sihan Xu²
Jianing Yang 1.2
Zexiang Xu
Kalyan Sunkavalli
Mohit Bansal
Joyce Chai Hao Tan
Adobe Research
University of Michigan
UNC Chapel Hill
Universityy of Virginia
SOregon State University
ht https: github.io/
Input Views
Render Any View At Any Time (<1.5
Ground Truth
A100)
0.00
GT
t=1
t= 4.25
t 8.25
t 14.75
t 21.00
GT
0.00
9.50
19.75
t= 23.00
GT
t=0
t=1
2=
22
0.75
5.50
9.25
17.00
Figure 1: Large Space-Time Reconstruction Model (4D-LRM) is data-driven 4D reconstruction model that takes sparse input views at any time and renders arbitrary novel view-time combinations.
Abstract
Can we scale 4D pretraining to learn general space-time representations that recon- struct an object from a few views at some times to any view at any time? We provide affirmative answer with 4D-LRM. the first large-scale 4D reconstruction model that takes input from unconstrained views and timestamps and renders arbitrary novel view-time combinations. Unlike prior 4D approaches, e.g., optimization- based, coometry-based. or generative, that struggle with efficiency, generalization, or faithfulness, 4D-LRM learns unified space-time representation and directly predicts per-pixel 4D Gaussian primitives from posed image tokens across time, enabling fast, high-quality rendering at, in principle, infinite frame rate. Our results demonstrate that scaling spatiotemporal pretraining enables accurate and efficient 4D reconstruction. We show that 4D-LRM generalizes to novel objects, interpolates across time, and handles diverse camera setups. It reconstructs 24-frame sequences in one forward pass with less than 1.5 seconds on a single A100 GPU.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Introduction
Reconstructing dynamic objects and scenes from video is a fundamental problem in computer vision full of challenges. The ability to accurately capture both spatial structure and temporal dynamics to build complete 4D representation from limited visual inputs, across varying views and time, would significantly advance applications, such as 4D asset generation [78, 53] for video games, film production, and AR/VR, as well as world modeling [32, 96] for embodied AI and robotics.
Prior work 4D modeling generally follows three directions, each shaped by different assumptions, target applications, or inherent limitations. The first direction is optinization-basel. These methods reconstruct space and time by optimizing per scene or object from multi-view videos [3, 28, 82). While these methods can produce -uuhliualtyy results, they require dense spatial and temporal sampling, limiting their practicality with sparse inputs. The second direction geometry- -based. Motivated by Geometry Transformers [71, 66]. they aim to estimate dynamic geometry and extract camera poses or depth maps directly from videos [90, 19, 70J. This line orthogonal to the previous line of work, as these methods are not intended for novel view or time synthesis and instead focus on per- frame geometry estimation [90]. The third direction is generation-
aiming to produce perceptually plausible 4D assets using visual generative models, particularly video diffusion models [73, 76, 78, 83). These methods require fewer inputs, but are still computationally expensive, sensitive to prompts [36], or tailored for single-view monocular videos (Figure 2a) [53, 78]. As Yao et al. [83] noted, generating dynamic 3D content from single-view video inherently ill illppsed for reconstruction due to motion ambiguity. Thus, these methods prioritize perceptual quality over faithful 4D reconstruction.
Recent advances in Large Reconstruction Models (LRMs) [24, 92, 98] offer a promising rendering- based alternative toward efficient and high-quality 3D reconstruction. Based on Transformer architectures. LRMs have shown strong performance on 3D reconstruction tasks by learning powerful priors over shape and appearance from large-scale 3D datasets. This also enables them to reconstruct detailed objects and scenes from only few posed images. However, existing LRMs are designed for static 3D objects or scenes. Although recent work has explored adapting them for 4D asset generation [53] and scene-level reconstruction with limited camera dynamics [81, 39], extending LRMs to general 4D reconstruction remains challenging, particularly when dealing with sparse multi-views and missing timestamps. We envision that an ideal 4D reconstruction model should be able to learn accurate spatiotemporal representations from limited set of input views few timestamps, enabling reconstruction at novel view-times by effectively sharing information across both viewpoint and time (Figure 2b). This motivates fundamental question for 4D modeling:
Can we scale 4D pretraining to learn a generic space-time representation that reconstructs an object from a few views at some time points, to any view at any time?
We introduce 4D-LRM, a Transformer-based large space-time reconstruction model for dynamic object reconstruction, trained in a data-driven manner. Inspired by 4D Gaussian Splatting (4DGS: [82]. 4D-LRM adopts a unified treatment of space and time, representing a dynamic object as a cloud of anisotropic 4D Gaussians. As illustrated in Figure 3. we patchify temporally posed input images into image tokens, which are processed by Transformer blocks. The model then directly regresses per-view, per- per-pixel 4D Gaussian primitives from contextualized multi-view tokens across time. These predicted 4DGS primitives enable fast, high-quality reconstruction and rendering from any viewpoint at any timestamp, with, in principle, an infinite frame rate. We train 4D-LRM on a curated subset of Objaverse4D [13], consisting of dynamic, articulated objects captured over time. The model scales effectively with more data and larger model size, and generalizes well to novel objects.
To the best of our knowledge, 4D-LRM is the first large-scale 4D reconstruction model that supports input from unconstrained posed views and timestamps, and renders arbitrary novel view-time combinations. With around 300M parameters, 4D-LRM achieves a high-quality reconstruction on Consistent4D [28] and the hold-out test set of Objaverse4D using only one input view per frame. reconstructs a 24-frame dynamic object in one forward pass with less than 1.5 seconds on a single A 1100 GPU. Compared to per frame 3D reconstruction, 4D-LRM exhibits strong and robust performance under diverse input camera configurations. We attribute this to 4D-LRM's 's ability to jointly model spatial and temporal contexts, effectively resolving motion ambiguities by sharing information across views and time. We further unlock its application to 4D asset generation, which surpasses baselines in both faithfulness and inference speed. Finally, we present detailed scaling behavior analyses for both training and inference, examining the scalability of different design choices and how inference

---

Time
Time
Camera
Camera
Input
Output
Input
Output
(a) Generative 4D Modeling.
(b) Generic 4D Reconstruction.
Figure 2: Comparison between previous generative 4D modeling methods (e.g., L4GM [53] and SV4D [78, 83)) and our goal of generic 4D reconstruction. Prior approaches take a single monocular video as input and use generative priors synthesize multi-view images for the first frame. In contrast, our objective is to reconstruct dynamic objects from any viewpoint at any timestamp.
performance varies with the number of input views. This highlights future directions in developing 4D-LRM variants that can handle longer contexts [98] and support test-time training [11].
Large Space-Time Reconstruction Model (4D-LRM)
Preliminary: 4D Gaussian Splatting (4DGS)
3D Gaussian Splatting. With 3D Gaussian Splatting (GS; [31]), static 3D scene can be represented as a cloud of amisotropic 3D Gaussians. Each Gaussian is in principle unbounded and, unless filtered, contributes point R via unnormalized density:
plalu, E) exp - (ar -- m)}
where (May My Ms) R is the mean and E E 3x3 is the covariance. The covariance is factorized Σ = RSST Rd, with S diag(s (5z, Sy: 8z) and R derived from unit quaternion q.
Pixel-Aligned Gaussians. GS-based reconstruction models [6, 60, 92] adopt pixel-aligned Gaussian rendering, where the center of each 3D Gaussian is computed from the ray distance and known camera parameters. Given ray origin ray direction ray,: and distance the center is μ = ray, + 8 ray Therefore, each Gaussian can be parameterized with dimanas = 12, including 3-channel RGB color, 3-channel scale, 4-channel rotation quaternion, 1-channel opacity, and I-channel ray distance.
4D Gaussian Splatting. When considering a dynamic scene, Yang et al. [82] observed that treating space and time independent, i.c., assuming (z y, 2/1) (Ix y, 2) for the i-th visible Gaussian, is undesirable. Instead, they extend the formulation of Kerbl et al. [31] to model dynamic scenes with unified treatment of spatial and temporal dimensions using coherent 4D Gaussian representation (4DGS; [82]). The mean of 4DGS is given by / (pz. (Pz. My; (4) IR,, which captures both the spatial and temporal centers. 4DGS parameterizes
covariance matrix Σ as 4D ellipsoid E = R RT. where S = diag(s,, Sy: S2, se) is diagonal scaling matrix and R E IR is a 4D rotation matrix. In 4D Euclidean space, R can be decomposed into a pair of left and right isoclinic rotations, each represented by quaternion. Together, general 4D Gaussian can be parameterized with dimanGS 20, including 3-channel RGB color, 4-channel scale, two 4-channel quaternions, 1-channel opacity, and the 4-channel space-time centers of order xyzt.
Sampling Conditional 3DGS from 4DGS. As is shown in Figure 3, the marginal probability (t(t) at time is one-dimension Gaussian A (t; p4, 4. The conditional 3DGS can be derived from the properties of the multivariate Gaussian with:
Mazzat P1. + E1.2, 14),
(2)
Σ Enyzlt E1:3.13
E1 5-15
This decomposition enables direct adaptation of the 3DGS tile-based rasterizer by first evaluating the marginal distribution (4), allowing for the accumulation of color and opacity over time. More details are available in Appendix B.2.
3

---

Image Tokens (V'NP"NP, D)
Free Gaussian Tokens [N. D)
t2?
Image (Linear)
NP'P,, 10*PS
Image Patchification
Transformer
Plücker Rays Frame Time
Self-Attention
MLP
Temporally Posed Images (V. W. 3+6+11
Input View-Times HV. W.
Unpatchify (Linear)
(Linear)
Pixel-Aligned Gaussians (VHHW, 20)
Gaussians [N. 20)
4D-LRM Overview
Merged Gaussians (V"H"W + N, 20)
Marginal
& 1
Sampling
P(t)
Input
Output
Render
Conditional
Novel View-Time
Gaussians
Figure 3: Overview of 4D-LRM. 4D-LRM adopts a unified treatment of space and time, representing dynamic object cloud of anisotropic 4D) Gaussians [82]. We train simple Transformer to regress 4D Gaussian primitives from a set of images with camera poses and timestamps. Each input image tokenized by patchifying the temporally posed frames. The resulting multi-view image tokens are concatenated in temporal order and passed through a series of transformer blocks. An optionalal set of N learnable free Gaussian tokens append the image tokens for greater generative flexibility.
2.2 Transformer-Bas Image-to-4DGS Decoder
Tokenizing Temporally Posed Images. As shown in Figure 3, the inputs to our model are images from arbitrary view-time combinations, denoted as {I, IR HRW x3] for j = 1,2, V, along with their corresponding camera intrinsic and extrinsic parameters. Here, H and W represent the image height and width, respectively. For pose conditioning, we compute Plücker ray coordinates [48] for each image, resulting in {P, E IR # x6), }. Instead of the standard canonical Plücker coordinates [ray d ray, rayal. we follow GS-LRM [92] and represent each ray as a direction plus its closest point to the origin [ray.. a ray ray, and pixel alignment. Temporal conditioning is provided by timestamp map (T, E IR Hxx x11. We channel-wise concatenate the RGB values, Plücker coordinates, and time to obtain per-view feature
(ray, o. ray a) rayal, which is suitable for pose-sensitive learning
ma Concat(I, (I, P,. T,) of 10 channels, enabling per-pixel pose and time conditioning, naturally serving as spatial and temporal embeddings to distinguish different patches. Therefore, we do not use additional positional embeddings. Following Vision Transformer (ViT) architectures [16]. we divide each pir-view feature map into non-overlapping patches of size PS² Each patch is flattened into a vector embedding of size 10 PS?, and then projected to token of dimension D (the Transformer width) using a linear layer.
Decoding Per- Gaussians with Transformer. We concatenate the image tokens and feed them through L layers of Transformer blocks. Each block follows a standard architecture consisting Pre- LayerNorm [2], multi-head self-attention [64], and MIP, all equipped with residual connections [23]. The output tokens are then unpatchified and decoded into pixel-aligned 4D Gaussian primitives using single linear layer. Given the same patch size, this results in V H x W Gaussians, each with dimADGS = 20. As prior GS-based LRMs, we adopt pixel-aligned Gaussian rendering. From each decoded 4D Gaussian parameter g E 20 split the 4-channel space-time vector (gx, gy E.. g.). retain the time 14 = gt, and normalize the xyz features to a scalar distance 6. Further details on 4DGS parameter initialization and differentiable rasterization are provided in Appendix B.
Optional Free Gaussians. Pixel-aligned Gaussians scale naturally with input resolution, making them well-suited for generalization to higher resolutions [92]. However, this design becomes suboptimal for very sparse views or setups with limited view coverage over motion, such those used in standard generative 4D modeling [53, 78]. To address this, we optionally introduce additional set of N

---

Time
Time
Time
Cam. Asimutt
180
Cam Asimutiy
30r
270
Cem. Aglmuth
-30"
270
Alternating Canonical Views (24in)
-187
Single View Video (27in)
Time
Time
Cam Azimutt
101
Cam Azimutiy
rand
800
rand
270
rand
Frame Interpolation (24in)
Two Rotating Cameras (24in)
Random View Each Frame (24in)
Figure 4: Different types of camera setups in evaluations: Alternating Canonical Views (4 camera
poses, 24/24 timestamps seen, 24 imput views in total); Frame Interpolation (4 camera poses, 12/24 timestamps seen, 24 input views in total); Two Rotating Cameras (24 camera poses, 24/24 timestamps seen, 24 input views in total); Single View Video [53, 78] (4 camera poses on the first frame plus a single view video for subsequent frames, 24/24 timestamps seen, 27 input views in total); Random Input Views (random poses, 24/24 timestamps seen, 24 input views in total).
learnable Gaussian tokens, concatenated with the image tokens. These tokens allow the model to generate freeform 4D Gaussian primitives. Unlike pixel-aligned Gaussians, these do not rely on pose time conditioning. Instead, use separate linear layer to decode the 4-channel space-time vector (gx: gy, 8z, g+) that directly defines the Gaussian center = (Jaz My 14 (42.14) after activation. We describe Section 3.1 how 4D-LRM can be fine-tuned for 4D asset generation with free Gaussians.
2.3 Training Objectives
During training, we render images at U supervision views using the predicted 4D Gaussians and minimize the image reconstruction loss. Let (Tş i' 1,...... U} denote the ground truth views and (Iş} the corresponding rendered images. The training loss combines Mean Squared Error (MSE) and Perceptual loss [9]:
U (SE I,) A Perceptual 13))
p==1
where À controls the weight the perceptual loss and is set to 0.5 empirically.
3 Experiments
3.1 Implementation Details
Training Data. To enable large-scale training, construct a 4D dataset derived from Objaverse [13], which provides subset of animated 3D assets. However, the raw dataset not directly suitable for 4D modeling: object motions often inconsistent, and the dataset contains duplicates and artifacts. To address this, we build upon the filtered subset curated by Diffusion4D [38], which removes static objects and unstable motion sequences, leading 32,000 animated objects. For each object, render a 24-frame video from diverse camera trajectories. We augment the dataset with 783,000 static 3D objects from Objaverse by treating each as 24-frame video, applying minor frame-by-frame displacements along a single random direction. More details are available in Appendix B.3.
Benchmark Data. We use the Consistent4D [28] objects for evaluation. Additionally, we hold out 56 challenging objects exhibiting more complex motion as an extended test set to support future benchmarking, For evaluation, and the first 24 frames of the Objaverse4D (Test) split. During training, we exclude 6 (out of the 7) Consistent4D test objects that also appear in Objaverse from our training set avoid data leakage.
re-render the first 48 (2 24) frames of the Consistent4D dataset
Curriculum Learning. We adopt a curriculum learning strategy to reduce computational cost. Specifically, we pretrain the model at resolution of 128 x 128 for 100,000 steps, and then continue 256 256 for an additional 20,000 steps. The continual pretraining stage uses the same model architecture, initialized with the same pre-trained weights, but processes more tokens due more
5

---

Table I: Breakdown evaluation of each camera setup on Consistent4D (Re-rendered) dataset. For each setup, we evaluate and average the score on 4 canonical views and 1 rrandolly sampled view.
We consider different Resolutions and model Initialization strategies and compare GS-LRM [92].
Init. Alter. Canonical Views
Frame Interpolation
Two Rotating Cameras
Random Input Views
Res. Models
PSNR LPIPS
SSIM
PSNR LPIPS SSIM PSNR LPIPS
SSIM
PSNR LPIPS SSIM
4D-LRM-Buse
No 29.233 0.047
0.961
29.194
0.047 0.961 25.014
0.071
0.926
25.788
0.081 0.926
128
4D-LRM-Larg,
No 30.274
0.038
0,969
30.260
0.038 0.969
25.904
0.061
0.935
26.525
0.070 0.934
4D-LRM-Large
Yes 31.023
0.031
0.974
30.917
0.031
0.973
28.703
0.042
0.959
28.789
0.049
0.957
SoM [69]
25.586
0.055
0.906
22.756
0.089
0.941
17.637
0.208
0.875
GS-LRM (Per Fr.)
19.327
0.097
0.902
20.037
0.094
0,935
16.826
0.212 0.801
GS-LRM (All in)
21.606
0,086
0.925
21.590
0.086
0,925
20.641
0.100
0,909
19.665
0.132 0.897
256
4D-LRM-Baas
No. 27.443
0.062
0.952
27.394
0.062
0.953
23.429
0.088
0.918
23.882 0.096
0.916
4D-LRM-Large
No 27.860
0.049
0,959
27.822
0.048
0.959 25.095
0.069
0.934
25.776
0.073 0.934
4D-LRM-Free
Yes 30.396
0.036
0.973
30.376
0.036
0.973 26.184
0.061
0.943
26.337
0.067
0.939
4D-LRM-Large
Yes 32.177
0.028
0,980
32.145
0.028
0.980
27.664 0.050
0.957
27.990
0.057
0.954
Input Views
4D-LRM Renderings
Ground Truth
Baselines
4D-LRMM
GT
GS-LRM
GS-LAM
V t- 0.75
V t.50
V
PSNR
PSNB: 24.70
PSNR
4D-LRM
GT
CS-LRM
GS-LAM
Alter. Canonical Views
t-
23
0.00
7.50
16.25
PSNR
:4D1RMM
GT
GS-LRMD MAA
GS-LAM
7 t
V -
23.00
PSNR.13.8
PSNR 12.05
4D-LRMM
GT
CS-LRM
GS-LAM
Alter. Canonical Views
Figure 5: Visual comparison with GS-LRM [92] using (a) all input views across time and (b) three random views from the same timestamp. 4D-LRM reconstructs novel view-time combinations by learning spatiotemporal representations from sparse inputs, outperforming per-frame reconstruction by effectively sharing information across both space and time.
pixel-aligned Gaussians in higher resolution. At each training step for object-level data, randomly sample 36 images (from 144 renderings over 24 frames) as a training example. From this set, independently select 12 input views and 24 supervision views, allowing overlap to improve convergence. Both pretraining stages are performed on 160 A 100 GPUs. The first stage adopted a per-GPU batch size of 16 and took approximately days, while the second stage adopted per-GPU batch size of 8 with an additional days. Unless otherwise specified, most training configurations follow GS-LRM [92]. Additional implementation and training details are provided in Appendix B.4.
Fine-tuning 4D-LRM for 4D Generation. In the pre-training stage, we do not include any free Gaussians. We fine-tune 4D-LRM with N 4096 free Gaussians for the 4D generation task similar to the setting of [53, 78]. For each training example over 24 frames, we select 4 canonical views at the initial frame and all views from single-view monocular video as input. We randomly sample 8 supervision views. This trained on 64 A100 GPUs with a per-GPU batch size for 16,000 steps.
3.2 Experiment Setups
Input Camera Setup. Since 4D-LRM supports arbitrary input views at any time, we define several camera setup configurations fore broad and systematic evaluation (see Figure 4 for illustrations):
Alternating Canonical Views. The input view alternates cyclically among four canonical directions (front, left, back, and right) across frames;

---

Table Breakdown evaluation of each camera setup on Objaverse4D (Test) dataset. For each setup, we evaluate and average the score on 4 canonical views and randomly sampled view. We consider
different Resolutions and model Initialization strategies and compare to GS-LRMM [92].
Init. Alter. Canonical Views
Frame Interpolation
Two Rotuting Caneras
Random Input Views
Res. Model
PSNR LPIPS
SSIM
PSNR LPIPS SSIM
PSNR LPIPS
SSIM
PSNR LPIPS SSIM
1D-LRM-Basc
No 27.374
0.068
0.937
27.287
0.068 0.987
25,496
0.075
0.924
24.174
0.112
0.893
128
4D-LRM-Large
No 28.489
0.055
0.949
28.440
0.055 0.949
26.312
0.064
0.934
25.023
0.096
0.905
4D-LRM-LLU -Large
Yes
29.251
0.042
0.9g9
29.169
0.043
0.958
28.676
0.043
0.957
27.586
0.064
0.939
GS-LRM (Per Frame)
18.79
0.164
0.854
19.729
0.143
0.911
18.300
0.176
0.845
GS-LRM (All in)
19.388
0.138
0.888
19.412
0.139
0.888
19.428
0.138
0.888
19.379
0.138
0.888
4D-LRM-Bass
No 25.806
0.085
0.928
25.711
0.085
0.929
23.974
0.085
0.924
22.409
0.125 0.889
256
4D-LRMM- L.M--Large
26.658
0.066
26.580
0.066
0.942
25.524
0.095
0.913
4D- ED-LRRM-Free
Yes
28.838
0.050
0.958
28.790
0.051
0,958
26.998
0.056
0.949
25. 25.267
0.082
0.923
4D-LRM-Large
Yos
30.094
0.041
0.967
30.028
0.042
0.967
27,810
0.049
0.957
26.694
0.072
0.939
V 3
15
T
Frame Interpolation
4D-LRM
PSMil:
PsNic
PSMil:
Psnil:
44
tt-
ft=31
t-
f
GT
Rand. View Rand. Frame
4D-LRM
t-24
1-24
t-25
1-27
PSNR
PSNS: 30.49
2S18:
PSNR 27.84
PSNB: 20.60
2S.8:
PSNB:27.61
PSNR-32.02
Figure 6: Qualitative examples of 4D-LRM taking views with missing frames, including Frame Interpolation (only half timestamps seen, 24 input views) and Random Views at Random Frames (24 input views from random views and times).
Frame Interpolation. Two canonical views are provided even wen-numbered frames, while all odd-numbered frames are omitted from the input. This setup is designed to evaluate the model's interpolation ability across time;
Two Rotating Cameras. Two virtual cameras rotate from front to back, one sweeping from the left and the other from the right. The left-rotating camera provides views at odd-numbered frames and the other for numbeedd frames, creating a complementary dual-view sequence;
Random Input Views. At each frame, single view randomly selected from the available camera positions. This setting introduces high variability that requires robustness to unstructured input.
Baselines. To the best of our knowledge, 4D-LRM is the first large-scale 4D reconstruction model that supports inputs from unconstrained views and timestamps, and enables rendering at arbitrary novel view-time combinations. The closest baselines are recent multi-view diffusion models [73, 76] and feedforward models [39] that support camera pose and time control, though these methods are not publicly available. For setups from moving cameras, compare 4D-LRM GS-LRMs run in per-frame fashion or directly with all views jointly. We also include Shape of Motion (SoM) [69], an optimization-bast method that models videos as causal, one-way motion trajectories. represents scene motion using compact SE(3) bases, enforcing consistent forward movement throughout dynamic scenes. Following guidelines, we manually segment the dynamic region and run 3,000 optimization iterations. In addition, we perform systematic ablation studies on 4D-LRM, evaluating two model scales: Large (300M parameters; 1024 hidden size, 24 layers, 16 attention heads) and Base (85M parameters; 768 hidden size, 12 layers. 12 attention heads). We also explore the effects of different Resolution settings and Initialization strategies, i.e., whether initializing the Transformer with weights from GS-LRM improves performance, using the same training setup and number of steps. Unless otherwise specified, we use 4D-L.RM-Large with initialization by default. More design choices are discussed with training-ting time scaling in Section 4 and Appendix B.4.
7

---

Table 3:
Comparison between 4D-LRM and GS- LRM under varying numbers of input views per frame (VPF). Rand.; randomly selected input views;
Table 4: Application 4D generation on the original Consistent4D benchmark. !ssnn ground truth multi-view reference the first frame the skyline.
Canon.: 4 canonical input views at each frame.
Input
VPF
Model
PSNR LPIPS SSIM
Objavense4D (Test)
PSNR LPIPS SSIM
Consisten:4D (Bo.)
Model
PSNR LPIPS SSIM
FVD
CLIP
Consistent-D [28]
0.160
1,133444
0.87
GS-LRM
18.738
0.165 0.852
17.201
0.166 0.846
DG4D [52]
0.160
0.87
4D-LRM
28.343 0.040 0.964
30.513 0.036 0.972
4Diffusion [89]
0.165
0.88
GS-LRM
22.252
0.105
0.898
22.425
0.097 0.904
Efficient4D [45]
0.130
0.92
4D-LRM
28.622
0.051
0.957
30.601 0.035
0,973
GaussianFlow [22]
4DGen [85]
0.130
0.89
3 GS-L.RM
24.381 0.083
0.917
24.661
0.079
0.924
STAG4D [88]
992.21
0.91
4D-LRM 28.212
0.954
30.554
0.035
0.972
SV4D [78]
0.129
677.68
0.93
GS-LRM 26.118 0.070
0.933
26.126
0.070
0.935
L4GM [53]
691.87
0.94
4D-LRM 27.940 0.053
0.953
30.445
0.034
0,972
4D-LRM-Luge
20.094
0.138
0,885
1,063,88
0.87
GS-LRM
28.710 0.047
0.962
30.067
0.038
0.967
4D-LRM-Free
23.777
0.117
0.916
677.58
0.94
Canon.
4D-LRM 27.839 0.055
0.952 30.850
0.039 0.968
4D-LRM- Free!
26.118
0.055
0.947
674.59
0.96
L4GM
LAGM
4D-LRM
SV4D
4D-LRM
4D-LRM
SV4D
Single Camera View
Single Camera View
t-O
b+1
t+2
t+3
L-0
1-3
Figure 7: Visual comparisons of AD-LRM((-Fres to generation based 4D models. For fair comparisons, initialize each model with the first frame with the ground truth multi-view images.
Metrics. We follow previous work to adopt PSNR [5], SSIM [72], LPIPS [93] metrics. For broad coverage of evaluation views, we evaluated and averaged the metrics on canonical views and I randomly sampled view at each frame.
3.3 Main Results: 4D Reconstruction
On both the re-rendered Consistent4D dataset (Table 1) and the Objaverse4D dataset (Table 2). 4D-LRM demonstrates strong performance across variety of camera configurations. Among the four tested setups, alternating canonical views provide the greatest view coverage over time and represent the least challenging configuration, under which 4D-LRM achieves PSNR scores exceeding 30. Notably, 4D-LRM remains robust even under more difficult settings, such as when half of the frames omitted, in configurations with limited spatiotemporal coverage, including and random input views. We also find that increasing model size leads to improved performance under identical data and training steps. Additionally, initializing the Transformer with weights from GS-LRM further enhances performance and accelerates convergence.
rotating cameras
The above experiments use 24 input views only one view per frame. Since GS-LRM is designed for sparse-view reconstruction, we further compare it with 4D-LRM under denser input conditions. Specifically,
24-frame motion sequence, corresponding to
evaluate both models using: (1) multiple randomly selected input views per frame, and (2) four canonical views per frame, with the task of rendering single randomly chosen novel view per frame. The results of these comparisons are presented in Table 3. We observe that GS-LRM performs comparably only when sufficient view coverage is available, as illustrated in Figure 5. In contrast, 4D-LRM consistently outperforms per-frame 3D reconstruction methods by leveraging spatial and temporal information jointly. As shown in Table 1, while SoM performs fair under structured settings such canonical or rotating views, it struggles in the random input view scenario due to its limited capacity to handle unconstrained spatiotemporal inputs. This highlights the advantage of 4D-LRM's learned, generalizable representation for arbitrary view-time combinations. Finally, qualitative results in Figure highlight 4D-LRM's ability generalize to novel objects and interpolate effectively over time, even when input frames are missing.
3.4 Application: 4D Generation.
We demonstrate that 4D-LRM can be extended 4D generation setup. Specifically, we chain 4D-LRM (fine-tuned with free Gaussians) with SV3D [65], and compare it against existing generation-

---

2 x100
2 x10
5.0 x10*
5.0 x10*
1
2.5
2.5
-1.0 -0.5
0.D
0.5
1.0
0
-0.5
0.0
0.5
1.0
0
0.2
0.4
0.8.0
0.2
0.4
Mean
T Mean
T Sigma
T Sigma
(a) al w/o interpolation.
(b) fie w/ interpolation.
(c) Me w/o interpolation.
(d) M. w/ interpolation.
Figure 8: We visualize the distributions of 14, and E, under A Alternating Canonical Views and Frame Interpolation setups on 24 frames with the same dynamic object.
0.90
0.30
24
0.85
0.25
wi HesPlane
a llangei 2
Align Free
4D4 base
SSIM
0.80
4D-LRMS-base
LPIPS
0.20
Thaget
Targat 2 Hixeilanee
Hextfane
0.75
0.15
20
lemp
18
2500
5000
10000
20000
40000
0.70
2500
5000
18800
20000
40000
0.10
2500
5000
18800
20000
40000 6 auiffbo
Training Steps
Training Steps
Training Steps
(a) PSNR vs. #Training Steps.
(b) SSIM vs. # #Training Steps.
(c) LPIPS vs. #Training Steps.
Figure 9: Training-time scaling curves. Tested Consistent4D (re-rendered). We compute the PSNR, SSIM and LPIPS for different number of different setups, with 4D-LRMMbase as the model.
based methods. When paired with diffusion model as generative prior, 4D-LRM outperforms baseline 4D generation approaches on the original Consistent4D benchmark. This improvement is due to 4D-LRM's is ability to produce much more faithful reconstructions of dynamic objects, even the presence of motion ambiguity. Since diffusion models introduce high variance, we provide a comparison in Figure 7, where evaluate 4D-LRM alongside other generative 4D models [53, 78] using ground-trute truth multi-view inputs from the first frame to ensure a fair comparison. Moreover, the core 4D-LRM model is efficient, requiring less than 1.5 seconds per forward pass; the primary computational bottleneck lies in the diffusion model.
4 Analyses and Discussions
Why Can 4D-LRM Interpolate Frames? 4D-LRM demonstrates strong frame interpolation capabilities, which aligns with its design: time is modeled as a continuous distribution rather than as discrete steps. To better understand this behavior, we analyze the 4DGS primitives predicted for the first 24 frames of the Guppie object in Consistent4D under two settings: Alternating Canonical Views (24 known timestamps) and Frame Interpolation (12 known timestamps). We visualize the distributions of the temporal mean μ4 = lu and variance M, Σ4,4 in Figure 10. Interestingly, when some timestamps are missing, 4D-LRM learns to reallocate certain Gaussians toward these missing regions, effectively filling the temporal gaps. Moreover, in the interpolation setting, the predicted 4DGS primitives tend have larger Σ,, increasing their temporal support. This allows each Gaussian to influence broader range of neighboring timestamps after sampling, improving interpolation quality and temporal coverage.
Training-Time Scaling. To understand how different design considerations affect the training efficiency, we provide the scaling behavior with the following configurations to 4D-L.RM-Ba
AD-LRM-Base Transformer with 768 hidden dimensions, 12 layers, and 12 attention heads, trained with 12 random input views and 12 random target views. No free Gaussians.
# Target x 2: Trained with 12 random input views and 24 random target views.
w/ Hexplane: Instead of unified space-time representation, Wu al. [75] proposed an alternative 4DGS representation with decomposed neural voxel encoding inspired by HexPlane [3].
w/ Temp Align: Similar to the idea of pixel-aligned Gaussians, we force per to the input frame time, reducing the parameterization to dimapos = 19.
w/ Free GS: Trained with N 1024 free Gaussian tokens from scratch.
We observe that increasing the number of target views slightly improves convergence speed, though at the cost of increased iteration time. Introducing free Gaussians from scratch does not significantly impact reconstruction quality but substantially slows down training. Additionally, we find that the 4DGS representation from [75] is less expressive than the unified space-time formulation proposed

---

0.055
30
0.968
0.055
38 28
0.960
g 0.004
0.952
0.040
27
0.944
0.035
6 12 18 48 96
12 1824 48 72
12 1824
48 72
Number of Input Views
Number Input Views
Number Input Views
(c) LPIPS vs. #luput View. Figure 10: Inference-time scaling eurves. Tested Consistent4D (re-rendered). We compute the PSNR, SSIM, and LPIPS for different numbers of randomly selected input views.
(a) PSNR vs. #Input View.
(b) SSIM vs. #Input View.
by [82], which informed our final design choice. We also note that enforcing strict temporal alignment degrades performance, whereas pixel alignment improves reconstruction quality. This supports carlier observation that 4D-LRM effectively redistributes Gaussians to unseen time intervals to handle sparse temporal supervision.
Inference-Time Scaling. Finally, we analyze inference-tines scaling as the number of input views varies. In terms of PSNR and SSIM, performance improves with more input views and peaks at 48, after which begins to decline slightly. We attribute this to two factors: (1) excessive Gaussians may overcrowd the 4D representation, reducing its quality, and (2) the Transformer struggles with very long input sequences. This observation suggests a promising future direction: designing 4D-LRM variants that can handle longer contexts with hybrid models [98] and incorporate test-time training [11, 94].
Related Work
Prior work on 4D modeling generally falls into three broad directions: optimization- geometry- based, and generation-
each shaped by different assumptions, data requirements, and target applications. The first direction is optimization- based, mostly covered in previous discussions of 4D) representations, where methods reconstruct dynamic scenes by optimizing per-object or per-scene representations using multi-view video [3, 28, 82, 69]. While capable of producing high-quality reconstructions, they are typically constrained by the need for dense spatial and temporal supervision. To improve generalization and reduce test-time cost, recent methods incorporate depth supervision or lightweight tuning [95, 62]. DyST [56] adopts a fully feed-forward approach, learning latent decomposition of content, dynamics, and pose from monocular videos via Transformer. However, it models space-time implicitly and remains limited in novel view synthesis quality. Although recent work has explored adapting LRMs for 4D asset generation [53] and scene-level reconstruction with limited input and target camera dynamics [81, 39), extending LRMs to general 4D reconstruction remains challenging, particularly when considering any target view any time from sparse multi-views and missing timestamps. The second direction is geometry- based, which aims to estimate dynamic scene geometry, such depth, flow, or camera motion, directly from input videos. Inspired by static sparse-view geometry methods like DUS [71], recent work has extended this paradigm to dynamic supervision or monocular approaches, they do not model the full spatiotemporal volume and are not intended for novel view or time synthesis. The third direction is generation- based, which leverages video generative models to synthesize perceptually plausible 4D assets. This includes both explicit 4D asset synthesis [53, 78, 83] and controllable video generation [36, 73, 76]. These methods reduce dependence on multi-view inputs by relying on learned priors, typically from large-scale video diffusion models. However, they are often computationally intensive at inference time, prompt-sensitive [36], and mostly limited to monocular inputs. Reconstructing faithful 4D geometry from a single-view video remains fundamentally ill-posed due to motion ambiguity [83]. Our goal is to learn generic space-time representation that reconstructs an object from a few views at some time points, to any view at any time. We include an expanded related work section in Appendix due to the page limitation.
scenes [90, 19, 70]. These methods often incorporate correspondence- depth priors to recover frame-wise geometry or trajectories. Unlike optimization-bant
6 Conclusion
This work introduces 4D-LRM, the first large-scale 4D reconstruction model capable of processing unconstrained views and timestamps to render arbitrary novel view-time combinations. By learning a unified spatiotemporal representation and directly predicting per-pixel 4D Gaussian primitives from posed image tokens over time, 4D-L.RM enables fast, high-quality rendering at, in principle, infinite frame rates.
10

---

References
[1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM, 54(10):105-112, 2011.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1m0iv:1607.064 2016.
[3] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130-141, 2023.
[4] Eric R Chan, Connor Z. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[5] Luen C Chan and Peter Whiteman. Hardwaare-oonstrainee hybrid coding of video imagery. IEEE Transactions on Aerospace and Electronic Systems, (1):71-84, 1983.
[6] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19457-19467,
2024.
[7] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang. Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124-14133, 2021.
[8] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference on computer vision, pages 333-350, 2022.
[9] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE international conference on computer vision, pages 1511-1520, 2017.
[10] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.0611 2016.
[11] Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025.
[12] Tri Dao. 2lashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2023.
[13] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13142-13153, 2023.
[14] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl A universe of 10m+ 3d objects. In Conference on Neural Information Processing Systems, 2024.
[15] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12882-12891, 2022.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition scale. In International Conference Learning Representations, 2020. 2020.

---

[17] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560. IEEE, 2022.
[18] Bardienus P Duisterhof, Zhao Mandi, Yunchao Yao, Jia-Wei Liu, Jenny Seidenschwarz, Mike Zheng Shou, Deva Ramanan, Shuran Song, Stan Birchfield, Bowen Wen, et al. Deformgs: Scene flow highly deformable scenes for deformable object manipulation. In The 16th International Workshop on the Algorithmic Foundations of Robotics, 2024.
[19] Haiwen Feng, Junyi Zhang, Qianqian Wang, Yufei Ye, Pengcheng Yu, Michael J Black, Trevor Darrell, and Angjoo Kanazawa, StÄrtrack: Simultaneous 4d reconstruction and tracking the world. arXiv preprint arXiv:2504.1311 1
[20] Sara Kridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12480 2023.
[21] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence, 32(8): 1376, 2009.
[22] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma. Le Chen, Danhang Tang, and Ulrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint arXiv:2403. / 12365, 2024.
[23] Kaiming He. Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[24] Yicong Hong. Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. In The Twelfth International Conference on Learning Representations, 2024.
[25] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In Special Interest Group on Computer Graphics and Interactive Techniques Conference, pages 1-11, 2024.
[26] Hanwen Jiang, Qixing Huang, and Georgios Pavlakos. Real3d: Scaling up large reconstruction models with real-world images. arXiv preprint arXiv:2406.0844 2024.
[27] Hanwen Jiang, Hao Tan, Peng Wang, Haian Jin, Yue Zhao, Sai Bi, Kai Zhang, Fujun Luan, Kalyan Sunkavalli, Qixing Huang, al. Rayzer: A self-supervised large view synthesis model. arXiv preprint 250.00500...02. 2025.
[28] Yanqin Jiang. Li Zhang, Jin Gao, Weiming Hu, and Yao Yao. Consistent4d: Consistent 360 dynamic object generation from monocular video. In The Twelfth International Conference on Learning Representations, 2024.
[29] Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: large view synthesis model with minimal 3d inductive bias. arXiv preprint arXiv:2410.17242. 2024.
[30] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 98365-18375. 2022.
[31] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 12(4):1-14, 2023.
[32] Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, and Angjoo Kanazawa. Robot see robot do: Imitating articulated object manipulation with monocular 4d reconstruction. In 8th Annual Conference on Robot Learning, 2024.
12

---

[33] Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. In European Conference on Computer Vision, pages 252-269, 2024.
[34] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: modular and hackable transformer modelling library. https://ithub.com/ffaee http:///....cccccoocseeeetttttta
2022.
[35] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71-91, 2024.
[36] Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai, Biao Zhang, Peter Wonka, and Bernard Ghanem. Vivid-zoo: Multi-view video generation with diffusion model. In Conference on Neural Information Processing Systems, pages 62189-62222, 2024.
[37] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. In The Twelfth International Conference on Learning Representations, 2024.
[38] Hanwen Liang. Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. In Conference on Neural Information Processing Systems, 2024.
[39] Hanxue Liang, Jiawei Ren, Ashkan Mirzaei, Antonio Torralba, Ziwei Liu, Igor Gilitschen- ski, Sanja Fidler, Cengiz Oztireli, Huan Ling, Zan Gojeic, et al. Feed-forward bullet-time reconstruction of dynamic scenes from monocular videos. arXiv preprint 242.:224, 2024.
[40] Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen-Phuoc, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2642-2652. IEEE, 2025.
[41] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In European Conference on Computer Vision, pages 210-227, 2022.
[42] Baorui Ma, Junsheng Zhou, Yu- Shen Liu, and Zhizhong Han. Towards better gradient consistency for neural signed distance functions via level set alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17724-1777 117344, 2023.
[43] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. In International Conference on Learning Representations, 2018.
[44] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, 2020.
[45] Zijie Pan, Zeyu Yang. Xiatian Zhu, and Li Zhang. Efficient4d: Fast dynamic 3d object generation from single-view video. arXiv preprint arXiv 2401.08742, 2024.
[46] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[47] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF international conference on computer vision, pages 5865-5874, 2021.
13

---

[48] Julius Plücker. Xvii. on new geometry of space. Philosophical Transactions of the Royal Society of London, (155))725-791, 1865.
[49] Mare Pollefeys, Luc Van Gool, Maarten Vergauwen, Frank Verbiest, Kurt Cornelis, Jan Tops. and Reinhard Koch. Visual modeling with hand-held camera. International Journal of Computer Vision, 59:207-232, 2004.
[50] Mare Pollefeys. David Nistér, J-M Frahm, Amir Akbarzadeh, Philippos Mordohai, Brian Clipp, Chris Engels, David Gallup. S-J Kim, Paul Merrell, et al. Detailed real-time urban 3d reconstruction from video. International Journal of Computer Vision, 78:1 143-1677, 2008
[51] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguse D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 13318-10377, 2021.
[52] Jiawei Ren, Liang Pan, Jiaxiang Tang. Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting, arXiv preprint arXiv:2312.7711 17142, 2023.
[53] Jiawei Ren, Cheng Xie, Ashkan Mirzaei, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, Huan Ling, et al. LAgm: Large 4d gaussian reconstruction model. In Conference on Neural Information Processing Systems, pages
2024.
[54] Johannes L Schonberger and Jan-Michael Frahm. fructure-from-mootee motion revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[55] Johannes I. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision, pages 501-518, 2016.
[56) Maximilian Seitzer, Sjoerd van Steenkiste, Thomas Kipf, Klaus Greff, and Mehdi S. M. Sajjadi. DyST: Towards dynamic neural scene representations on real-world videos. The Twelfth International Conference on Learning Representations, 2024.
[57] Noah Snavely, Steven Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. ACM Transactions on Graphics (TOG), 25(3):835 846,88555666688 2006.
[58] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Generalizable patch-based neural rendering. In European Conference on Computer Vision, pages 156-174, 2022.
[59] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8269 0279,8877,200 2022.
[60] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi- view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1-18, 2024.
[61] Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mv-dust3r+: Single-stage scene reconstruction from sparse views in 2 seconds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[62] Fengrui Tian, Shaoyi Du, and Yueqi Duan. Mononerf: Learning a generalizable dynamic radiance field from monocular videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17903-17913, 2023.
[63] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 12959-12970, 2021.
14

---

[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Conference on Neural Information Processing Systems, 2017.
[65] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. In European Conference on Computer Vision, pages 439-457 457, 2024.
[66] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
[67] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. In The Twelfth International Conference on Learning Representations, 2024.
[68] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Bai BBruala,, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image- based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
[69] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint arXiv:2407.13764 2024.
[70] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[71] Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20697-20709, 2024.
[72] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. 13(::: 600-612, 2004.
[73] Daniel Watson, Saurabh Saxena, Lala Li, Andrea Tagliasacchi. and David J Fleet. Controlling space and time with diffusion models. In The Thirteenth International Conference on Learning Representations, 2024.
[74] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint 120.222...1135,0002
[75] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie. Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20310-20322 20320, 2024.
[76] Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T Barron, and Aleksander Holynski. Cat4d: Create anything in 4d with multi-view video diffusion models. arXiv preprint ar XrXi:22411.186.3 2024.
[77] Desai Xie, Sai Bi, Zhixin Shu, Kai Zhang, Zexiang Xu, Yi Zhou, Soren Pirk, Arie Kaufman, Xin Sun, and Hao Tan. Lrm-zero: Training large reconstruction models with synthesized data. In The Thirty-eigh eighth Annual Conference on Neural Information Processing Systems, 2024.
[78] Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dy- namic 3d content generation with multi-frame and multi-view consistency. arXiv preprint 240.12400..74700, 2024.
15

---

[79] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In The Twelfth International Conference on Learning Representations, 2024.
[80] Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[81] Jiawei Yang, Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl, Peter Karkus, Danfei Xu, et al. Storm: Spatio-temporal reconstruction model for large-scale outdoor scenes. In The Thirteenth International Conference on Learning Representations, 2025.
[82] Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. The Twelfth International Conference on Learning Representations, 2024.
[83] Chun-Han Yao, Yiming Xie, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d 2.0: Enhancing spatio-temporal consistency in multi-view video diffusion for high-quality 4d generation. arXiv preprint arXiv:2503.16396 2025.
[84] Wang Yifan, Noam Aigerman, Vladimir G Kim, Siddhartha Chaudhuri, and Olga Sorkine- Hornung. Neural cages for detail-preserving 3d deformations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 75-83, 2020.
[85] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17235, 2023.
[86] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4578-4587, 2021.
[87] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-fred 3d gaussian splatting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 19447-19456, 2024.
[88] Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. In European Conference Computer Vision, pages (63,179,00 2024.
[89] Haiyu Zhang, Xinyuan Chen, Yaohui Wang. Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi- Multi-view video diffusion model for 4d generation. In Neural Information Processing Systems, volume 37, pages 15272-15295 2024.
[90] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deging Sun, and Ming- Hsuan Yang. Monst3r. A simple approach for estimating geometry in the presence of motion. In The Thirteenth International Conference on Learning Representations, 2025.
[91] Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields. In European Conference on Computer Vision, pages 717-733, 2022.
[92] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference on Computer Vision, pages 1-19, 2024.
[93] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
16

---

[94] Tianyuan Zhang, Sai Bi. Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William Freeman, and Hao Tan. Test-time training done right. arXiv preprint arXiv:2505.3388 2025.
[95] Xiaoming Zhao, R Alex Colburn, Fangchang Ma. Miguel Ángel Bautista, Joshua M. Susskind, and Alex Schwing. Pseudo seeuud-genrallzedd dynamic view synthesis from a video. In The Twelfth International Conference Learning Representations, 2024.
[96] Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: Learning 4d embodied world models. arXiv preprint arXiv:2504.20995, 2025.
[97] Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, and Javier Romero. Drivable 3d gaussian avatars, In International Conference on 3D Vision, 2025,
[98] Chen Ziwen, Hao Tan, Kai Zhang, Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, and Zexiang Xu. Long-Irm: Leng-sequence large reconstruction model for wide-coverage gaussian splats. arXiv preprint arXiv:2410. 12781, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the central goal scaling 4D pretraining to enable generalizable space- time reconstruction. The proposed 4D-LRM model directly addresses this goal, demonstrating high-quality view-time rendering, generalization across scenes and time, and fast reconstruction speed. The claims made well-supported by the model design, experiments, and results throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer Appendix D.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limiittions" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper provides empirical study of 4D representation. There is no proof for theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See Appendix B foe details.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should be possible for other researchers to have some path reproducing verifying the results.
19

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release code and data in order to support reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines bttiss:/nniss.c/ppblii
BuudessCodeSubmissioooiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines SttpaadweeeCeeeSuuiiiiooooiiiiie
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
We discuss the training and test details Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Majority of experiments are deterministic or based on fixed evaluation protocols (c.g., reconstruction from fixed views and times), making standard deviation or error bars less informative. Due computational cost and the nature of the 4D task, we focus representative results rather than repeated trials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification:
We discuss all computing requirements in the Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics https://eeurissccppubiffhhhiii hwt.c/ppbbiiittiiccccncceeess
Answer: [Yes]
Justification:
Our research involves 4D representation learning and does not involve human subjects, sensitive data, reptoyeent-related risks. We adhere all ethical standards regarding transparency, reproducibility in accordance with the NeurlPS Code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix E.
Guidelines:
The answer NA means that there is socictal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Our work does not involve the release of models or datasets with known high risk for misuse. Our research focuses on 4D representation learning which does not include any language models and generative models.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We acknowledge all external code and models used in this paper by properly citing their original sources, and we provide a detailed discussion of how each asset is used in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
22

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode. httrswiihhoodccccdataeeeee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We clearly discuss and document all new assets. We only used open-sourced data order to train our model and we will release our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve human subjects or any form crowdsourcing. All experiments conducted using publicly available datasets without human participation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve human subjects or study participants. Therefore, ethical risks or IRB approvals are applicable.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used as part of the core methodology, experimental design, or analysis in this research. Any LLM usage was limited to minor writing or formatting support and did not affect the scientific contributions of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original. or non-standard components.
Please refer to our LLM policy (ttts://Deeriisscccooocccceeeess for what should or should not be described.
24

---

A Related Work (Expanded)
3D Representations. At the heart of 3D reconstruction lies the choice of scene representation. Classical multi-view geometry methods [49, 57, 21, 1, 54] rely calibrated images and epipolar constraints to recover structure via ceneturre-fomm-otte
(SIM; [49]) and multi-view stereo (MVS: [50, 21, 55]. While robust under densely sampled views, these pipelines degrade in the presence of occlusion, textureless surfaces, or wide baselines. To improve generalization, recent geometry-basee models [71, 35] extend reconstruction to unposed and sparse view settings by leveraging large-scale pretraining. In parallel, neural optimization approaches fit implicit representations per scene, such as neural radiance fields (NeRF; [44, 4, 8, 15]) signed distance functions [46, 42]. In this work, adopt Gaussian Splatting (GS: [31, 25, 87]) as our 3D representation, which prioritizes efficiency and real-time performance through explieit point-based representations and offers faster rendering and competitive quality with minimal optimization per scene.
4D Representations. Modeling dynamic scenes in 4D has evolved from implicit, per-scene neural fields to more efficient, explicit spatio-temporal representations. Early methods [51, 47, 63] extend NeRF to dynamic scenes by conditioning on time or deformation fields, but they remain computationally heavy and lack real-time inference. A more practical shift arrives with HexPlane [3] and K-Planes [20], which decompose the 4D volume into planar factorizations. Concurrent with these NeRF-based approaches, dynamic Gaussian Splatting emerges as an explicit, high-performance framework for dynamic scenes [18, 33, 40]. D3GA [97] adapts this paradigm human avatars by embedding Gaussian splats into deformable tetrahedral cages [84], allowing anatomical consistency and real-time control via joint-driven pose signals. Notably, 4D-GS [75] represents dynamic scenes using a canonical set of 3D Gaussians combined with 4D boxel-based features. These voxel encodings, inspired by the spatio-temporal factorization of HexPlane [3], are decoded by lightweight MLP to predict per- Gaussian deformations across time. In contrast, we adapted 4DGS [82], which introduces a more elegant and unified formulation by directly parameterizing 4D Gaussians over space and time, allowing anisotropic deformation and smooth, time-varying appearance via 4D spherical harmonics.
Reconstruction Models. Recent advances in generalizable radiance field-based methods have achieved state-of-the-ar quality in novel view synthesis by leveraging NeRF--ttll volume rendering [86]. These approaches typically employ 3D-to-2D geometric projections sample per-view image features, using architectural priors such as epipolar feature sampling [68, 59, 58] or plane-swept cost volumes [7, 30, 41] inspired by MVS. In contrast, we explore simpler and more flexible design: a large Transformer-based model without explicit 3D inductive biases, which directly regresses Gaussian primitives. Parallel to radiance field approaches, separate line of research investigates feed- forwadd geometry-centric reconstruction models [61, 80], building upon DUS:3R [71] and leveraging large-scale training. This work aligns more closely with large reconstruction models (LRMs), which have recently emerged as unified framework for producing view-consistent 3D reconstructions. These models are trained on massive 3D datasets and use triplane-based NeRFs [37, 24, 79, 67, 26] or 3D Gaussian Splatting [92, 60, 77, 98] to encode strong priors over shape and appearance, enabling high-quality reconstruction from just few posed views. While early efforts have begun extending LRMs to generate 4D assets [53], we present the first LRM for general 4D reconstruction that can handle sparse multi-views and missing timestamps.
B 4D-LRM Implementation and Training Details
B.1 4DGS Parameterizing and Initialization
Given the decoded 4D Gaussian parameter g 20 we split it into (Bxyz E RR3 ge R. grgb R², Xycale,xy2 E 3 gacale,t E IR, Brotation,lett E Rd, grotation, E IR gopacity E IR).
Space and Distance. Given the ray origin ray,, direction ray.. and distance scalar ö, the pixel- aligned Gaussian center in space along the ray is computed as Mxyz = ray. + 8 raya- To derive 8 from the decoded 4D Gaussian primitive (gx: By. gz), we adopt an interpolation scheme bounded by two empirically determined depth limits, ònear and ò which define the permissible depth interval along each ray. This range constrains the model's s predictions to lie within a spatially valid and semantically meaningful region of 3DD space, consistent with the geometry captured during training.
gy.g2)
w sigmoid [[g. + gy + +g)/33, ö (1 w) 8near + w Star-
(4)
(5)
25

---

Following the setup in [92], we set δuear = 0.1 and ōtar = 4.5. The predicted Mxyz values are further clipped to the range 1-1,1|.
Scale and Opacity. We follow the default activation functions used in 3DGS to ensure the predicted scale and opacity values fall within valid ranges: all scales are mapped to I and opacity to (0, 1). Specifically, apply the exponential activation to scale, mapping real-valued inputs positive values, and the sigmoid activation to opacity. We observe similar training dynamics reported in [92], that the learned scale of 3D Gaussians become excessively large. In such cases, the Gaussian degenerates into a highly anisotropic distribution, resembling a thin stick or line stretched across space and time. This can lead unstable training dynamics, slow convergence, and temporal ghosting artifacts. To mitigate this, we apply constant biases to the Transformer's s output to shift the initialization, and we clip the predicted scales remain within reasonable range.
scale = min (exp (gscale.xz - 2.3), 0.3).
(6)
scale; min(exp exp (gcalee, 2.3), 1.0),
(7)
opacity o (gopacity - 2.0),
(8)
All hyperparameters are empirically chosen and primarily serve to stabilize training. We observe that model performance relatively insensitive to their specific values.
Rotation. We predict unnormalized quaternions and apply I.2 normalization to ensure they lie on the unit hypersphere, producing valid unit quaternions for rotation. This approach simplifies optimization, as the model can freely output real-valued 4D vectors while normalization guarantees valid rotations. Following [82], we use pair of qi (a, b, c, d) and gr (p, q, r, s) for the left and right unit quaternions, respectively. We use them to represent isotropic rotations in symmetric form. R can be constructed by:
b
d
b d
q
p
-F
R R(g)) =
C
a
r
8
p
[
c b
a
F
9
P
Spherical Harmonics RGB. A 3D Gaussian stores set of spherical harmonics (SH) coefficients to represent view-dependent color, along with scalar opacity value a. 4DGS extends this representation to enable both view-dependent appearance and temporal evolution, by incorporating time-variant extension of the SH basis. This allows the appearance of each Gaussian change smoothly both viewpoint and time. In our implementation, we directly interpret the model's output as the zero-order SH coefficients, following the convention used in 4DGS [82]. For simplicity, we do not include higher-order SH terms this work.
B.2 Differentiable Rasterization and Deferred Rendering
In the rendering process, given a pixel (u, v) in an image at time t, along with the camera's 's extrinsic matrix E and intrinsic matrix K, the pixel color I(u, n, t) is computed by blending the contributions of all visible conditional 3D Gaussians. We build on the tile-based rasterization pipeline introduced in 3DGS and 4DGS [31, 82], and adopt deferred backpropagation [91] during rendering reduce GPU memory consumption. We describe more details for completeness.
Filtering. At inference time, the final opacity of each conditional 3D Gaussian weighted by its temporal marginal p(t), which reflects its relevance at the rendered time step. To improve rendering efficiency and visual clarity, we apply two filtering strategies: (1) Gaussians with marginal probability p(t) < 0.05 are removed, and (2) Gaussians whose weighted opacity a < 0.05 are removed. These filters are applied only during inference. Applying them during training would prematurely eliminate potentially useful Gaussians, leading to degraded convergence or dead Gaussians that fail to get optimized in training.
Rasterization. These filtered Gaussians are projected onto the image plane and sorted in front-to-back order based on their depth. For each pixel, the final color is computed using alpha blending, where the contribution of each Gaussian is weighted by its projected 2D density pi(u, v, 1). its opacity on. and its view-dependent color c: cd t). Additionally, transmittance term II; [1 py(u,, v, t,y)),)
26

---

Algorithm Image-to-4DGS Pseudo Code.
Input dimensions:
batch size; v number views; h, mmage height and width
Input tensors:
images v, W, 3] # RGB image frames
frame_time [b, V, 1] # Frame timestamp per view
extrinsics [b, V,
Camera-to-worle (c2w) transformation matrices
intrinsics [b, V, 4] # Camera intrinsics (fx, fy, ex, ey)
Output tensors (4DGS parameters):
xyzt : [a, *, 41 4D Gasssian an centers (x, y, 2, t) rgb [b,
3] $ RGB color
acale : [b, *, 4] # Anisotropic Caussian scale (3D apace time)
rotation [b, " opecity [b,
Rotatien matrix
1] Opacity value
# Augment and patchify input for 4D representation x x_grid, y_grid neshgrid x) # [h, w] ray wi_ ray
compute cameraaraaaaaaeaaaaaaaaaaaaaaaae didddrrwwoldd ray_dir_word
y_grid, intrinsics) $ [b, V, h, = transform directions dir__air_cam extringics) $ [b, V, 3, h, wl
ray origin o_dot_d = dot dat_product paaayariiinn myy_di_werldd, dim=2) $ [b, M, 1, h, w]
ext=act_cameeat paaaeraaorigiinneaaeeaaaaa (et.ireetttittiiessss textrlnsiss) [b, v, 3, b,
nearest_pts ray_origin o_dor_d ray_dir_xorld $ [b, V, 3, h, M]
Concatenate and patchify augmented images into transformer input
concatenate(
[b, v, 3, h, wh, RGB scaled to [-1, 1] [b, v, 1, w], time scaled to [-1, 1] ray_dir_world, $ [b, V. 3, h, w] nearest_pts [b, V, 3, h, w]
tormalize_rgb (Lmagges))
normalize_t (immm__t_ee____
} Final: [b, v, 10, h, w]
x patchify (x, patch_size-9)
[b
num_patches, patch_dim)
Transformer
linear(x) i [b, v num num_patches, hidden_ding dim] X cransformer (x)) $ LayerNorm Transforner
x (gg out dim-2i)) [b, v h W, 20]
4DGS Parameterization
step Spli: tranaformer output into Individual 4DGS fields xyz, t, reb, scale_xyz, xyz, scale_t, retation_leffs rotation_righ,, opacity split (x, sizes=[3, 1, 3, 3, 1, 4, 4, 1I, dim=-1)
step 2:
Compute center position (xyz +) sigmoid (xorm(xy)))
Soft depth interpolation weight
delta near (1 - w) + far ww & Range interpolation (near, far] xyz ray pay_origia ray dir_dirworddd xyzt = concatenate (xyz, t) [b, v w, 4]
delta
center point
Step 3: Compute scale (clipped exp)
scale_xy xyz selppexp(scalee.x xyz 2.3), max-0.3) Spatial scale 2.3), max=1.0) àt Temporal scale
scale_t clip(exp scale concatenate (scale scale_t) $ [b, v w, 4]
Step 4: Normalize quaternions
[b, v w h 4] q1 normalize (sotation_left) # [b, v v h
r (oomaliieootttionnnF......
41
Step Construct rotation matrix R T((_1) R(gTT)
L banalleft_ggatemmoonnntteeee
# [b, v
w, 4, 4] M, 4,
buiid_right,__. wwwrighh__aatteneeeeee ias.rriiiimmmttiii____. 4, 41 rotation = matmal (L, R) [b,
[b, v
Step 6: Compute opacity
opacity si signoid (opacity 2.0) $ [b, v veh 8, 11
Final 4DGS outputs return xyzt, rgb, scale, rotation, opacity
models the amount of light that reaches the i-th Gaussian after being attenuated by all previous ones. This formulation enables differentiable, order-dependent compositing. Yang et al. [82] noted that (u, v t) can be factorized as the product of a conditional and a marginal probability at time t:
I(u, Σ p(ttp,uu t)
I
[1 (1)),
(10)
27

---

To compute the image-space density (u, 2/t), we start with the 4-channel space-time features in the order of xyzt, and compute the conditional 3DGS:
Mi,yyalt = M1,13 + 1,,,3 2 i,1,at - Mi,a),
(11)
D Myyy 1,13113 1,33 2
We approximate the projection of 3D Gaussian (Hu, Ei) using linearized perspective transfor- mation as is in [31]. The resulting 2D Gaussian is
Pu v) (H Eg,v),
(12)
where the mean and covariance are computed as Pl,uv = Proj( E, K)222 and Ei (JEE, E J 11,,,,, 1:2, with Proj(..-. :) denoting projection from world to image coordinates using extrinsic and intrinsic K, and the Jacobian the perspective projection Miayystt
B.3 Dataset Curation
To enable large-scale training, construct 4D dataset derived from Objaverse [13]. which provides a subset of animated 3D assets." However, the raw dataset is not directly suitable for 4D modeling: object motions often inconsistent, and the dataset contains duplicates and artifacts. We build upon the filtered subset curated by Diffusion4D [38], which removes static objects and unstable motion sequences. For each object, render 24-frame video from diverse camera trajectories. If the original animation has fewer frames, we pad by repeating the last frame. The views include four canonical static views (front, back, left, right), elevated moving trajectories, and randomized orbits at varying distances, similar to [92]. This design encourages robustness to both viewpoint and motion variation. To further ensure quality, we compute the maximum LI distance across frames for each sequence to select high-quality subset with sufficient but not overly aggressive motion. Our final dataset contains 3,000 high-quality animated objects (HQ4D) selected from 32,000 animated objects (4D). We augment the dataset with 783,000 static 3D objects from Objaverse by treating each as a 24-frame video, applying minor frame-by-frand displacements along a single random direction. During pretraining, we sample from HQ4D, 4D, and 3D with mixing ratio of 200:50:1.
B.4 Training Details
We keep most settings identical to GS-LRM [92] so we can initialize 4D-LRM training upon it. Below, we describe the details for completeness. We use patch size of 8 x for the image tokenizer. 4D-LRM-Large employs a 24-layer Transformer with a hidden dimension of 1024, 16 attention heads, and two-layer MLP with GeLU activation. 4D-LRM-Bass uses 12 layers with hidden dimension of 768 and 12 attention heads, sharing the same MI.P design. All Transformer blocks are equipped with Pre-Layer Normalization and residual connections. Additionally, Layer Normalization is applied after the patchifying linear layer and before the unpatchifying linear layer to stabilize training. To enable efficient training and inference, we adopt Flash-Attention v2 [12] via the xF ormers library [34], along with gradient checkpointing [10] and mixed-precisien training using the BF16 data type [43].
C
Additional Results
C.1 Evaluation on 3D Reconstruction
Table reports performance on the GSO dataset [17], comparing various models under two resolutions. Notably, when adapting 4D-LRM for static 3D reconstruction by setting all timestamps to zero, we observe modest drop in performance relative GS-LRM 256x256 resolution. Despite this, 4D-LRM still outperforms the LGM and many models evaluated at higher resolution. This suggests that the spatiotemporal representations learned by 4D-LRM remain effective for conventional 3D tasks, highlighting its versatility and robustness.
C.2 Failure Cases
We exclude Objaverse-XX XL [14] due license restrictions.
28

---

t-
P t-
t-
---11
t--12
Alter. Canonical Views
1-0
t-1
t-22
t-
Figure 11: A typical failure case as 4D-LRM sometimes struggles with non-linear motion trajectories. When an object follows a non-linear path, the structures cannot be efficiently captured by a single ellipsoidal Gaussian. As result, the model requires multiple Gaussians placed along the trajectory to approximate the motion, increasing complexity and often leading to artifacts if not properly aligned.
We provide typical failure case in Figure 11. 4D- LRM still struggles with challenging 4D reconstruc- tion scenarios involving self-occlusion and fast mo- tion, which often result temporal ghosting artifacts. These failures are primarily due to the limitations of both (1) under-training (the current model has not yet reached the saturation for scaling) as well as (2) the current Gaussian representation, which models ap- pearance and motion smooth, continuous functions. In the presence of rapid or discontinuous changes, such as objects moving in and out of occlusion undergoing abrupt non-rigid deformation, the model cannot accurately localize update the corresponding Gaussians in time. As result, outdated Gaussians persist across frames, leading to visually noticeable residuals and motion trails. 4D-LRM sometimes falls short in non-linear trajectories. The kernel density of Gaussian is ellipsoidal, defines mass aligned with principal directions. When an object follows a non-linear path, the optimal support is curved branched, not ellipsoidal. To approximate non-linear trajectory, the model needs more Gaussians along the curve.
Table 5: Performance on the GSO dataset [17] for 3D reconstruction. 4D-LRM is evaluated by collapsing frame times to 0.
Res. Model
PSNR LPIPS
SSIM
SparseNeus [41]
20.62
0.199
0.836
Triplane-LRMy [37]
26.54
0.064
0.893
512
Mesh-LRM [74]
27.93
0.081
0.925
GS-LRM [92]
30.52
0.050
0.952
LGM [60]
21.44
0.122
0.832
256
GS GS-LRM [92]
29.59
0.051
0,944
4D-LRM
27.35
0.061
0.929
C.3 Additional Qualitative Examples
We provide additional qualitative examples in Figure 12 and 13.
D Limitations
We highlight the following future directions:
Long Context, Issues such as limited resolution, short video duration, and occlusion are fundamentally challenges of memory and long-range dependencies in sequence modeling. Although 4D-LRM achieves high-quality reconstruction from sparse posed images, several limitations remain. First, cannot yet efficiently process hundreds of input images in single forward pass. Second, its maximum training resolution is 256 x 256, though it generalizes up to 512 x 512. Unlike GS-LRM, which was fine-tuned at 512 resolution, fine-tuning 4D-LRM at this scale is significantly more expensive, requiring approximately 75 seconds per training step. A promising direction for future work is to develop high-resolution 4D reconstruction models capable of handling hundreds of 1K or 2K resolution inputs. This will require fundamental architectural advances, such as hybrid models for long-context handling [98] and test-time training strategies [11, 94].
Removing 3D Inductive Bias. Currently, 4D-LRM relies on posed images and explicitly learns 4D Gaussian primitives for rendering. To scale up 4D representations from in-the-wild videos, future work should aim to remove strong 3D inductive biases. This includes learning to reconstruct from unposed images [67, 27], and designing architectures that forgo explicit 3D representations such as NeRF or 3DGS [29, 27, 94].
From Objects to Scenes. Currently, 4D-LRM is not trained at the scene level, the concept of "any view" is less well-defined, e.g., we cannot observe what lies behind walls. Although GS-LRM has shown that this architecture can scale to scene-level reconstruction, we currently lack
29

---

t
Alter. Canonical Views
4D-LAM
b-0
t+1
I=2
t+3
PSNI: 28.14
PSNI: 26.44
PSMR: 28.44
PSNR 27.18
PSMB: 29.71
PSNI: 27.79
PSNR: 20.1
PSNR: 26.48
V
3
t7=
v tt=22
t=
t
t
t=
Frame Interpolation
4D-LRM
ta0
tx
PSNP: 22.32
PSNR: 21.35
PSMR: 32.50
PSNR: 20.94
PSMR: 32.32
PSNR: 32.11
PSNR: 31.14
PSNR: 30.33
V
V
L-16
L-
4
L-
hocnstrucier
Two Rotating Cameras
t-O
t-2
t-p
ta5
t=7
PSN0:31.04
t-
Rand. View Each Frame
4D-LRM
1:0
1=1
1-2
PSNI: $1.18
PSNB: 30.08
PSMI:: 32.50
PSNie: $2.61
PSNR: 33
PSNN:: 20N00:30.00
PSNR:: 34.59
PSNP: 31.19
Figure 12: Qualitative examples of 4D-LRM under varying camera setups. We show the performance of 4D-LRM when taking input views captured with different camera configurations, demonstrating its robustness to diverse spatial arrangements and viewpoints.
license-compliant high-quality 4D scene dataset for training. Moreover, the data augmentation strategies used for object-level data do not directly transfer to scene-level setups. While we start to attempts this line with limited camera movement domain-specifie applications and limited input/target camera dynamics [81, 39], more future work should investigate both scalable 4D datasets and training methods for extending 4D-LRM scene-level reconstruction.
E
Broader Impact
While this paper does not explicitly address societal impact, the proposed 4D representation learning method has the potential to benefit range of downstream applications, including robotics, AR/VR, and digital content creation, by enabling more accurate and efficient modeling of dynamic scenes. However, the work primarily focuses technical contributions, and we do not identify immediate ethical concerns or direct social implications. As with any foundation model capable of detailed ppatial-temporal understanding, future applications should consider issues of privacy, surveillance, and potential misuse, especially if deployed in real-world environments involving human data.
F Acknowledgment.
The authors would like thank Ang Cao, Junyi Zhang, and Wenhao Chai for their helpful discussions and feedback.
30

---

4-13
D-16
Alter. Canonical Views
t:0
t=1
1-22
2-23
Alter. Canonical Views
L-0
1-1
1-22
t-23
4D-LRM
CT
4D-
4D LRM
GT
4D-LAM
GT
Alter. Canonical Views
5-0
b-23
18.00
27.76
30.11
PSNR
-R-L
4D-LAM
4D-LRM
4D-LRM
Alter. Canonical Views
t=
tm1
t:22
PSNEL 28.45
11.00
DEN8U27.91
12.00
PANE 27.8
L- 13.00
PSNE BLNE
14.00
4D-LRM
GT
4D-LRM
GT
4D-LAM
GT
4D-LRM GT
Alter. Canonical Views
t-24
2:25
1:46
1:47
PSNP: 29.72
32.00
29.08
L-30000
32,50
PSNR- 31.13
45.00
Figure 13: Additional frame interpolation examples. We insert 4x denser frames between Alternating Canonical Views as input.
31


Reviews:
[{"review_id": "4OrLkrCYak", "reviewer": "Reviewer_NLGT", "summary": "The paper proposes a novel, any-to-any formulation of the temporal scene reconstruction task, where given unconstrained posed views from different timestamps, the task is to render arbitrary novel view-time combinations (within the range of input space and time). The paper then introduces a large scale reconstruction model for this task. Importantly the paper shows that pre-training on the any-to-any reconstruction task and then fine-tuning for the 4D generation task, the method improves SOTA in an established benchmark by a large margin.", "questions": "-  Have you considered using latent gaussians, similar to e.g. in [70], to address the resolution issue?", "limitations": "Limitations are explicitly discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n-\tThe problem formulation of any-to-any space-time reconstruction is an interesting, and useful generalization of prior problem formulations. \n\n-\tThe proposed method is reasonable, the design choices are well motivated and explained. \n\n-\tThe demonstrated results, including on the Constistent4D benchmark, are impressive, although some aspects need to be clarified (see weaknesses). \n\n-\tThe evaluation is thorough and informative, including scalability with data.\n\n-\tThe paper is generally well written and easy to follow.\n\n-\tI appreciated the limitations section.\n\nWeaknesses\n\n-\tSome choices for the experimental setup need to be clarified. Line 150 mentions “we hold out 56 challenging objects with more complex motion”. What was the motivation for removing challenging cases? It makes me wonder whether the results were less favorable on these examples. It would be important to provide a strong justification, or include results on the full benchmark dataset, even if they are not as strong. \n\n-\tTraining and rendering times should be reported along rendering quality metrics.\n\n-\tThe compute requirements could be more clearly communicated in the main text, especially in Sect 4.2 it would be good to specific training times explicitly.\n\nI am willing to raise my score if the above weaknesses can be addressed.", "was_revised": true, "final_justification": "All my concerns have been addressed. I reviewed the concerns raised by other reviewers, and i believe they have been mostly addressed adequately. I raised my score accordingly.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. We are encouraged that the reviewer found the problem formulation of \"any-to-any space-time reconstruction\" to be an \"interesting, and useful generalization of prior problem formulations,\" and appreciated that \"the proposed method is reasonable\" and that \"the design choices are well motivated and explained.\" We are also glad that the \"evaluation is thorough and informative, including scalability with data.\" \n## Weakness 1: Misunderstanding about Challenging Cases\n\nSorry for the misunderstanding, and we apologize for the ambiguity in the wording. To clarify: the 56 challenging objects with more complex motion were **not excluded** from evaluation. Rather, they were **explicitly held out from pre-training** to serve as an extended test set for further evaluation to **avoid data leakage**. They are in addition to the re-rendered Consistent4D test set, which is less challenging compared to this test set. We agree this could be misread as exclusion and will revise the sentence for clarity.\n\n## Weakness 2: Training & Rendering Time\n\nWe agree that reporting training and inference efficiency alongside rendering quality is important. Due to space constraints, we briefly described runtime in Section 3.2 and included full details in the Appendix, but will surface this information more clearly in the camera-ready version. For clarity:\n\n- Training: 4D-LRM is trained in two stages on 160 A100 (40G VRAM) GPUs. Pretraining (128×128) takes ~5 days, and fine-tuning (256×256) takes an additional ~5 days. For 4D asset generation, fine-tuning with free Gaussians takes ~16K steps on 64 A100 GPUs.\n- Inference: Rendering a 24-frame sequence (which is the case for the main experiments) with 24 input views takes ~1.25s for 4D-LRM-Large and ~1.90s with free Gaussians, measured on a single A100 GPU averaged over 25 trials.\n\nWe will make these numbers more prominent in the main text.\n\n## Weakness 3: Training Time For Scaling Analysis\n\nWe appreciate the suggestion. In Section 4.2, we report training steps (rather than time) on the x-axis for scaling analysis as it provides a consistent and implementation-independent measure of progress, which is standard practice in prior work. Wall-clock time is less reliable due to fluctuations caused by memory fragmentation in mixed-precision buffers, which accumulate over time. For example, during 128×128 pretraining, the step time increases from ~3s initially to ~5.25s later, leading to a total of ~113 hours for 100K steps. While mitigations like periodic restarts exist, they introduce additional variability. For this reason, we report total training time (e.g., 5 days for each stage) in Section 3.2, and will clarify this rationale in the revised text. \n\n## Question: Latent Gaussian in STORM\n\nThank you for the thoughtful suggestion. We appreciate the reference to [70] (STORM) and the discussion around latent Gaussians.\n\nWe note that in Table 1 of [70], the latent Gaussian variant is slightly less performant than the full STORM model on the general Waymo Open Dataset. As the authors also acknowledge, the latent design is primarily introduced to improve performance in settings with **large novel-view extrapolation** or in **domain-specific applications** like human body modeling.\n\nTo empirically test this idea in our setting, we conducted a small-scale comparison by integrating the **patch-aligned Gaussian mechanism** (central to latent Gaussians in [70]) into **4D-LRM-base**, using a similar setup as Section 4.2. After 40K training steps, we observed:\n\n| Gaussian Alignment | PSNR   | LPIPS | SSIM  |\n| ------------------ | ------ | ----- | ----- |\n| Pixel (Ours)       | 25.913 | 0.114 | 0.867 |\n| Patch ([70])      | 24.135 | 0.154 | 0.832 |\n\nThese results suggest that our pixel-aligned Gaussians offer better fidelity in our reconstruction task. This also echoes the findings of Table 1 in [70]. That said, the concept of using a latent Gaussian backbone is promising, especially for **extrapolation-heavy or low-coverage regimes**. We consider this an exciting direction for future work."}, {"review_id": "YuKqRlwIPl", "reviewer": "Reviewer_3UsT", "summary": "This paper introduces a 4D reconstruction approach, which is able to recover s sequence of 3D-GS from a set of sparse views in temporal or spatial dimension. In general, the task setting is interesting and the visualized results on synthetic data is good. Also, the method is clearly presented and the whole architecture is not very surprising that it can work on current task setting. The major concern is about the generalization ability to real world. Previously, people show real world 3D reconstruction from single RGB image (such as LRM), where readers can clearly expect the reconstruction ability. However, authors didn't show any real-world results in current manuscript, and all used examples look very artificial and very similar to training data in Objaverse-4D. Hence, i am concerning if the proposed method can really generalize to real-world, which is more useful and more convincing (after using 160 GPUs). For now i suggest a borderline rejection, and i will consider to increase score if authors show more convincing results in rebuttal.", "questions": "My main question is regarding the promised code release of the manuscript. Authors don't have any claim in the abstract like normal papers, but promise the code and data released in the 5. Open access to data and code in the checklist. Could authors confirm if the original training / inference / data in the submission will be released? If yes, could you please add in the abstract of the revised version?\n\nAs mentioned in the summary and weakness sections, if authors can show real world 4D reconstruction with scenarios like moving human / moving pets reconstruction, the proposed methods will be more convincing, and i will increase the score.", "limitations": "Yes, limitations are discussed in the main paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- The manuscript has very clear writing which is easy to follow. Also the presented figures are intuitive and easy to understand.\n- As mentioned in the summary, authors didn't present any real world objects. The proposed task is interesting and attractive for such a real-world scenario: if i capture several photos of my dog from different perspective and it is moving during the capture, how does the method perform? I think it should still be feasible to perform since authors show the goldfish 4D reconstruction and i assume there are other animals in the training data. If authors can provide several of these results, i will consider to increase score. \n- Another application could be reconstructing moving human. Authors could leverage multi-view human performance capture data for training / evaluation.\n- The method requires camera poses for the input views. This is a big limitation as estimating camera pose from few-views captures of deforming objects is a very complex and unsolved problem. Authors should consider how to address this problem, and i will consider to increase score from boarder line to accept / strong accept.", "was_revised": true, "final_justification": "After authors explanation in the rebuttal. My primacy concern is addressed. I appreciate authors for promising the code release hope authors can include the additional visualization in the final version. I am also curious to see the future works regarding changing to pose-free input video setting.", "rebuttal": "We thank the reviewer for the detailed and constructive feedback. We are glad that the task setting was found to be \"interesting\" and that the visualized results were \"good.\" We appreciate the recognition that our method is \"clearly presented.\"\n\n## Question 1: Code Release\nWe confirm that a cleaned version of the training and inference code, along with data, will be released upon publication. We will also update the abstract in the revised version to explicitly reflect this release.\n\n## Weakness 1 / Question 2: Real Objects\n\nWe appreciate the suggestion and agree that real-world testing is important. To address this, we adapted the Plenoptic Video dataset, which contains 6 human motions in real indoor scenes. We preprocess them via center cropping and downsampling to 256×256 resolution with FOV 50 to match our training settings. For each sequence, we sample 24 frames across time and render to 6 novel views. We evaluate under two input settings: using raw frames vs. using SAM2 to segment the dynamic human.\n\n| Input Type | PSNR  | LPIPS | SSIM |\n|------------|-------|-------|------|\n| Raw        | 24.73 | 0.06  | 0.95 |\n| Segment    | 27.53 | 0.03  | 0.98 |\n\nWhile there is a performance drop compared to pure object-centric data (due to domain shift and background clutter), the results are non-trivial and demonstrate generalization, especially considering the small number of input views (24) without optimization over hundreds of posed images. Also, scene-level LRMs usually adopt different normalization strategies from the start.\n\nWe note that Objaverse already includes real scene-normalized data, and prior work (e.g., GSLRM) has shown that object-level training can generalize to simple scene-level layouts. While we have not trained a full scene-level 4D-LRM due to the lack of license-permissive 4D scene datasets, we view this as promising future work. The underlying design of 4D-LRM is extensible to more complex real-world scenarios.\n\n## Weakness 2: Camera Poses\nWe use posed RGB images as input because our primary goal is to investigate what 4D representation and training objectives enable scaling LRM-style reconstruction models. Thus, camera poses are a controlled variable, not a limitation of the concept. That said, adapting our model to work without known poses is a promising next direction. Recent methods like PF-LRM and RayZer demonstrate how to jointly predict camera poses and scene representation from unposed, uncalibrated multi-view inputs, using RGB-only supervision:\n\n- PF-LRM [1] predicts coarse geometry and solves for pose using a differentiable PnP module combined with self-attention across object and image tokens.\n- RayZer [2] recovers both camera parameters and scene representations end-to-end, without any ground-truth pose or geometry, using a transformer-based ray-aware architecture.\n\nWe believe pose-free approaches developed for 3D can transfer to the 4D setting, as even when part of the object deforms, static or rigid regions can serve as anchors for relative pose prediction and joint optimization. Due to the limited time in the rebuttal phase, we are unable to train such a model from scratch. Yet we expect that integrating such techniques into a pose-free variant of our 4D-LRM pipeline is feasible and would be a natural extension toward making the approach applicable to real-world RGB-only captures.\n\n[1] PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction. ICLR 2024.\n\n[2] RayZer: A Self-supervised Large View Synthesis Model. ICCV 2025."}, {"review_id": "1ZflxhqgPr", "reviewer": "Reviewer_eKZZ", "summary": "This paper studies how to reconstruct dynamic 3D objects from posed, discrete space-time RGB frames. The input setting is object-centric RGB images captured at arbitrary times and poses (in contrast to continuous views or monocular videos). The output is a 4D Gaussian representation (a true 4D distribution, not a deformable one).\n\nTo generate this set of 4D distributions, multiple input images are first encoded with RGB, pose Plücker coordinates, and a time ID, and then fed into a simple transformer. Both pixel-aligned Gaussians are queried using the input patches, and additional free queries are introduced to fill in unobserved or missing areas. This simple transformer is trained via regression on filtered Objaverse sequences. Some visual results and ablation studies are presented, primarily using photometric reconstruction metrics.", "questions": "Why is this paper novel? Why is this task hard/non-trivial given the prior art?\n\nIs the motion quality good?\n\nIs this a realistic problem setup?\n\nIs this research direction correct?", "limitations": "There is a section describing the limitations, including from object to scenes and resolutions, but compared to the above-mentioned questions, I believe these are minor points.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- At a high level, 4D is a fundamental direction for computer vision in the coming years.\n- The design of the network architecture is straightforward but reasonable.\n- Despite the very limited data source (Objaverse 4D data is indeed simple and constrained), some “scaling” behavior is observed in the experiments, at least indicating the network’s fitting dynamics for this specific transformer.\n\n---\n\n**Weaknesses:**\n\n- **Unrealistic problem setup and unclear novelty**: The paper emphasizes that reconstruction can happen at arbitrary poses and times. However, such settings are not reflective of real-world scenarios. In contrast, studying monocular video input and augmenting it to multiview is arguably a more “real” and practical problem. Therefore, the reviewer believes the premise of this paper may be fundamentally flawed. Moreover, there is no evidence that the model works on real data — how would one even obtain object-centric, multiview, discretely posed frames at different time steps in the real world? Using multiview camera setups? Given these concerns, the reviewer notes that prior work has explored reconstructing 4D deformable objects from monocular videos. Simply permuting input pairs and stacking them with a basic GS transformer does not constitute sufficient novelty.\n- **Possibly wrong research direction**: Although the paper benefits from industrial-level resources, the authors themselves may have noticed the poor quality of the data. The relative motion in the sequences is minimal, making this model closer to a 3D LRM rather than a true 4D model. This limitation is not overcome by “scaling.” The bottleneck in this area is not in network design or output representation, but in the data itself. Fitting Objaverse is straightforward because its motion distribution is small and simple. This is unlike 3D generation/reconstruction, where challenges lie in the high variance of detailed geometry and texture. For 4D objects, motion is the core difficulty; otherwise, the task reduces to standard 3D reconstruction. Thus, building a regression model on Objaverse for 4D appears trivial. The real research direction should focus on how to scale such models using self-supervision or rendering-based supervision on real data without known geometry-appearance.\n- **Conceptual flaws**: A key aspect of 4D modeling is leveraging temporal and camera smoothness — not discrete multiview observations at random poses. In contrast, monocular video is a more realistic and meaningful setup, rather than the “toy” setting used in this paper.\n- **Generating Gaussian Splatting is not fundamental**: Stepping back, if we consider this as a simple supervised regression problem on Objaverse, it’s unclear why GS is necessary as a representation. Hopefully, the choice isn't just due to its popularity. GS is suitable for rendering, but if ground-truth 4D geometry and appearance are available, why not supervise directly in 3D? Why is rendering needed at all if the goal is not to bridge toward real-world data?\n- **Lack of correspondence**: The representation uses  the real 4D Gaussians, not deformation-based dynamic Gaussians, so it fails to establish temporal correspondence — a critical, as important aspect of 4D reconstruction/generation alongside visual quality.\n- **Low output quality (especially under 160 A100s)**: The motion is relatively simple and small, largely constrained by the limited distribution of the Objaverse dataset.\n- **Incomplete literature review**: Many relevant works on 4D reconstruction and generation are missing; the paper lacks a comprehensive discussion of related literature.", "was_revised": true, "final_justification": "See my comment below", "rebuttal": "We thank the reviewer for the thoughtful feedback and for recognizing the high-level motivation of our work. We appreciate the acknowledgement that “4D is a fundamental direction for computer vision in the coming years,” and that our network architecture is “straightforward but reasonable.” We are also encouraged by the observation that scaling behavior was observed, suggesting the model’s fitting dynamics are promising for this transformer design. We address the reviewer’s comments and clarifications in detail below.\n\n## Weakness 1/3 and Question 1/3: Problem Setup / Novelty / Conceptual Flaws\n\n### Problem Setup / Conceptual Flaws\nReconstruction at arbitrary poses and times is not only our training objective, but it is also the most general form of the 4D reconstruction problem. We agree that monocular video (and augmenting it to multiview) is a (but not the only) practical and well-motivated setting, as argued in prior work like L4GM. However, **it is only a special case of our any-to-any formulation**, which **we include as a part of our evaluation** but go beyond. Therefore, our setting is not a “toy setting” but the superset of monocular input.\n\n### Real-World Applicability\nOur setup reflects a broader range of practical scenarios (see Section 3.3 and Figure 10 in the Appendix) studied in the vision community:\n- A few fixed cameras (e.g., surveillance cameras or motion capture),\n- A few continuous moving cameras (e.g., sports broadcasting, mobile robotics),\n- Single fixed monocular video augmented with multiview diffusion prior (e.g., 4D asset generation),\n- Sparse asynchronous recordings (e.g., crowd-sourced phone videos or event capture).\n\nThese settings all involve varying degrees of spatial and temporal sparsity, which our model is explicitly designed to handle.\n\n### Novelty\nWe strongly disagree with the characterization that our method merely permutes input pairs and stacks them in a basic GS transformer. A core novelty of 4D-LRM is the unified space-time treatment using anisotropic 4D Gaussian primitives, enabling continuous-time interpolation, which is a capability not supported by prior feedforward 4D reconstruction methods, and achieved here in a simple, elegant design.\n\n## Weakness 2/6 and Question 2/4: Research Direction / Motion\n\n### Modeling vs. Data\nWe disagree that the bottleneck lies solely in data and not in modeling. While high-quality dynamic 4D data is limited and remains a challenge, very few works have offered a clean and scalable solution for 4D reconstruction beyond generation tasks. Our work not only provides such a general solution, but also presents ablation studies analyzing which 4D representations are expressive and which model designs impact scaling behavior.\n\n### Data Quality\nAlthough Objaverse is not ideal, it is the only available large-scale dataset with open licenses, and all existing baselines (including L4GM and SV4D) are trained on it. Within this shared training data for fair comparison, we show that model design and representation choices still matter: 4D-LRM outperforms baselines and supports more flexible input configurations. \n\n### Motion\nFundamentally, what matters is not the magnitude of relative motion, but the frame rate at which the model receives input. We agree that motion is a key challenge in 4D modeling, and this is why, in evaluation, we manually hold out 56 challenging objects with more complex motion as the test set. Our experiments also show against the reviewer’s hypothesis that “the task reduces to standard 3D reconstruction” by showing 4D-LRM outperforms strong per-frame 3D models by effectively leveraging spatiotemporal correlations across frames. This highlights that our model learns and exploits non-trivial spatiotemporal dynamics. \n\n## Weakness 4: Generating Gaussian Splatting is not fundamental\nOur goal is to identify the most effective 4D representation for scaling large reconstruction models. We initially explored HexPlane-style representations but found them lacking. We adopted 4D Gaussian Splatting based on strong preliminary results, and we compared two different 4DGS parameterizations in Section 4.2, which informed our final design choice. At the time of submission, alternative approaches (e.g., diffusion-based or decoder-only models) had not been open-sourced or made reproducible, so GS was the most practical and validated representation for this setting.\n\nRegarding direct supervision in 3D, we respectfully discussed (line 26) recent feedforward methods that rely on monocular depth priors or explicit correspondences to recover per-frame geometry or trajectories. These methods are not designed for novel view synthesis, which is central to our task. In practice, ground-truth 3D or 4D geometry is even less available than multiview RGB, making rendering-based supervision a practical option for scaling RGB-only data.\n\n## Weakness 5: Correspondence \n\nThe notion of explicit temporal correspondence assumes discrete time steps. In contrast, our model treats time as a continuous dimension unified within the 4D Gaussian representation. Each 4D Gaussian can be interpreted as a local trajectory of a 3D Gaussian over continuous time, meaning temporal coherence is inherently embedded in the representation.\n\nWe acknowledge that tracking long-term correspondences—e.g., across complex trajectories—remains beyond the current model's scope, as these often require multiple Gaussians to represent extended motion. Dedicated tracking models like St4RTrack address this, but they are not designed for novel view-time synthesis, which is the focus of our work.\n\n## Weakness 7 / Formatting Concerns: Literature Review\n\nWe placed the related work section toward the end to maintain the narrative flow between the introduction and preliminaries. We are happy to move it earlier if preferred.\n\nThe current version was trimmed to meet the page limit, but we have updated it to the best of our knowledge at submission time. We appreciate the reviewer’s feedback and would welcome any specific suggestions on additional works to include. We will revise and expand the related work section accordingly in the camera-ready version.\n\nBelow is a compressed version:\n\n- - -\nPrior work on 4D modeling generally falls into three directions: optimization-based, geometry-based, and generation-based, each shaped by different assumptions and data requirements.\n\n- Optimization-based methods reconstruct dynamic scenes by optimizing object- or scene-level representations using multi-view videos [1–4]. While high-quality, they require dense spatiotemporal supervision. To improve generalization and test-time efficiency, recent work introduces depth supervision or lightweight tuning [5,6]. DyST [7] adopts a feedforward Transformer to decompose content, dynamics, and pose from monocular videos, but remains limited in novel view synthesis. Recent approaches explore adapting large reconstruction models (LRMs) to 4D generation [8–10], yet generalizing to arbitrary views and timestamps from sparse inputs remains challenging.\n- Geometry-based methods directly estimate dynamic scene geometry (e.g., depth, flow, camera motion) from videos [11–13]. Inspired by sparse-view geometry for static scenes [14], these approaches extend to dynamics using correspondence-based or monocular depth priors. However, they do not model the full spatiotemporal volume and are unsuitable for novel view/time synthesis.\n- Generation-based methods leverage video diffusion models for 4D synthesis, including explicit asset generation [8,15–17] and controllable video generation [18–20]. These models reduce dependency on multi-view inputs via learned priors but are computationally heavy, prompt-sensitive, and often limited to monocular inputs. Accurate 4D geometry from single-view videos remains fundamentally ill-posed [17].\nOur goal is to develop a generic space-time representation that reconstructs an object from a few views at sparse time points to any view at any time.\n\n[1] Cao, Hexplane: A fast representation for dynamic scenes, CVPR 2022.\n\n[2] Jiang, Consistent4D: Consistent 360° dynamic object generation from monocular video, ICLR 2024.\n\n[3] Yang, Real-time photorealistic dynamic scene representation and rendering with 4D Gaussian splatting, ICLR 2024.\n\n[4] Wang, Shape of motion: 4D reconstruction from a single video, arXiv 2024.\n\n[5] Zhao, Pseudo-generalized dynamic view synthesis from a video, ICLR 2024.\n\n[6] Tian, MonoNeRF: Learning a generalizable dynamic radiance field from monocular videos, ICCV 2023.\n\n[7] Seitzer, DyST: Towards dynamic neural scene representations on real-world videos, ICLR 2024.\n\n[8] Ren, L4GM: Large 4D Gaussian reconstruction model, NeurIPS 2024.\n\n[9] Yang, STORM: Spatio-temporal reconstruction model for large-scale outdoor scenes, ICLR 2025.\n\n[10] Liang, Feed-forward bullet-time reconstruction of dynamic scenes from monocular videos, arXiv 2024.\n\n[11] Feng, ST4RTrack: Simultaneous 4D reconstruction and tracking in the world, arXiv 2025.\n\n[12] Zhang, MonST3R: A simple approach for estimating geometry in the presence of motion, ICLR 2025.\n\n[13] Wang, Continuous 3D perception model with persistent state, CVPR 2025.\n\n[14] Wang, DUSt3R: Geometric 3D vision made easy, CVPR 2024.\n\n[15] Xie, SV4D: Dynamic 3D content generation with multi-frame and multi-view consistency, arXiv 2024.\n\n[16] Yao, SV4D 2.0: Enhancing spatio-temporal consistency in multi-view video diffusion for high-quality 4D generation, arXiv 2025.\n\n[17] Yao, SV4D 2.0: Enhancing spatio-temporal consistency in multi-view video diffusion for high-quality 4D generation, arXiv 2025.\n\n[18] Li, Vivid-Zoo: Multi-view video generation with diffusion model, NeurIPS 2024.\n\n[19] Watson, Controlling space and time with diffusion models, ICLR 2024.\n\n[20] Wu, CAT4D: Create anything in 4D with multi-view video diffusion models, arXiv 2024."}, {"review_id": "UyJyPsXOJq", "reviewer": "Reviewer_DFse", "summary": "The paper proposes a transformer-based model to reconstruct 4DGS from view-varied and time-varied input images. By enforcing unified modeling of space and time inherited from 4DGS, the proposed method can theoretically render arbitrary novel viewpoints and timestep given unconstrained input views and timesteps. Extensive experiments are done on testset from Objaverse and Consistent4D to validate the superior reconstruction performance of the proposed method. However, similar to many previous work trained on Objaverse, the proposed method is still limited to the foreground of object-level scenes.", "questions": "-\tIt is pretty interesting that the proposed method uses a unified representation of space and time on the output side (4DGS), however, it uses a modeling as temporal + image patches on the input side. I’m curious that will this “asymmetric” modelling brings pixel-aligned Gaussian features? It would be great to have a visualization of learned per-pixel feature for Gaussian primitives on the spatial-temporal 4D grids.\n-\tIn case sufficient views and timesteps are provided, how does the performance curves of optimization-based 4D reconstruction method look like? It would be great to have a similar observation like Figure 9 to see the upper bound.\n-\tAlthough the generative capability is supported through free Gaussians, will it have sampling diversity just like native generative models?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and organized with a clear illustration of key concept and difference to prior work (e.g., Figure 2)\n- As the very few papers on generic 4D reconstruction (M-in-N-out), the proposed method finds a principled but scalable solution with limited assumptions on the input.\n- Personally, I really like the design of introducing generative capacity by a set of optional learnable free Gaussian tokens. It works surprisingly well as shown in Table 4, especially in FVD.\n- The performance of the proposed method is extensively evaluated by quantitative results under different test settings and model specifics.\n\nWeaknesses:\n- As a 4D reconstruction method, qualitative results (videos) of comparisons and ablation studies are expected in the supplementary, which brings more direct evaluation of the reconstruction quality especially temporal smoothness.\n- The generalization to more general scenes is not validated. All the results are shown as object-level scenes. It somehow limits the usecases of the proposed method and makes it a bit unfair when compared with methods based on video diffusion models. The latter one can solve general 4D generation while the proposed method only focus on object-level scenes.", "was_revised": true, "final_justification": "All my concerns have been addressed. I will keep my original rating towards acceptance", "rebuttal": "We sincerely thank the reviewer for the encouraging and detailed feedback. We appreciate the recognition that the paper is \"well-written and organized,\" with a \"clear illustration of key concept and difference to prior work (e.g., Figure 2).\" We are glad the reviewer found our approach to be a \"principled but scalable solution with limited assumptions on the input,\" especially as one of the very few works tackling generic 4D reconstruction in an M-in-N-out setting.\n\n## Weakness 1: Qualitative Results (Videos)\nWe confirm that qualitative videos have already been included in the supplementary ZIP file. These provide visual comparisons of reconstruction quality and temporal smoothness. We did not include sample videos for every ablation study, as training all ablated variants to full convergence is computationally infeasible. However, we present a comprehensive set of quantitative ablations in Section 4.2, covering key design choices such as 4D representation, time encoding, initialization strategy, and number of target views. We also include training-time scaling curves to reflect convergence behavior and performance trends across variants.\n\n## Weakness 2: Generalization\n\nWe appreciate the suggestion and agree that real-world testing is important. To address this, we adapted the Plenoptic Video dataset, which contains 6 human motions in real indoor scenes. We preprocess them via center cropping and downsampling to 256×256 resolution with FOV 50 to match our training settings. For each sequence, we sample 24 frames across time and render to 6 novel views. We evaluate under two input settings: using raw frames vs. using SAM2 to segment the dynamic human.\n\n| Input Type | PSNR  | LPIPS | SSIM |\n|------------|-------|-------|------|\n| Raw        | 24.73 | 0.06  | 0.95 |\n| Segment    | 27.53 | 0.03  | 0.98 |\n\nWhile there is a performance drop compared to pure object-centric data (due to domain shift and background clutter), the results are non-trivial and demonstrate generalization, especially considering the small number of input views (24) without optimization.\n\nWe note that Objaverse already includes real scene-normalized data, and prior work (e.g., GSLRM) has shown that object-level training can generalize to simple scene-level layouts. While we have not trained a full scene-level 4D-LRM due to the lack of license-permissive 4D scene datasets, we view this as promising future work. The underlying design of 4D-LRM is extensible to more complex real-world scenarios.\n\n## Question 1: Gaussian Features\n\n### On pixel-aligned Gaussian features:\nYes, the predicted Gaussians are still pixel-aligned. During tokenization (Section 2.2), each pixel is augmented with Plücker coordinates (encoding camera pose) and timestamp, along with RGB. The spatial center of the Gaussian is derived from the corresponding ray direction and distance, see Appendix A.1 for details. While spatial alignment is preserved, temporal alignment is not enforced. As shown in our ablation (Section 4.2), forcing Gaussians to match input timestamps degrades performance.\n\n### On feature visualization:\nFigure 7 already provides a case study illustrating temporal reallocation behavior: when timestamps are missing, 4D-LRM learns to shift Gaussians toward these gaps and increases their temporal support. This behavior supports our design choice to preserve pixel-level spatial alignment while allowing flexible temporal modeling.\n\n## Question 2: Optimization-based 4D Reconstruction\n\nWe agree that optimization-based 4D reconstruction methods can serve as useful upper bounds or baselines. We provide two supporting experiments with 256 resolution on Consistent4D:\n\nShape of Motion (SoM) is included in our main experiment as an optimization-based baseline. SoM models 4D videos as causal, one-way trajectories, but it does not support frame insertion or multiple frames per timestamp without significant modification. We manually segment the dynamic region and run 3,000 optimization steps per object.\n\n| Model          | Alter. Canonical Views (PSNR / LPIPS / SSIM) | Two Rotating Cameras (PSNR / LPIPS / SSIM) | Random Input Views (PSNR / LPIPS / SSIM) |\n|----------------|----------------------------------------------|---------------------------------------------|------------------------------------------|\n| Shape-of-Motion | 25.586 / 0.055 / 0.906                       | 22.756 / 0.089 / 0.941                      | 17.637 / 0.208 / 0.875                   |\n| 4DLRM          | 32.177 / 0.028 / 0.980                       | 27.664 / 0.050 / 0.957                      | 27.990 / 0.057 / 0.954                   |\n\nFor upper-bound analysis similar to Figure 9, we perform direct optimization of 4D Gaussians (4DGS) from the full set of input frames. However, this setup becomes unstable when the number of views is low (especially when <48), with high sensitivity to initialization and hyperparameters. We report the best results achieved for each input size, but emphasize that this is not the intended setting and use case of 4DGS and serves only as an empirical reference.\n\n| Model | #Views | PSNR  | LPIPS | SSIM |\n|-------|--------|-------|-------|------|\n| 4DLRM | 24     | 30.51 | 0.04  | 0.97 |\n| 4DLRM | 48     | 30.60 | 0.04  | 0.97 |\n| 4DLRM | 96     | 30.45 | 0.03  | 0.97 |\n| 4DGS  | 48     | 25.02 | 0.08  | 0.92 |\n| 4DGS  | 96     | 30.64 | 0.03  | 0.98 |\n\n## Question 3: Sample Diversity\n\nUnfortunately no. As is also mentioned in the original LRM paper, Section 4.3.2, the model \"is deterministic and is likely producing averaged modes of the unseens.” While free Gaussians increase flexibility and generative capacity, they do not introduce sampling diversity by design. To enable diversity, a promising future direction is to integrate 4D-LRM into a probabilistic or diffusion-based pipeline (such as Turbo3D), where the deterministic backbone can be paired with latent sampling mechanisms. We consider this an exciting avenue for extending our framework. We will mark this in the limitations section in the next version."}]

Author rebuttal:

Meta-review: The paper received mixed ratings initially. After the rebuttal, three reviewers are positive and only one remains negative. During the discussion period, the negative reviewer raised many issues which also received very detailed responses from authors. The AC carefully read all responses and think the issues are well addressed. Regarding the negative reviewer did not reply again, the AC finally recommend acceptance.
Decision: Accept (poster)