L4GM: Large 4D Gaussian Reconstruction Model
Authors: Jiawei Ren
Kevin Xie
Ashkan Mirzaei
hanxue liang
Xiaohui Zeng
Karsten Kreis
Ziwei Liu
Antonio Torralba
Sanja Fidler
Seung Wook Kim
Huan Ling
Venue: NeurIPS 2024 poster (2024)
Keywords: 4D Reconstruction; 4D Generation
Primary area: generative_models
TL;DR: We present the first 4D Large Reconstruction Model that produces animated objects from a single-view video input -- in a single feed-forward pass that takes only a second.

Abstract: We present L4GM, the first 4D Large Reconstruction Model that produces animated objects from a single-view video input -- in a single feed-forward pass that takes only a second.
Key to our success is a novel dataset of multiview videos containing curated, rendered animated objects from Objaverse. This dataset depicts 44K diverse objects with 110K animations rendered in 48 viewpoints, resulting in 12M videos with a total of 300M frames. 
We keep our L4GM simple for scalability and build directly on top of LGM, a pretrained 3D Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview image input.
L4GM outputs a per-frame 3D Gaussian splat representation from video frames sampled at a low fps and then upsamples the representation to a higher fps to achieve temporal smoothness. We add temporal self-attention layers to the base LGM to help it learn consistency across time, and utilize a per-timestep multiview rendering loss to train the model. The representation is upsampled to a higher framerate by training an interpolation model which produces intermediate 3D Gaussian representations. 
We showcase that L4GM that is only trained on synthetic data generalizes well on in-the-wild videos, producing high quality animated 3D assets.

L4GM: Large 4D Gaussian Reconstruction Model
Jiawei Ren
Kevin Xie
Ashkan Mirzaei
Hanxue Liang 1.3
Xiaohui Zengl2
Karsten Kreis*
Ziwei Liu
Antonio Torralba'
Sanja Fidler12
Seung Wook Kim!
Huan Ling
NVVIDIA
University of Toronto
University of Cambridge
"MIT
SSLaab, Nanyang Technological University
Project page:
https://researchh nwwdiaiiiii... www.iaa...//aa///orroor.eteeea..
Figure 1: L4GM generates 4D objects from in-the-wild input videos.
Abstract
We present L4GM. the first 4D Large Reconstruction Model that produces animated objects from a single-view video input in single feed-forward pass that takes only a second. Key to our success a novel dataset of multiview videos containing curated, rendered animated objects from Objaverse. This dataset depicts 44K diverse objects with 110K animations rendered in viewpoints, resulting 12M videos with total of 300M frames. We keep our L4GM simple for scalability and build directly on top of LGM [49], a pretrained 3D Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview image input. L4GM outputs a per-frame 3D Gaussian Splatting representation from video frames sampled at a low fps and then upsamples the representation smoothness. We add temporal self-attention layers to the base LGM to help it learn consistency across time, and utilize a per-timestep multiview rendering loss to train the model. The representation is upsampled to higher framerate by training an interpolation model which produces intermediate 3D Gaussian representations. We showcase that L4GM that is only trained on synthetic data generalizes well on in-the-wild videos, producing high quality animated 3D assets.
higher fps to achieve temporal
Introduction
Animated 3D assets are essential bringing 3D virtual worlds to life. However, these animations time consuming to create the procedure involves rigging and skinning of objects, and crafting keyframes the animation all with minimal automation in tooling. The ability to generate animated 3D assets from widely available monocular videos simply from would be desirable capability for this application. This the goal of our work. Building more advanced 4D content editing tooling, the ultimate goal of this line of research, is out of scope for this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Past work on automatically generating animated 3D objects, which we refer to as 4D modeling in this paper, falls into different categories. The first line of work aims faithfully reconstruct 4D objects from multiview video data, and oftentimes requires many views across time to achieve high quality (29 S 36.. Such data expensive to collect which limits applicability. Another line of work instead relies on the power of video generative models. Most commonly, the video score distillation technique is used which optimizes a 4D representation, for example 3D deformation field, by receiving iterative feedback from the video generative model. Score distillation is known to be fragile (sensitive to prompts), and time consuming (hours per prompt) as oftentimes many iterations are needed to achieve high quality results [4 26 11
Recently, a promising method emerged for the task of single-image 3D reconstruction. This method leverages large scale synthetic and real datasets to train large transformer model, dubbed 3D Large Reconstruction Model (LRM)) [20 19]. to generate 3D objects represented as neural radiance fields from a single image in single forward pass - thus being extremely fast. We build on top of this idea to achieve fast and high quality 4D reconstruction.
We present L4GM, the first 4D Large Reconstruction Model FFigure 2). which aims to reconstruct sequence of 3D Gaussians [22] from a monocular video, in a feed-forward fashion. Key to our method, is a new large-scale dataset containing 12 million multiview videos of rendered animated 3D objects from Objaverse 1.0 [T]. Our model builds on top of LGM 49 pre-trained 3D Large Reconstruction Model that is trained to output 3D Gaussians from multiview images. We extend take sequence of frames as input and produce a 3D Gaussian representation for each frame. We add temporal self-attention layers between the frames in order to learn a temporally consistent 3D representation. We upsample the output to a higher fps by training an interpolation model that takes two consecutive 3D Gaussian representations and outputs a fixed set of in-betweens. L4GM is trained on our multiview video dataset with per-timestep image reconstruction losses by rendering the Gaussians in multiple views.
We showcase that although only trained on synthetic data, the model generalizes well to in-the-wild videos. e.g., videos generated by Sora [34] and real-world videos in AcitivityNet [12]. On the video-to-4D benchmark, we achieve state-of-the-art quality while being 100 to 1,000 times faster than other approaches. L4GM further enables fast video-to-4D generation in combination with
multiview generative model, e-g., ImageDrream 53
2 Related Work
2.1 Large 3D Reconstruction Models
Reconstructing 3D representations from posed images typically requires a lengthy optimization process. Some works have proposed to greatly speed this up by training neural networks to directly learn the full reconstruction task in way that generalizes to novel scenes [65
55
54
57). Recently,
LRM (20 was among the first to utilize large-scale multiview datasets including Objaverse [ to train transformer- based model for NeRF reconstruction. The resulting model exhibits better generalization and higher quality reconstruction of object-centric 3D shapes from sparse posed images in single model forward pass. Similar works have investigated changing the representation to Gaussian splatting 44 68), introducing architectural changes to support higher resolution [61 and extending the approach to 3D scenes 76. Methods such LRM can generalize input images supplied from sampling multiview diffusion models which enables fast 3D generation 24].
2.2 Video-to-4D Reconstruction
Recent works have made impressive progress in reconstructing dynamic 3D representations from multiview video inputs [28 (2]] However, for single-view video inputs, the problem of dynamic reconstruction becomes ill-posed and requires hand-crafted or data-driven priors. Good results have been achieved when targetting specific domains using shape templates [62]. Template- ffre methods typically require accurate depth inputs and cannot fill in portions of the object that are occluded or not visible [14 [60]
Recently, a few works have attempted to extend feed-forward generalizeable novel view synthesis to the challenging setting of dynamic monocular videos. DYST 43] introduces the synthetic 4D Dyso dataset which they use to train transformer for generalizeable dynamic novel view synthesis

---

Generated Multivew Images J4
Gaussian Sequence
Copy
ResBlock
[RTS]ONE STTpppiKK
Cross-Veew Senf Attention
t=2
Temporal Seit Attention
LBPaC on (8
E R vy w'c
t-T
Monocular Video I
Figure 2: L4GM. The overall model architecture of L4GM. Our model takes single-view video and single-time step multiview images as input, and outputs a set of 4D Gaussians. It adopts a U-Net architecture and uses cross view self-attention for view consistency and temporal cross-time self-attention for temporal consistency.
but their model lacks explicit 3D representation of the scene and camera. PGDVS (70 extends the generalizeable 3D NVS model GNT [54] to dynamic scenes but relies on consistent depth maps obtained through per-scene optimization. Other methods [50] 4 leverage pretraining but still require some amount of test-time finetuning to achieve acceptable novel view synthesis. Several works tackle
generalizerable human shape reconstruction but rely on human template meshes [31 23
2.3 Text-To-4D Generation
Dreamfusion [35] introduced the score distillation framework for text-to-3D shape generation. Such per-object optimization methods have also been extended the 4D domain 47 where they leverage video diffusion models 46 3 30). Some methods combine the guidance from video diffusion models with multiview (45) and Langle-image diffusion models 41 boost 3D consistency and individual
frame quality 772 26)) and they utilize different forms of score distillation 25 66 [56].
By utilizing image conditional diffusion priors such Zero123 27 and ImageDream 53 as guid- ance, this approach has also been applied to the image-conditional setting 71 and video-conditionall 4D generation setting by Consistent4D (21) and DreamGaussinnDD
39 GaussianFlow [15] intro- duces a Gaussian dynamics based representation which supports them to add optical flow estimation additional regularization to SDS. STAG4D (67 and 4DGen [64] use diffusion priors sample additional pseudo-labels from "anchor" views to expand the set of reference images for image- conditional SDS and as direct photometric reconstruction loss.
To accelerate the generation process, some works eschew the use of score distillation guidance altogether. For reconstruction from monocular video input, Efficient4D [33] and Diffusion? [63] utilize a two stage approach. In the first stage they craft schemes for sampling multiview videos conditioned on the input video. Standard optimization-basee However, this optimization process can still take on the order of tens of minutes. In this work we directly train feed forward 4D reconstruction model instead.
reconstruction is then used for stage 2.
3 Background
Our L4GM builds on the success of single-image 3D reconstruction models 20 19 49 specifically
the Large Multi-View Gaussian Model (LGM) [49]. LGM accepts a set of multiview images of an object and directly outputs a 3D reconstruction of the object, represented by set of Gaussian ellipsoids P. Each Gaussian ellipsoid is represented by 14 parameters, including a center z E R3, scaling factor s E R quaternion rotation q E R4, an opacity E R. and color feature E 3 The multiview images J {JeYYn are taken from V camera poses O = {O.1v f--1- These camera poses are encoded as image embeddings via Plücker ray embeddings and concatenated the RGB channels of the multiview images as input to the model. They are fed through an asymmetric U-Net 42 yielding V 14-channel image feature maps, where each of the pixels will be interpreted as the parameters of a 3D Gaussian.
3

---

When only single input image is given, an image- conge-coniiiinal multiview diffusion model such as ImageDream [53] first used to generate plausible completions for the missing multiview images. These generated views are then fed to I.GM for reconstruction.
4
Our Method
Given a monocular video of a dynamic object, denoted as I {4))- where T video length, our objective is to rapidly reconstruct accurate 4D representation of the object. Our approach is grounded two conceptually simple yet impactful insights.
Our inspiration stems from video diffusion models. Recent advances in video generation have highlighted the benefits of first pretraining on image data and then extending and finetuning the model on video datasets to effectively model temporal consistency [3] 16 59 []. Similarly, recognizing the scareity of 4D data, we want to leverage pre-trained Large Multi- View Gaussian Model (L.MM) 49 that operates on images and has been extensively trained on a large-scale 3D dataset of static objects. This strategy leverages the robustness of pre-trained models to effectively train 4D reconstruction model with limited data.
Secondly, in constrast to most existing methods for 4D reconstruction, we found that utilizing a single set of multiview images at the initial timestep sufficient. We can obtain these multiview images casily by leveraging multiview image diffusion models to expand the first frame of the view. By adding temporal self-attention layers, our model capitalizes on the initial multiview input by propagating and adapting this information across subsequent timesteps (subsection 4.2) This approach significantly reduces the computational complexity and challenges typically associated with generating consistent multiview videos, while still enhancing the quality of the reconstruction, as our results demonstrate.
28
29
that are required to use multiview videos
Thus, in the following, we introduce L4GM, a model that processes a monocular video to output set of 3D Gaussians for each timestep, denoted by P = {Pr)/- where each P; is a set of 3D Gaussians time I. L4GM is extension of pretrained 3D LGM, enhanced with temporal self-attention layers for dynamic modeling. We generate V multiview images based on the first frame of the input monocular video. These generated views, along with the input video, are fed into L4GM to reconstruct the entire 4D sequence. We also explore further finetuning L4GM into a 4D interpolation model, allowing us to generate 4D scenes at a higher FPS than the monocular input video, offering smoother and more detailed motion dynamics within the 4D reconstructions.
4.1 Generate Multiview Images with ImageDream
Similar to the single-image scenario of LGM, we use ImageDream 53 to generate four orthogonal views conditioned on the initial frame I,. We denote J1 the set of generated multiview images taken from camera poses at the initial time step t = 1. We would like our generated multiview images contain three extra views that are orthogonal the input first frame of the original video.
However, often none of the viewing angles of the generated multiview images match the input frame 41. An example be found in Appendix Figure 8, To address this, we first use the 3D LGM to reconstruct an initial set of 3D Gaussians, Pinits from the generated multiview images, and render this reconstruction from views that are orthogonal to / as desired. We provide exact details in Amppeddix
4.2 Turning the 3D LGM into 4D Reconstruction Model
Model Architecture. We adopt the asymmetric U-Net structure from the pretrained LGM as backbone. To align with LGM's input specifications, we replicate the generated multiview images in Jr at 1 (except I1) across all other time steps to construct T V grid (see left side of Figure for an example). For simplicity, we assume the camera in the reference monocular video is static and only the object is moving so we also copy the camera poses O (except O1) across time steps. Similar to LGM,, these poses are then embedded using Plücker ray embeddings 48).. We concatenate this camera embedding with the RGB channels of the input images. The concatenated inputs are reshaped into the format (B T v) H W and fed into the asymmetric U-Net, where B is batch size.

---

As illustrated in the middle section of Fig 2 each U-Net block within L4GM consists of multiple residual blocks [18], followed by cross-view self-attention [52] layers. To maintain temporal consis- tency across different timestamps, we introduce a new lemporal self-attention layer following each cross-view self-attention layer. These temporal self-attention layers treat the view axis V as batch of independent videos (by transferring the view axis into the batch dimension). After processing, the data is reshaped back to its original configuration. In einops [40] notation, this process looks as:
rearrange (B T V) H W C (B H C)
X x + TempSelfAttn(x
(2)
rearrange(x, (B V) (T H) C (B
where x is the feature, B C are batch size, height, width, and the number of channels.
The output of the U-Net consists of 14-channel feature maps with shape B x X Hau x Won x 14. Each of the 1 14 units is treated the set of parameters of per-pixel Gaussian. We concatenate these Gaussians along the view dimension V to form single set of Gaussians for each timestamp, resulting in T sets of 3D Gaussians, {P.) This collection forms our final 4D representation.
Loss Functions. Besides the input camera poses O, we select another set of camera poses Osup for multiview supervision. We train the model with simple reconstruction objective the video rendering of the output 4D representations from camera poses O U Osup. as detailed in Appendix B
4.3 Autoregressive Reconstruction and 4D Interpolation
In practice, one may want to apply L4GM on long videos and obtain temporally smooth 4D out- puts. To this end, L4GM also enables autoregressive reconstruction which processes videos in an autoregressive fashion, one chunk of T frames after another. We additionally train 4D Interpolation Model to upsample the 4D representation to higher fps. Details can be found in Section A
left). Our base model is designed to accept monocular video of fixed length T. For long videos that exceed T, we partition the video into chunks of size T which we process sequentially. We first apply L4GM to the initial T frames to generate the first set T Gaussians. Subsequently, instead of generating multiview images for the first frame of the next chunk using a multiview diffusion model, we render the last set of Gaussians from four orthogonal angles to obtain new multiview images. These newly rendered multiview images, along with the next set of video frames, are then used to generate the next set of Gaussians. The process is repeated until all frames of a long video have been reconstructed. We empirically observe that such an autoregressive reconstruction method can be repeated more than 10 times without significant drop in quality.
Autoregressive Reconstruction (Figure 3
4D Interpolation Model (Figure 3] right). As our model does not track Gaussians across frames, directly interpolating Gaussian trajectories not feasible (2 58 Hence, we develop an interpolation model that operates in 4D, fine-tuned on top of L4GM. Similar interpolation methods have been used successfully the video generation literature R As shown in Figure 3 the input to the 4D Interpolation Model consists of two sets of multiview images and the interpolation model is trained to produce additional intermediate sets of Gaussians. It leverages the weight- of the RGB pixels between the multiview images for the newly created intermediate frames. The 4D Interpolation Model then outputs the corresponding sets of Gaussians. In practice, we insert two additional time frames.
5
Objaverse-4D dataset
Dataset Collection To collect a large-scale dataset for the 4D reconstruction task, we render all animated objects in Objaverse 1.0 [II].
Out of 800,000 objects, only 44,000 have animations. Since each object can have multiple associated animations, we obtain a total of 110,000 animations. All of the animations are 24 fps.
The dataset consists mostly of animations on rigged characters or objects, e.g., "a T-Rex walking" or "a windmill spinning". The motions include diverse scenarios such as dynamic locomotions, fine-grained facial expressions, and smoke effects. Interestingly, there are considerable amount of deforming motions in the dataset, since many animations feature a fantasy style thus, even rigid real-world objects deformable. See Appendix Figure 7 and supplementary video for examples.
5

---

12
Is
P
P1
Jo
L4GM
render
render
J
RGB interpolated frames
Jo4a
P1
P2
P3
Pr
Jy41
ITv2
12T-1
render
Interpolation Model
L4GM
Pr
Pr-1
Pr-2
Pzr-1
Pt
Pros
Pl+z
Pi+3
Figure 3: Left: Autoregressive reconstruction. We use the multiview rendering of the last Gaussian as the input to the next reconstruction. There is one- one-frame overlap between two consecutive reconstructions. Right: 4D Interpolation. The interpolation model takes in the interpolated multiview videos rendered from the reconstruction results and outputs interpolated Gaussians.
we adopt the assumption that the real-world monocular videos mostly have 0 elevation camera poses, and thus render input views for our training data accordingly. Specifically, since animations are of varying lengths, we split each animation into second subclips and render each 4D object into views second long clips. The views are from /) 16 fixed cameras, where cameras are placed at 08 elevation with uniformly distributed azimuths, and 2) random cameras, where cameras are placed random elevations and azimuths. During training, sample input camera poses O from the 16 fixed cameras and sample the supervision cameras Osup from the 32 random cameras. Furthermore, following (2), we further filter out approximately 50% of the 26M videos with small motion based optical flow magnitude, resulting in total of 12M videos in Objaverse-4] dataset.
Dataset Rendering Following 53 32
6 Experiments
6.1 Implementation Details
We rendered the dataset with Blender and the EEVEE engine 9 We used fixed camera intrinsics and lighting detailed in the appendix. For L4GM, we downsample the clips to 8 FPS and train the model for 200 epochs. In training, we set T 8 and use input cameras and supervision cameras. During inference, we used 7 = 16. which we empirically show to work well for longer videos in section 6.3] Each forward pass through L4GM takes about 0.3 seconds, while generating sparse views requires about 2 seconds. For training the interpolation model, we use the 24 FPS clips without downsampling and fine-tune L4GM for another 100 epochs. The 4D interpolation model takes 0.065 seconds to interpolate between every two frames (see Appendix for details).
6.2
Comparisons to State-of-the-Ard Methods
We focus our evaluations on video-to-4D reconstruction. Although L4GM can also be used for text-to-4D or image-to-4D synthesis by taking text-to-video image-to-video outputs from video generative models as input, existing text-to-4I) [66
1 46) and image-to-4D [71] approaches typically rely on score distillation and orders of magnitudes slower than L4GM, preventing meaningful com- parisons. Hence, we concentrate on video-to-4D. Results are presented in the following paragraphs.
Quantitative Evaluation. We evaluate our I 14GM on the benchmark provided by Consistent4D 21 It consists of cight dynamic 3D animations of 4 seconds in length, at 8-FPS. A video from view is

---

Figure 4: Qualitative results from L4GM, showcasing renderings from 4D reconstructions produced from two in-the-wild videos.
used as imput and videos from other viewpoints are used for evaluation. Three metrics are computed: 1J Perceptual similarity (LPIPS) between the generated and the ground truth novel views, 2) the CLIP 38 image similarity between the generated and the ground truth novel views, and 3) the FVD [11 against the ground truth novel views, which measures video quality. We also report the runtime. We reconstruct the video at the original frame rate without using interpolation model. The results shown in Table I] L4GM outperforms existing video-to-4D generation approaches all quality metrics by significant margin, while being 100 to 1,000 times faster.
Qualitative Evaluation. Figure illustrates the renderings produced by L4GiM on two videos from different angles and timesteps. These examples are taken from the Activi- tyNet (12) and Consistent4D [21] datasets. As shown in the figure, LAGM produces high- quality, sharp renderings while exhibiting strong temporal and multiview consistency.
Table I: Quantitative results for video-to-4D. Best is bolded. results from Gao al. [15]
Method
LPIPS
CLIP
FVD
Timel
Consistent
21
0.16
0.87
1133.44
2 hr 1 hr
4DGen [4
0.13
0.89
Gaussian
15
0.14
0.91
STAGAD 67
0.13
0.91
992.21
1 hr 10 min 6 min
DG4D1
30
0.16
0.87
We compare visual results to DG4D [3]] STAG4D 677. and OpenLRM 19 DG4D and Ours
Efticient-D
33
0.14
0.92
0.12
0.94
691.87
STAG4D are optimization- based approaches
that take 10 minutes and 2 hours on 64-frame videos, respectively. OpenL.M is an opensource work reproducing LRM (20] that reconstructs 3D shapes from single-vier images. We run OpenLRM on every video frame to construct a 3D sequence. We collect 24 evaluation videos from Emu [16), Sora 34 Veo [10,, and ActivityNet [12.. covering both generated videos and real-world videos.
For our approach, we use T = 16 and use the interpolation model. Some qualitative comparisons are presented in Figure 5 Notably, a significant improvement from our approach
higher 3D resolution. Optimization based approaches use only thousands of Gaussians to keep the optimization tractable, while our feed-forward approach can casily reconstruct more than 60,000 Gaussians per frame at dramatically faster speed.
We further conduct user study based on qualitative comparisons, and the results shown in Table Our approach is the most favorable on all evaluation criteria including overall quality, 3D appearance, 3D alignment with input video, motion alignment with input video, and motion realism. More details about the evaluation dataset and the user study are in Appendix G
6.3 Ablation Studies
We carry out variety of ablation studies. For training models in the ablation study, only keep animations from high-quality objects in GObjaverse [37), which accounts for 2255 of the data.
3D Pretraining. Without 3D pre-taining, i.e. without initializing from LGM 49 our model fails to converge (using the same training recipe). The explanation is likely that without large-scale pre-training on static 3D scenes our Objaverse-4D dataset is insufficient for L4GM to not only learn temporal dynamics but also its 3D understanding from scratch. Moreover, starting from fully random initializion may also contribute to training instabilities. Note that when reducing the model size the "small" LGM variation [49], the model starts converge. However, as shown in Table (a). the model converges to significantly lower PSNR in the same training epochs.
7

---

OpenLRM
STAG4D
DG4D
L4GM (ours)
Input View
Novel View
Input View
Novel View
Input View
Novel View
Input View
Novel View
time 1
time 2
time 1
time 2
time
time 2
Figure Qualitative comparisons L4GM's results against the baselines.
Frozen LGM. In this experiment, freeze the layers of the original LGM model and only train the temporal attention layers. As shown in Table (b), the model improves faster at the beginning of the training but converges lower PSNR. This suggests that end-to-end fine-tuning of L4GM, including the non-temporal layers of the 3D LGM, is preferable for the 4D reconstruction task.
Temporal Attention. Without temporal attention, the model falls back into 3D reconstruction model, while receiving asynchronous multiview inputs. The model does surprisingly good job using only the 3D information, but it still converges lower PSNR, shown Table (b). The reconstructed novel view videos contain visible flickering due to the lack of temporal modeling. Please refer the supplementary video for comparison.
Deformation Field. Since different types of deformation fields and HexPlane representations have recently been used the text-to-4D literature 47 26 72]. modify the model predict canonical 3D representation and deformation field based on a HexPlane [5]. Concretely, we average the Gaussians as a canonical 3D representation and introduce a new decoder after the middle block in the U-Net to predict a HexPlane. The representation follows Wu et al. [38] A detailed illustration can be found in the appendix. Although the model can successfully overfit single 4D data sample, fails to learn a reasonable deformation field during large-scale training. As shown in Table (b), the PSNR only slowly improves and output always static. This observation is very different from previous optimization-base. 4D generation works 33 26 We speculate that SDS-based methods often rely on the smoothness of implicit representations to regularize their generations, whereas our L4GM model may have directly learned to output smooth representations from the 4D training data.

---

25
22
pretrain
prosrain
stamp.
16, which offers a balanced Gobjereee sither
20
ahreare LGM
dela
+trerplane
100
200
150
200
epach
epoch
trame
(a) Pretrain and data.
(b) Design choices.
(c) AR reconstruction.
Figure 6: PSNR plot. a) Training with different pretrain and training data. b) Training with different design choices. c) Per-frame PSNR with different video lengths T, AR denotes autoregressive.
Table 2: Comparison to baselines by user study on synthesized 4D scenes with examples. Numbers are per-
centages. Numbers do not add up to 100; difference is due to users voting "no preference" (details in Appendix).
L4GM (ours)
v.s DG4D [39
v.s OpenLRM [9
v.s STAG4D [67
Overall Quality
65.4/25.0
57.9/33.3
54.2/35.0
3D Appearance
67.1/25.8
58.8/31.7
55.0/34.2
3D Alignment w. Input Video
61.3/26.3
51.3/32.1
50.0/36.7
Motion Alignment Input Video
61.3/22.6
50.9/29.2
50.4/31.7
Motion Realism
62.1/30.0
54.2/35.0
50.4/36.7
Autoregressive Reconstruction. Our model allows taking a video with length different from the training video length. Here, we analyze the effect of the test-time video length T and the number of autoregressive steps on the reconstruction quality. We use a long animation in Objaverse-4D dataset and compute the per-frame reconstruction PSNR. Results are in Table (c). When the ground- truth multiviews are provided, the quality slightly drops when using longer video length. When reconstructing autoregressively. the quality decreases with more autoregressive runs. A shorter video length will start with higher quality, but the quality drops faster than a longer video length because more reffstrconssuuction are required. In practice, we select performance for different video lengths.
Time Embedding. Here, explore adding time embedding L4GM so that the model is aware of the ordering of the frames. Timestamps are encoded by a sinusoidal function and added to the camera embedding. However, the PSNR did not improve after adding the time embedding. We speculate that the image frames from the input video are already giving sufficient information about the temporal relation between the timesteps. Therefore, we do not train with temporal embedding to add more flexibility to the model at inference time.
4D Interpolation. Finally, we show comparison of using model on an 8-FPS video from the Consistent4D dataset in the supplementary video. Notably, the 4D interpolation model can successfully improve the framerate beyond the input video framerate.
using the 4D interpolation
7 Conclusions
We presented L4GM, the first large reconstruction model for dynamic objects. It produces dynamic sequences sets of 3D Gaussians from single-view video. The model leverages prior 3D know knowledge from pretrained 3D reconstruction model, and learns the temporal dynamics from synthetic dynamic dense-view 4D dataset, Objaverse-4D dataset, which we collect. L4GM can reconstruct long videos and uses learned interpolation to achieve high framerates. We achieve orders of magnitude faster inference times than existing 4D reconstruction or text-to-4D methods. Moreover, our model generalizes well to on-the-wild real and generated videos. Our work is early attempt Al tooling for 4D content creation, with many challenges remaining. For example, for making this technology really useful for professionals need to develop more advanced human-in-thee
4D editing capabilities.
Broader Impact. L4GM allows fast 4D reconstruction from in-the-wild videos, which is useful for various graphics and animation applications. However, this model should be used with abundance of caution to prevent malicious impersonations. The model also trained on non-commercial public datasets and is for research-only purpose.

---

References
[1] S. Bahmani, Skorokhodov, V. Rong. Wetzstein, L., Guibas, Wonka, S. Tulyakov, J. J. Park, A. Tagliasacchi, and D. B. Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311.179% 2023.
[2] A. Blattmann, T. Dackhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
[3] A. Blattmann, R. Rombach, H. Ling, Dockhorn, W. Kim, S. Fidler, and K. Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[4] M. Büsching. Bengtson, Nilsson, and M. Björkman. Flowibr: Leveraging pre-training for efficient neural image-based rendering of dynamic scenes. arXiv preprint arXiv:2309.05418 2023.
[5] A. Cao and Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130-141, 2023.
[6] D. Charatan, S. Li, A. Tagliasacchi, and V. Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312.12337, 2023.
[7] Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T.-J. Cham, and Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.1400 2024.
[8] Y. Cheng, Li, Y. Xu, X. Li, Z. Yang. W. Wang, and Y. Yang. Segment and track anything. arXiv preprint arXiv:230..06558 2023.
[9] B. O. Community. Blender- a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. Utt.http: wttp:///wwwwbbwee.e.m..be www.beleedd...o org
[10] G. Deepmind. Veo: our most capable generative video model. 2024. URL https: sttpss//deepmind google/technologies//
[11] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1115313153, 2023.
[12] B. G. Fabian Caba Heilbron, Victor Escorcia and J. C. Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
[13] S. Fridovich-Kets Keil, G. Meanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[14] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712-5721, 2021.
[15] Q. Gao, Q. Xu, Z. Cao, B. Mildenhall, W. Ma, L. Chen, D. Tang, and U. Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint arXiv:2403. 2024.
[16] R. Girdhar, M. Singh, A. Brown, Q. Duval, Azadi, S. S. Rambhatla, A. Shah, X. Yin, D. Parikh, and I. Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:231..10700 2023.
[17] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, Hamburger. H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
10

---

[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[19] Z. He and T. Wang- OpenIrm: Open-source large reconstruction models. https://gitthb com Com//@Dopiaa/Dpenlaa
2023.
[20] Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprini arXiv:231 1.04400 2023.
[21] Y. Jiang, L. Zhang, J. Gao, W. Hu, and Y. Yao. Consistent4d: Consistent 360 { (A,eg) dynamic object generation from monocular video. arXiv preprint arXiv:231 1.02848, 2023.
[22] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[23] C. Li, J. Lin, and G. H. Lee, Ghunerf: Generalizable human nerf from a monocular video. arXiv preprint arXiv:2308. 16576, 2023.
[24] S. Li, C. Li, W. Zhu, B. Yu, Y. Zhao, C. Wan, H. You, H. Shi, and Y. Lin. Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1-13, 2023.
[25] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng. X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3D: High- Righ-Resolution Text-to-3D Content Creation. In Proceedings of the IEEE/CVF Conference
Computer Vision and Pattern Recognition (CVPR), 2023.
[26] H. Ling, S. W. Kim, A. Torralba, S. Fidler, and K. Kreis, Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:2312.1376: 2023.
[27] R. Liu, R. Wu, B. Van Hoorick, Tokmakov, Zakharov, and C. Vondrick. Zero-1-to-3 Zero-shot one image to 3d object. In Proceedings of the IEEE/CVP International Conference on Computer Vision (ICCV), pages 9298-9309, October 2023.
[28] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint 1906.007/07.7 2019.
[29] J. Luiten, G. Kopanas, B. Leibe, and Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308. 09713, 2023.
[30] Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang. Y. Shen, D. Zhao, J. Zhou, and T. Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[31] M. Masuda, J. Park, S. Iwase, R. Khirodkar, and K. Kitani. Generalizable neural human renderer. arXiv preprint arXiv:2404.144 14199, 2024.
[32] I. Melas-Kyriazi, I. Laina, C. Rupprecht, N. Neverova, A. Vedaldi, O. Gafni, and F. Kokkinos. 1m-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. arXiv preprint arXiv:2402.08682, 2024.
[33] Z. Pan, Z. Yang. X. Zhu, and L. Zhang, Fast dynamic 3d object generation from a single-view video, 2024.
[34] B. Peebles, T. Brooks, C. Brooks, C. Ng. D. Schnum, E. Luhman, J. T.yyoor,, L. Jing, N. Summers, R. Wang, and et al. Creating video from text. 2024. URL https:
ssora
[35] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. DreamFusion: Text-to-DD using 2D Diffusion. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[36] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 00318-10327, 2021.

---

[37] L. Qiu, G. Chen, X. Gu, Q. zuo, M. Xu, Y. Wu, W. Yuan, Z. Dong, L. Bo, and X. Han. Richdreamer: generalizable normal-depth diffusion model for detail richness in text-to-3d. arXiv preprint 2rXiv:2331... 16918,, 2023.
[38] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 PMLR, 2021.
[39] Ren, L. Pan, J. Tang, C. Zhang, A. Cao, G. Zeng, and Z. Liu. Dreamgaussian4d: Generative 4d gaussian splatting, 2023.
[40] A. Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference on Learning Representations, 2021.
[41] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[42] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assistdd Intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[43] M. Seitzer, S. van Steenkiste, T. Kipf, K. Greff, and M. S. Sajjadi. Dyst: Towards dynamic neural scene representations on real-world videos. arXiv preprint arXiv:2310.06020, 2023.
[44] Q. Shen, X. Yi, Z. Wu, P. Zhou, H. Zhang, S. Yan, and X. Wang. Gamba: Marry gaussian splatting with mamba for single view 3d reconstruction. arXiv preprint 2rXiv:2403.18795, 2024.
[45] Y. Shi, Wang, J. Ye, Mai, K. Li, and X. Yang. Mydream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
[46] U. Singer, A. Polyak, T. Hayes, X. Yin, An, S. Zhang, Q. Hu, H. Yang. O. Ashual, O. Gafni, D. Parikh, S. Gupta, and Y. Taigman. Make-A-Video: Text-to-Vided Generation without Text- Video Data. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[47] U. Singer, S. Sheynin, A. Polyak, Ashual, I. Makarov, F. Kokkinos, N. Goyal, A. Vedaldi, D. Parikh, Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint 1tti:2200../2880 2023.
[48] V. Sitzmann, S. Rezchikov, Freeman, J. Tenenbaum, and Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 1911999119925... 2021.
[49] J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 2024.
[50] Tian, S. Du, and Y. Duan. Mononerf: Learning a generalizable dynamic radiance field from monocular videos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 17903-17913, 2023.
[51] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. To- wards accurate generative models of video: new metric challenges. arXiv preprint arXiv: 2018.
[52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[53] Wang and Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.022011 2023,
12

---

[54] P. Wang, X. Chen, T. Chen, S. Venugopalan, Z. Wang, et al. Is attention all that nerf needs? arXiv preprint arXiv:2207.13298 2022.
[55] Q. Wang, Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view image- based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2021.
[56] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu. ProlifieDreamer: High-Fidelity and Diverse Text-to-DD Generation with Variational Score Distillation. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[57] C.-Y. Wu, J. Johnson, J. Malik, C. Feichtenhofer, and G. Gkioxari. Multiview compressive coding for 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9065-9075, 2023.
[58] G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint 2xXx:2310.088266 2023.
[59] J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7623 2-33, 2023.
[60] W. Xian, J.-B. Huang. J. Kopf, and C. Kim. Space-time neural irradiance fields for free- viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 9431, 2021.
[61] Y. Xu, Z. Shi, W. Yifan, H. Chen, C. Yang, S. Peng, Y. Shen, and G. Wetzstein, Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint 2tti:2403.146211. 2024.
[62] G. Yang, D. Sun, V. Jampani, D. Vlasic, F. Cole, H. Chang, D. Ramanan, W. Freeman, and C. Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15980-15989, 2021.
[63] Z. Yang, Z. Pan, C. Gu, and L. Zhang. Diffusion": Dynamic 3d content generation via score composition of orthogonal diffusion models. arXiv preprint arXiv:2404.02144 2024.
[64] Y. Yin, D. Xu, Z. Wang, Y. Zhao, and Y. Wei. 4dgen: Grounded 4d content generation with spatial-tend temporall consistency. arXiv preprint arXiv:2312.17225, 2023.
[65] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[66] X. Yu, Y.C. Guo, Y. Li, D. Liang, S.-H. Zhang, and X. Qi. Text-to-3D with Classifier Score Distillation. arXiv preprint arXiv22310.19411 2023.
[67] Y. Zeng, Y. Jiang, S. Zhu, Y. Lu, Y. Lin, H. Zhu, W. Hu, X. Cao, and Y. Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians, arXiv preprint arXiv:2403. 140.99999, 2024.
[68] K. Zhang, S. Bi, H. Tan, Y. Xiangli, N. Zhao, K. Sunkavalli, and 7. Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:24044 19702, 2024.
[69] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595 2018
[70] X. Zhao, A. Colburn, F. Ma, M. A. Bautista, J. M. Susskind, and A. G. Schwing. Pseudo- generalized dynamic view synthesis from a video, 2024.
[71] Y. Zhao, Z. Yan, E. Xie, Hong, 7. Li, and G. Lee. Animatel24: Animating one image to 4d dynamic scene. arXiv preprint 2rXiv:2311.14603, 2023.
13

---

[72] Y. Zheng, X. Li, K. Nagano, S. Liu, K. Kreis, O. Hilliges, and S. D. Mello. A unified approach for text-and amage-guided 4d scene generation. arXiv preprint arXiv:2311.16844 2023.
14

---

Contents
Introduction
2
Related Work
2.1
Large 3D Reconstruction Models
Video-to-4D Reconstruction
2.3
lext-Too-44 Generation
Background
4
Our Method
Generate Multiview Images with ImageDream
4.2
Turning the 3D LGM into 4D Reconstruction Model
4.3
Autoregressive Reconstruction and 4D Interpolation
Objaverse-4D dataset
6
Experiments
6.1
Implementation Details
6.2
Comparisons to State-of-the- Art Methods
6.3
Ablation Studies
Conclusions
Autoregressive Reconstruction and 4D Interpolation Model Details
17
More Implementation Details
17
Example Training Data
18
D Full Quantitative Results on Consistent4)) Benchmark
18
Azimuth Alignment
20
F HexPlane Model Details
20
More Qualitative Eyaluation Details
G. Evaluation Dataset
21
G.2 User Study
H More Qualitative Results
22
H.
More comparisons with state otate-of-the of-the-rttttttto approaches
22
H.2
Qualitative Results on Consistent4D Data
22
Limitations
22
15

---

More discussions
23
16

---

A Autoregressive Reconstruction and 4D Interpolation Model Details
Autoregressive Reconstruction for Longer Videos. Our model accepts a monocular video I = {1+)=1 with a fixed length 7 input, where T is a hyperparameter set during training (note that during inference 7' can in principle be longer than during training). For videos longer than T, we can also operate our model in an autoregressive manner. Consider long video {1+] where the frame length L significantly exceeds T'. We first apply L4GM to the initial '"' frames to generate the first set of T Gaussians {Pr) Subsequently, we render the last Gaussian Pr from four orthogonal angles to obtain new generated multiview images Jr = {f( A0) s0E 11001100,00 ,270 ). The rendered multiview images Jr along with frames I = t=) - are used construct the next set of T 1 Gaussians (P This process is repeated until all L frames have been reconstructed. We find that this autoregressive reconstruction method can be repeated more than 10 times without significant drop quality (see Figure left).
}
4D Interpolation Model. Since our model does not track Gaussians across frames, interpolating
Gaussian trajectories is not feasible (29 58). Hence, we developed an interpolation model that operates directly within the 4D space, fine-tuning it on top of L4GM. As demonstrated in Figure 3
right side, the input to the 4D Interpolation Model consists of two sets of multiview images, J, and J++3- During training, inference, J, and Ji+3 are rendered from reconstructed sets of gaussians P. and P:+3- To create the T x V input grid, we insert two time steps between the input views by simply calculating the weighted-average on the RGB pixels between the multiview images Jj and Ji+3, resulting in total of 4 x V images. The 4D Interpolation Model then outputs the corresponding four sets of Gaussians, which are supervised using the ground truth data at 24 FPS. During inference, we render the reconstruction results into multiview videos and feed them into the interpolation model to produce a sequence of Gaussians at a three times higher framerate.
render J. and J++3 FPS from asset in our dataset. During
B More Implementation Details
Dataset. We render the dataset with the EEVEE engine in Blender (9, which requires three days with 200 GPUs. For random cameras, we select a random elevation from [-5%, 60% For the fixed camera setting, we rendered from 16 views equally distributed at 0 elevation looking at the origin. We the camera radius to 1.5 align with LGM so that we can leverage its 3D pretrain. The camera FOV is set to 49.1". We use the lighting from the Objaverse-XX XL codebase while fixing the strength to the mean value.
Following Blattmann et al. (2], we also employ motion filtering. Specifically, we downsample the 4D animation clips of Objaverse-4D dataset to 2-FPS and compute the optical flow magnitude averaged on videos rendered from four orthogonal fixed O-elevation cameras. A histogram of the optical flow magnitude is shown in Table 3] We only keep the animation clips with average optical flow magnitude larger than 0.15, resulting in 51K 4D animations with 28K different objects, 246K 1-second clips, and 12M videos rendered from 48 camera poses.
Model. We use the pretrained LGM model [49 released in the official code which has 6 down- sampling blocks with channels [64, 128, 256, 512. 1024, 1024], middle block with channel [1024], and upsampling blocks with channels [1024, 1024, 512, 256, 128). The input image size is 256x256. The number of output Gaussians for each frame is 128 x 128 x 4 = 65,536. The output Gaussians rendered into images at 512x5 12 resolution for supervision. The added temporal attention layers are the same as the cross-view attention layers, and are inserted after the same U-Net blocks.
hhtps://gihhuh.om//lllaaaaaii Puaps://gthubb.oom/////ppssoooLt
17

---

14000
12000
10000
15000
Count
Count
10000
4000
5000
2000
60
-50
40
30
20
10
6
8
10
flow magnitude (log scale)
flow magnitude
(a) Flow magnitude histogram in log scales.
(b) Flow magnitude histogram in [0.1, 10]. Table 3: Optical flow magnitude histogram on Objaverse-4D dataset.
Loss Functions.
Following LGM, combination of an LPIPS 669 loss and an MSE loss on RGB images, and an MSE loss on segmentation masks to supervise the reconstruction,
LRGB Σ
Σ
- (PP,((P (I
(4)
M Σ
Σ g 0)1
(5)
L= C=GB + LMask
(6)
where f.g are Gaussian volume rendering functions for RGB and alpha masks, To and of are the image and mask rendered from camera pose O time f, and À is loss weight.
We apply the grid distortion augmentation [49] to non-reference views to improve the model robustness. We train the model with one 8-frame clip per GPU on 128 80G A100 GPUs. In each epoch, we iterate through all animations and sample a one-second clip from it regardless of the animation length. The model is trained for 200 epochs, which takes about one day. For ablation models, dataset size to the 25% GObjaverse subset and reduce the number of GPUs to 32 correspondingly while keeping other settings unchanged. For training the interpolation model, we fine-tune L4GM with 4 frames per GPU on 64 80G A100 GPUs for 100 epochs.
Training. For training L4GM, downsample the videos 8 FPS and set T T =,
reduce the
Inference. We test on 16G RTX 4080 Super GPU. We set the video length to T = 16 Each forward pass through L4GM takes about 0.3 seconds while generating sparse views requires about seconds, including ImageDream generation, LGM reconstruction, and azimuth selection. The 4D interpolation model takes 0.065 seconds interpolate between every two frames. For example, for 10-second 30-FPS video, we can reconstruct the input video in 15 FPS and interpolate the result to 45 FPS in seconds, consisting of seconds for sparse view generation, seconds for reconstruction, and 10 seconds for interpolation. Video segmentation time is not included.
C Example Training Data
show example training data Figure
D Full Quantitative Results Consistent4D Benchmark
We present the detailed metric for every test sample the Consistent4D benchmark in Table 4
18

---

Input
View2
Input
Super. View1
Super. View2
Super. View
Super. View
time
time
time
time
time 2
time
time
Figure 7: Example training data. Masked input views will not be visible to the model. They will be replaced by the copy of the multiview images at time t 1.
19

---

Table 4: Comparison between L4GM and state-of-the-arts approaches on full metrics in the Consis- tent4D benchmark. Baseline results are from Gao et al. [5]
Gupple
Tramp
Method
LPIPS)
CLIPT
LPIPS]
CLIPT
LPIPS.
CLIFT
LPIPS.
CLIFT
LPIPS.
CLIPT
CLIPT
LPIPSL
CLIPT
D-NoRF 16
0.52
0.66
00:92
0.76
0..56
0.61
0.52
0..5
0.51
0.73
0.55
0.90
0.96
01165
0.51
0.68
glanes
13
0.40
0.24
10.29
0.75
01.19
0.25
0.47
0.77
0.41
0.72
0.51
0.65
0.37
0.67
0.38
0.72
C4D 2]
0.30
0.50
00.12
0.50
0..22
0.82
0.15
0.90
0.17
0.88
0.23
0.85
0.17
0.83
0.16
0.87
DG4D 1g
0.32
0.92
0.12
0.91
0..2
0.88
0.19
0.99
0.18
0.90
0.22
0.83
0.17
0.85
0.16
0,87
GPlww
13
0.30
0.94
0..0
0.93
0.00
0.90
0.17
0.92
0.17
0.92
0.20
0.85
0.15
0.89
0.14
0.91
Otes
0.88
0.12
0.98
0.89
0.89
0.15
0.92
0.13
0.95
0.15
0.96
0.13
0.92
6.12
0,94
Misaligned
ImageDream
LGM
Input frame
Aligned Camera
Initial multi-views
Aligned multi-views
Figure 8: Azimuth aligment. ImageDream often generates multiviews that misalign with the input frame. We first use LGM to generate 3D from the multiview images, then render it from different azimuths, and finally render it from the most-aligned azimuth and its other three orthogonal camera poses.
Gaussian Sequence
Canonical Gaussian
Deformation Field
F-
AVG
AVG
Reshape
t=T
HexPlane
Figure 9: HexPlane model.
E Azimuth Alignment
After getting Piits render Pintt from series of azimuth angles 0 E {-180-, ..... 180%} and select the azimuth that best aligns with the input frame. This determined by Oatign lf static 0) Lil where f a Gaussian volume rendering function. The final uti- lized sparse views, J,. are defined as (f static Oalign + A0)) sdE (ue 900, 100,,1800.20 2700,27000. An illustration shown in Figure 8
F HexPlane Model Details
In this ablation study, we predict a canonical Gaussian and deformation field represented by HexPlane. additional decoder decodes the output from the U-Net middle block into 6 planes. The decoder is also equipped with cross-view and temporal attention. The canonical Gaussian then goes through the deformation field to produce 7 sets of 3D Gaussians. The decoder has channels [1024, 128, 128]. We show illustration in Figure 9
20

---

Grandd Video
Given reference video shown above, Two different approaches are used to create the corresponding animated character in different views, resulting animations Please answer follawing questions. When answer questions, focus on video view diffrent from reference video.
Overall wiich deeracter animation iooka more appaning nn and hum hightr quilly" A BDO Expally
Look one framw each video. Which animated character mierr hight-quulity appestanoa? O
O
which unimated chanadter motion close shomm relenance video? O Equalyy
For which uniiated charadeer the motion more maaidiic raturalf Redlecto and wawuraaaulllloooooo mean les. flcloring and los artiactts OAO. Fipully
Figure 10: Screenshot of instructions provided to participants of the user studies for comparing L4GM and baselines.
G More Qualitative Evaluation Details
G.1 Evaluation Dataset
We collect 24 evaluation videos from multiple sources, including /) 10 videos from Emu video [16]: these are generated 4-second 16-FPS videos. 2) videos from Sora [34]; these are generated long 30-FPS videos. 3) 5 videos from Veo [10; these are generated long 30-FPS videos. 3) videos from ActivityNet [1]; these are real-world long videos of weight lifting.
We segment the foreground object with SAM-Track [8] and then crop and rescale the video to 512x512. Since the optimization-baat based baseline methods are very memory intensive, we trim all videos to 4 seconds and downsample all 30 FPS videos to 15 FPS, so that all videos have 64 frames.
G.2 User Study
We conducted human evaluations (user studies) through Amazon Mechanical Turk to assess the quality of our generated 4D scenes, comparing them with DG4D [39,, OpenLRM [99, STAG4D 677 and performing ablation studies.
We used the 24 examples evaluation dataset as described in Section 6.2 We rendered both baselines and our dynamic 4D scenes from similar camera perspectives and created similar videos. We first asked the participants to watch the reference input vidco and then asked them to compare the two videos with respect to 5 different evaluation axes and indicate preference for one of the methods with an option to vote for "equally good' a non-forced-choit format. The 5 categories measure overall quality, 3D appearance quality, as well as motion alignment to reference video and motion realism.
For visual reference, see 10 for a screenshot of the evaluation interface. In all user studies, the order of video pairs (A-B) was randomized for each question. was evaluated by five participants, totaling 120 responses for each of the baseline comparisons. We selected participants based on specific criteria: they had be from English--peaking countries, have an acceptance rate above 95%, and have completed over 1000 approved tasks the platform.
user studies, each video pair
21

---

OpenLRM
STAG4D
DG4D
Ours
Input
Input
Input
Input
time
tima
time
time
time
time
Figure 11: More qualitative comparison on generated videos.
H More Qualitative Results
H.1 More comparisons with state-of-the-art approaches
We show more qualitative comparisons with state-of-the-art approaches both generated (Figure 11 and real-world (Figure 12) videos.
H.2 Qualitative Results on Consistent4D Data
We show more qualitative results on the Consistent4D dataset in Figure 13
I Limitations
We show some limitations of L4GM in Figure 14
Our model can not handle motion ambiguity well. For example, in some walking motions, model can successfully align with the reference view the leg motion is not natural from other views. The model also cannot reconstruct multiple objects well, particularly when they occlude each other. Finally, the model fails to reconstruct objects from an ego-centric viewpoint, since the model assumes input views to be taken from 0 elevation.
22

---

OpeNLRM
STAG4D
DG4D
Ours
Imput View
Input
Novet
Input
Novel
time
time
time
time 2
Figure 12: More qualitative comparison on real-world videos.
J
More discussions
Ablation results on the Consistent4D benchmark. In Table 5 we evaluate ablation models and interpolation models on the Consistent4DD. Note that the quantitative results can be noisy since the benchmark only has 7 test samples. Comparing to a), we confirm that the pretrain plays a critical role. Comparing f) to a), we observe that the 4D representation is crucial. To evaluate the 4D interpolation model, we downsample the evaluation videos' framerate by 3, reconstruct downsampled videos with a), and then interpolate the 4D reconstruction with our interpolation model. Notably, g) achieves an on-par performance to a), which suggests effectiveness of the interpolation model.
Full attention. We explore replacing the tem- poral attention in L4GM with full attention. Full attention computes the self-attention on all im- ages regardless of time and view, while temporal attention only computes the self-attention on im- ages under the same viewpoint. As result, full attention requires more computing. We show memory-time analysis training iteration in Table 6 implemented with Efficient Atten- tion. c) requires larger memory usage and longer processing time than b). In Figure 15, We further show a PSNR plot that compares temporal attention with full attention, trained on the GObjaverse subset. They achieve al- most identical PSNR curves. Considering both the computation cost and empirical results, temporal attention would be a better design choice.
Table 5: Ablation results the Consistent4D benchmark. Method
LPIPS CLIP1
FVDJ
baseline
0.1 0.94037 652.81
b) pretrain
0.13702 0.89055 851.62
temp attn.
0.11792 0.93776 650.27
d) timeemb
0.11888 0.93888 659.09
e) freezelgm
0.11840 0.94006 729.24
f) + Hexplane
0.12602 0.92639 950.65
interpolation 0.11896 0.94001 622.86
GObjaverse
0.11895 0.93960 630.55
Multi-view multi-step videos input. In Fig- ure 16, we show a multi-view multi-step video generated by ImageDream, after azimuth align- ment. Concretely, we input each individual frame of the reference video to the ImageDream model and generate a multiview image for each time step. The obtained multiview video lacks a temporal consistency. Since ImageDream is probabilistic generative model, even the same image input could produce very different multi-view generation. Therefore, we believe that such temporally inconsistent multi-view videos not ideal input to the reconstruction model.
Table 6: Attention.
Method
memory (Gb) time (s)
a) cross view
51.39
8.53
b) cross-view temp.
53.83
9.64
с) cross-view full
55.27
11.82
d) full
52.96
11.28
23

---

Input View
Novel
Novel View
Novel View
Input View
Novel View1
Nevel View
Novel View
time
time
time
time
time
time
time
time
time
time 2
Figure 13: Qualitative results the Consistent4D dataset.
New evaluation on the GObjaverse subset We provide quantitative results using the GObjaverse subset as training data in Table 5lnn h. We manually verified that these Consistent4D testing samples not part of the GObjaverse subset. The new results remain saae-of-the-art and show no significant difference from the numbers reported the main paper.

---

Input View
Novel View!
Novel View
Input
8)
time
time
time
time
time
Figure 14: Failure cases. Motion ambiguity. b) Multiple objects with occlusions. c) Ego-centric videos taken from Ego4D [77
V=2
v=4
t=1
25
temp. attn
Full attn.
24
t=2
23
22
t=3
21
20
t=4
50
100 epoch
150
200
Figure 15: Full-attention PSNR plot.
Figure 16: Multi-step multi-views.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer:
[No]
Justification: code to be released upon internal approval.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
cessionnooccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuii wwwiiicgguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: repeating large model training is too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///arr.....ooooooo//caaaaaa
Question:
Answer: [Yes]
Justification: the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
User study details are in Section G.2
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
User study details are in Section G.2 and it docsn't contain any potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "DHVJ9VQuek", "reviewer": "Reviewer_4G9y", "summary": "This paper proposes a model for 4D reconstruction from a single video, building upon dynamic 3D gaussians and LGM architecture [49] previously applied to static 3D scenes. By processing generated multi-view images (derived from the first frame using prior method) alongside the video, the model outputs 4D Gaussians to reconstruct the video dynamics. The model was trained on a synthetic multi-view video dataset (Objaverse) and shows qualitative generalization to real-world images. Based on the good performance and extensive ablation studies, I recommend accepting this paper.", "questions": "- Could you provide more information about various axis of failure cases and the generalization limits to in-the-wild videos? \n- Could you provide more details about \"grid distortion\" mentioned on line 594? The referenced paper also lacks clarity on this.\n- Given the use of 128 80GB A100 GPUs, could you provide details on training the model with fewer resources? (particularly memory use)", "limitations": "Authors have acknowledged the limitations, however, the claim about generalizing \"extremely well\" on in-the-wild videos is unsubstantated and likely not true due to training assumptions (e.g. masks, static camera at 0 degree elevation).", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written, with good figures, structure, and well-explained motivation of design decisions\n- The task of 4D reconstruction is both timely and of significant interest to the community.\n- Though the paper contains strong-worded claims (see below), the demonstrated results show strong 4D reconstruction capabilities\n- Ablation studies highlight the benefits of pretraining and the chosen representation.", "weaknesses": "- Claiming that model generalises “extremely well” to in-the-wild lacks empirical support (apart from cherry-picked qualitative results) and likely not true due to training assumptions (e.g. masks, static camera at 0 degree elevation). One possible evaluation to substantiate the claim would be a comparison with other works such as HyperNeRF. \n- Since the code is not available, the details about the architecture and training are not sufficient for reproducing experiments and should be expanded. E.g. I could not find how exactly 32 heldout views are sampled neither in supplementary nor in the main paper. \n- Minor: The architecture and training are not new and relies heavily on prior techniques, including a multi-view image generator. This perhaps aligns work more closely with the computer vision and engineering community (e.g., CVPR)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Claiming that model generalises “extremely well” to in-the-wild lacks empirical support (apart from cherry-picked qualitative results) and likely not true due to training assumptions (e.g. masks, static camera at 0 degree elevation). One possible evaluation to substantiate the claim would be a comparison with other works such as HyperNeRF.\n\nA1. We will modify the tone of the sentence. By “in-the-wild” we emphasize on unseen data domains that are different from our synthetic training data, like generated videos from Sora or real-world videos ActivityNet. We agree that the method may not handle some real-world settings like non-zero elevation angles or unsegmented videos very well. Although we have discussed some failure cases in Appendix I, we thank the reviewer for bringing up HyperNeRF and will consider using that example to strengthen the limitation sections.\n\n> Q2. Since the code is not available, the details about the architecture and training are not sufficient for reproducing experiments and should be expanded. E.g. I could not find how exactly 32 heldout views are sampled neither in supplementary nor in the main paper.\n\nA2. We will release the code upon the acceptance of the paper and internal approval. The mentioned details can actually be found in Appendix B. L574, “For random cameras, we select a random elevation from [-5◦, 60◦]...We set the camera radius to 1.5“, where random cameras are the 32 heldout views.\n\n> Q3. Minor: The architecture and training are not new and relies heavily on prior techniques, including a multi-view image generator. This perhaps aligns work more closely with the computer vision and engineering community (e.g., CVPR)\n\nA3. We believe that this work is closely related to a wide range of artificial intelligence fields, like AI content generation, and 3D perception/simulation in robotics, so it should be well-fitted into the NeurIPS community.\n\n> Q4. Could you provide more information about various axis of failure cases and the generalization limits to in-the-wild videos?\n\nA4. As mentioned in A2, a brief analysis has been provided in Appendix I, including motion ambiguity, multi-object scenes, and ego-centric scenes. We will expand the discussion to include more failure case analysis.\n\n> Q5. Could you provide more details about \"grid distortion\" mentioned on line 594? The referenced paper also lacks clarity on this.\n\nA5. It is a data augmentation technique from the reference paper LGM[49]. Grid distortion simulates 3D inconsistency by grid sampling an image with a distorted grid. This makes the model more robust to inconsistent multiview input images. Please find the detailed implementation at https://github.com/3DTopia/LGM/blob/main/core/utils.py#L63.\n\n> Q6. Given the use of 128 80GB A100 GPUs, could you provide details on training the model with fewer resources? (particularly memory use)\n\nA6. Various practical techniques could be applied to trade of longer training time for less memory consumption, for example, gradient accumulation. Our training takes 1 day on 128 GPUs, so the same model training can be finished on 32 GPUs in 4 days with gradient accumulation, which is also a reasonable training time. \n\n> Q7. Authors have acknowledged the limitations, however, the claim about generalizing \"extremely well\" on in-the-wild videos is unsubstantated and likely not true due to training assumptions (e.g. masks, static camera at 0 degree elevation).\n\nA7. We agree that such limitations exist and will make the claims in the paper more accurate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DtezYCYF3y", "reviewer": "Reviewer_UkTx", "summary": "This work introduces a novel framework for generating animated 3D objects from single-view videos. The proposed framework employs a feed-forward approach, thus eliminating the need for computationally expensive optimization. The core idea is to create a large-scale synthetic multi-view video dataset and train a corresponding 4D LGM model, which is initialized from the 3D LGM [49] and enhanced with temporal self-attentions. Additionally, an interpolation network is introduced to improve the frame rate further. Overall, this method achieves superior performance in producing 4D assets, both in terms of efficiency and reconstruction quality.", "questions": "Some Questions:\n\n- Regarding the temporal self-attention layer, why is only the time dimension considered in the self-attention mechanism rather than incorporating both multi-view and temporal information (full attention)? An analysis focusing on efficiency and performance would be valuable to understand the rationale behind this design choice.\n- Why is the ablation study performed using PSNR (a reconstruction-based metric) instead of the metrics provided in Table 1?\n- It appears that only qualitative comparisons are offered for the interpolation network. Could the performance be evaluated using the metrics shown in Table 1?\n- Given that this work is an original contribution to the feed-forward generation of 4D assets, the community would greatly benefit from the release of the dataset and code to enable the reproduction of the work. Are there any plans to release these resources?", "limitations": "- This work also necessitates a \"synthetic\" multi-view video dataset for training. According to the implementation details, generating this dataset takes approximately 3 days using 200 GPUs. While this is easier to obtain than a real dataset, it still demands significant computational resources. Including a more comprehensive discussion on this in the limitations section would be beneficial.\n- The proposed framework is claimed to generalize extremely well to in-the-wild videos (Abstract). However, there are only a few samples provided (4 samples in Figures 1 & 4), and many of these are in an animated style, with the exception of the left example in Figure 4. This left example displays various artifacts on the produced objects, such as blurred arms. Given the current evidence, this claim appears to be inadequately supported, and it is recommended to either include more samples or lower the claim accordingly.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To the best of my knowledge, this is the first work to achieve feed-forward generation of 4D assets from a given video. The successful utilization of a large-scale dataset to tackle this challenging task may inspire further research in the community on related topics.\n- The proposed framework demonstrates superior performance compared to existing methods across all evaluation metrics, as evidenced by the current evaluation (Figure 5, Tables 1). Additionally, the user study indicates a general preference for the results produced by this framework.\n- Moreover, several significant ablation studies have been conducted on different components of the framework. Notably, the justification for using a pre-trained LGM (Figure 6(b)) is particularly convincing.\n- Furthermore, the autoregressive generation method can produce animated objects with longer intervals, beyond the training time intervals.", "weaknesses": "Technical contributions\n\n- Despite its impressive performance, this work's technical contributions are somewhat limited. The two main contributions are temporal self-attention and the 4D interpolation network.\n- The temporal self-attention can be seen as a direct extension of the multi-view self-attention proposed in [49].\n- While the 4D interpolation network can speed up inference, its overall performance does not seem to have a substantial impact, as demonstrated in the supplementary material video.\n\nHaving said that, I agree that a straightforward solution to a new problem should be recognized, and the above weakness is relatively minor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Technical contributions:\n\nA1.  We appreciate that the reviewer agrees that “a straightforward solution to a new problem should be recognized”.We would also like to highlight that L4GM is the first feed-forward 4D reconstruction model, which could open up more possibilities for generating high-quality 4D assets.\n\nWith regard to detailed technical contributions, as the reviewer pointed out, our core technical contribution is “a novel framework for generating animated 3D objects from single-view videos.” In terms of new techniques required to build such a model, besides the ones mentioned in the review, there are also 1)  autoregressive reconstruction and 2) multi-view copying for input batching. \n\n> Q2. The temporal self-attention can be seen as a direct extension of the multi-view self-attention proposed in [49].\n\nA2. We agree. We choose temporal self-attention to keep the 3D reconstruction pre-train as much as possible and also to keep the model simple for better scalability.\n\n> Q3. While the 4D interpolation network can speed up inference, its overall performance does not seem to have a substantial impact, as demonstrated in the supplementary material video.\n\nA3. Besides inference speed, the interpolation model also helps to alleviate the autoregressive reconstruction error so that we can reconstruct longer videos. Considering a reconstruction model that reconstructs a 1-second 30-FPS video at a time and can maximally self-reconstruct 10 times without quality drop, then the longest video to reconstruct will be 10 seconds. Our 3x-upsample interpolation model allows the reconstruction model to reconstruct a 3-second 10-FPS video at a time, thus improving the maximum video length from 10 seconds to 30 seconds.\n\n> Q4. Regarding the temporal self-attention layer, why is only the time dimension considered in the self-attention mechanism rather than incorporating both multi-view and temporal information (full attention)? An analysis focusing on efficiency and performance would be valuable to understand the rationale behind this design choice.\n\nA4. We use the temporal attention inspired by video diffusion models [2,3], which have been verified to work well when extending image generation to video generation at a large scale. Nonetheless, We agree that full attention is also a reasonable design choice, so we perform an additional experiment comparing full attention to temporal attention and show the results in the rebuttal PDF. Full attention requires more computation but brings no visible improvement to the results.\n\n> Q5.Why is the ablation study performed using PSNR (a reconstruction-based metric) instead of the metrics provided in Table 1?\n\nA5. The Consistent4D evaluation benchmark only has 7 test samples so the numbers can be noisy, thus we choose to show PSNR plots. Nonetheless, we carry out evaluations on the Consistent4D benchmark and present results in the rebuttal PDF.\n\n> Q6. It appears that only qualitative comparisons are offered for the interpolation network. Could the performance be evaluated using the metrics shown in Table 1?\n\nA6. Thanks for the suggestion. We have additionally evaluated the interpolation network on the Consistent4D benchmark by downsampling the test video framerates by 3 times. Results are presented in the rebuttal PDF. The numbers achieved by the interpolated 4D reconstruction and the baseline 4D reconstruction have no significant difference.\n\n> Q7.Given that this work is an original contribution to the feed-forward generation of 4D assets, the community would greatly benefit from the release of the dataset and code to enable the reproduction of the work. Are there any plans to release these resources?\n\nA7. We will release the code upon the acceptance of the paper and internal approval.\n\n> Q8. This work also necessitates a \"synthetic\" multi-view video dataset for training. According to the implementation details, generating this dataset takes approximately 3 days using 200 GPUs. While this is easier to obtain than a real dataset, it still demands significant computational resources. Including a more comprehensive discussion on this in the limitations section would be beneficial.\n\nA8. Thanks for the suggestion, we will discuss the approach’s requirement on computational resources and improve our limitation section.\n\n> Q9. The proposed framework is claimed to generalize extremely well to in-the-wild videos (Abstract). However, there are only a few samples provided (4 samples in Figures 1 & 4), and many of these are in an animated style, with the exception of the left example in Figure 4. This left example displays various artifacts on the produced objects, such as blurred arms. Given the current evidence, this claim appears to be inadequately supported, and it is recommended to either include more samples or lower the claim accordingly.\n\nA9. We will modify the tone of the claim and make it more accurate.  By “in-the-wild” we meant to emphasize that the model generalizes on unseen data domains that are different from our synthetic training data, like generated videos from Sora or real-world videos ActivityNet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TaXGIYjvVl", "reviewer": "Reviewer_hQSp", "summary": "This paper utilizes rendering of animated objects from objverse(-xl) to extend lgm into 4D generation. Specifically, L4GM uses four orthogonal images of an object and the object's monocular dynamic video to obtain 3D Gaussians at each moment, enhancing the consistency between different moments through temporal self-attention layers. Subsequently, the smoothness of generated actions is further improved through a 4D Interpolation Model. L4GM leads in generation speed and evaluation metrics in the task of video-to-4D.", "questions": "1. In section 5, it is mentioned that some objects come with animations. Does this mean the objects already have predefined animations?\n2. In the supplementary materials, I noticed that some generated 4D objects have limited motion range and motion rationality, which is a common issue in other 4D generation works. Since evaluating these is a resource-intensive task, are there any possible evaluation methods that could be discussed? This could be part of future work, and once an evaluation method is established, the motion quality of 4D generation could potentially be further improved.", "limitations": "The main issue lies in the amplitude and controllability of the animations. Is there a potential direction for this in the future?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper significantly improves the generation speed of text-to-4D by utilizing a large-scale, object-centered dynamic dataset to extend lgm into 4D generation, and can model 4D objects in wild videos, achieving highly generalized results.\n2. The paper proposes a 4D interpolation Model that can increase the frame rate of generated 4D objects, making the motion smoother. This will alleviate the problem of insufficient frame rates in video generation models.", "weaknesses": "The paper discusses extensively how to use dynamic datasets for pre-training, which is also a very important part of this work and will have a significant impact on the community. Whether this dataset is open source is also extremely important for evaluating this work, but the paper does not mention this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The paper discusses extensively how to use dynamic datasets for pre-training, which is also a very important part of this work and will have a significant impact on the community. Whether this dataset is open source is also extremely important for evaluating this work, but the paper does not mention this point.\n\nA1. We will release the code for both the model and dataset building upon the acceptance of the paper and internal approval. Besides, we have tried our best to provide enough technical details in the paper to reproduce this work.\n\n> Q2. In section 5, it is mentioned that some objects come with animations. Does this mean the objects already have predefined animations?\n\nA2. Yes. The Objaverse dataset provides 3D objects with predefined 3D animations. Please kindly refer to this example in the Objaverse dataset https://skfb.ly/FWLt, where the dragon has been animated.\n\n\n> Q3. In the supplementary materials, I noticed that some generated 4D objects have limited motion range and motion rationality, which is a common issue in other 4D generation works. Since evaluating these is a resource-intensive task, are there any possible evaluation methods that could be discussed? This could be part of future work, and once an evaluation method is established, the motion quality of 4D generation could potentially be further improved.\n\nA3. In terms of \"limited motion range and motion rationality\", we highlight that we have performed a user study (in Table 2) on 24 videos from diverse sources (as detailed in Appendix G.1)  to evaluate the motion quality.  Our method achieves a high win rate over existing works on \"Motion Realism\" and \"Motion Alignment w. Input Video\". In addition, limited motions are not bound by our approach. As a reconstruction model, the reconstructed motion largely depends on the input video. \n\nWe agree with the reviewer that a large-scale, comprehensive evaluation benchmark for 4D generation will be important to standardize and simplify evaluation. Although there exists a popular evaluation benchmark Consistent4D that automatically computes metrics, where we also evaluate our approach, the benchmark only provides 7 test samples and does not have a metric for motion quality. Improving the 4D evaluation benchmark is a crucial future work.\n\n> Q4. The main issue lies in the amplitude and controllability of the animations. Is there a potential direction for this in the future?\n\nA4. As mentioned above, the animation depends on the input video. Controlling the animations via video generation or editing is feasible but lies beyond the scope of this work. Our work only serves as a deterministic reconstruction model that reconstructs the animation from 2D to 3D.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Bw8oGfGOq", "reviewer": "Reviewer_2k2z", "summary": "This paper proposed L4GM, an efficient large 4D Gaussian reconstruction model to produce animated objects from videos by a single feed-forward. L4GM leverages ImageDream and LGM to achieve multiview images of the first frame as the input. The overall model is built upon the pre-trained LGM with cross-view and temporal attention blocks, which could be easily modified by rearranging the feature dimension. Besides, this paper proposed autoregressive reconstruction and 4D interpolation, largely improving the overall smoothness of the generated 4D. To train L4GM, the authors rendered animated objects from Objaverse, and conducted sufficient experiments to verify the effectiveness.", "questions": "1. Maybe there is a typo of temp. embed. in Figure 6(b), which should be time embed?\n2. Why there are 12M videos at all? The authors said that 110k animations are captured with 48 views. So the overall videos should be about 5M.\n3. Since L4GM is trained with animated objects from Objaverse,  the comparison of Consistent4D in Table 1 is not convincing enough. To the best of my knowledge, most objects in the Consistent4D test set are from Objaverse too. Please refer to https://github.com/yanqinJiang/Consistent4D/blob/main/test_dataset_uid.txt for more details.", "limitations": "The authors have discussed limitations in the supplementary.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. As a feed-forward technique, L4GM enjoys good efficiency and performance, especially for the high-resolution 4D generation.\n2. The overall pipeline is convincing while the paper is also clearly written.\n3. This paper contributes to large-scale 4D data with 12M videos rendered from Objaverse.", "weaknesses": "1. The main concern is the novelty. Although the overall pipeline is convincing, this paper is more like an extension of LGM for 4D generation. Most techniques are very straight-forward, such as temporal and cross-view attention, and multi-view synthesis by ImageDream+LGM.\n\n2. Another concern is the setting of repeating the multiview images from the initial timestep as inputs for other frames. It works more like seeking temporary relief rather than a solid solution. For example, this repeating suffers from solving 4D generation with the motion of turning around, whose multiview images would contain conflict to the inputs. Why not use ImageDream+LGM to achieve more multiview inputs? Moreover, more quantitative and qualitative ablation studies should be considered for this setting, such as no-repeating views, repeating views, and multi-timestep-multi-view inputs respectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The main concern is the novelty. Although the overall pipeline is convincing, this paper is more like an extension of LGM for 4D generation. Most techniques are very straight-forward, such as temporal and cross-view attention, and multi-view synthesis by ImageDream+LGM.\n\nA1. As pointed out by reviewers (4G9y, UkTx, hQSp), we would like to emphasize that L4GM is the first 4D reconstruction network. To achieve this, we propose several methods to successfully generate 4D assets in this setting. One of our core technical contributions is the novel framework itself. Regarding the reviewers’ concern about the techniques being \"very straightforward,\" as mentioned in the main paper, we intentionally kept the model simple to ensure better scalability.\n\nWe would also like to highlight that building a large 4D object dataset and training a reconstruction model on it have not been explored before. While the design choices may seem straightforward in hindsight, several uncertainties existed during the model's development. These included: 1) whether a model could learn to reconstruct dynamic 3D objects over time from a single multi-view, 2) identifying an appropriate input format for combining single-view video and multi-view images, and 3) determining a suitable 4D representation for the feed-forward model to predict.\n\nThus, we believe L4GM is a major step towards generating high-quality 4D assets.\n\n\n> Q2. Another concern is the setting of repeating the multiview images from the initial timestep as inputs for other frames. It works more like seeking temporary relief rather than a solid solution. \nFor example, this repeating suffers from solving 4D generation with the motion of turning around, whose multiview images would contain conflict to the inputs. Why not use ImageDream+LGM to achieve more multiview inputs? Moreover, more quantitative and qualitative ablation studies should be considered for this setting, such as no-repeating views, repeating views, and multi-timestep-multi-view inputs respectively.\n\nA2.  “Multi-timestep-multi-view\" setting: Using ImageDream to generate more multiview inputs will generate very different 3D objects in different time steps. Please refer to an example in the rebuttal PDF. Generating temporally consistent multi-view videos is in fact a challenging task at the early stage of research [A]. \n\n“No repeating views and repeating views” setting: We would like to reiterate that repeating views are only designed to let the model better handle the “T+V” input. Thus, no repeating views means that we need to change the model architecture to take a different input format, which we deem non-trivial and with the risk of hurting the pretrain weights.\n\nWe agree that there are cases when the generated multiview images would conflict with the input video. We believe that the challenge can be tackled by improving the model's robustness to inaccurate multiviews, with strategies like augmentation and random dropout, which we leave for future works.\n\n[A] Liang et. al., Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models, 2024\n\n> Q3. Maybe there is a typo of temp. embed. in Figure 6(b), which should be time embed?\n\nA3. Yes, it is a typo. Thanks for pointing this out.\n\n> Q4. Why there are 12M videos at all? The authors said that 110k animations are captured with 48 views. So the overall videos should be about 5M.\n\nA4. One animation could be longer than one second. The 110k animations have 250k seconds after filtering, which sums up to 250k*48 = 12M 1-second videos.\n\n> Q5. Since L4GM is trained with animated objects from Objaverse, the comparison of Consistent4D in Table 1 is not convincing enough. To the best of my knowledge, most objects in the Consistent4D test set are from Objaverse too. Please refer to https://github.com/yanqinJiang/Consistent4D/blob/main/test_dataset_uid.txt for more details.\n\nA5. Thanks for pointing this out. We refer the reviewer to a new evaluation of the model only trained on the GObjaverse subset in the rebuttal PDF. We manually checked that these test samples are not part of the GObjaverse subset, so the issue can be avoided. The new results remain state-of-the-art and have no significant difference from the numbers we report in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank reviewers for the encouragement and insightful feedback. We are glad that the reviewers found:
- (4G9y, UkTx, hQSp) This is the first work to achieve feed-forward generation of 4D assets from a given video. The task of 4D reconstruction is timely and of significant interest, demonstrating strong results and superior performance compared to existing methods .
- (4G9y, UkTx) Ablation studies effectively highlight the benefits of pretraining and the chosen representation, with significant insights on different components .
- (hQSp, 2k2z) The proposed 4D interpolation model increases the frame rate and smoothness of motion, extending the capabilities of L4GM in high-resolution 4D generation.
- (2k2z, hQSp) The paper contributes significantly to large-scale 4D data with 12M videos rendered from Objaverse, showcasing highly generalized results.

We refer the reviewers to the uploaded PDF file for additional experiments, including:
- (UkTx) Evaluating ablation models and interpolation models on the Consistent4D benchmark. 
- (UkTx) Explore replacing temporal attention with full attention.
- (2k2z) Generating multi-view multi-step videos using ImageDream.
- (2k2z) New evaluation on the GObjaverse subset, which has no overlap with the Consistent4D benchmark.

In addition, we would like to address some common concerns, including :
- (4G9y, hQSp, UkTx) Code and data release: We will release the code upon paper acceptance and internal approval.
- (4G9y, UkTx) Claim on in-the-wild:  We will modify the tone of the claim. By “in-the-wild” we emphasize on unseen data domains that are different from our synthetic training data,

Please kindly let us know if you have any further questions.

Meta-review: The work received positive feedback from reviews, highlighting the impressive results, clear writing and acknowledging that this is the first work presenting a feed forward approach for 4D reconstruction from video. Slight concerns remained with respect to technical novelty. All in all though, the reviewers believe that this paper is of value to the community, due to the currently relevant topic and state of the art results.

I agree and follow with my accept recommendation.

Congratulations to the authors!
Decision: Accept (poster)