L4GM: Large 4D Gaussian Reconstruction Model

Authors: jiawei  ren
kevin  xie
ashkan  mirzaei
hanxue  liang
xiaohui  zeng
karsten  kreis
ziwei  liu
antonio  torralba
sanja  fidler
seung wook kim
huan  ling
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 18:31:38

Abstract: We present L4GM, the first 4D Large Reconstruction Model that produces animated objects from a single-view video input -- in a single feed-forward pass that takes only a second.
Key to our success is a novel dataset of multiview videos containing curated, rendered animated objects from Objaverse. This dataset depicts 44K diverse objects with 110K animations rendered in 48 viewpoints, resulting in 12M videos with a total of 300M frames. 
We keep our L4GM simple for scalability and build directly on top of LGM, a pretrained 3D Large Reconstruction Model that outputs 3D Gaussian ellipsoids from multiview image input.
L4GM outputs a per-frame 3D Gaussian splat representation from video frames sampled at a low fps and then upsamples the representation to a higher fps to achieve temporal smoothness. We add temporal self-attention layers to the base LGM to help it learn consistency across time, and utilize a per-timestep multiview rendering loss to train the model. The representation is upsampled to a higher framerate by training an interpolation model which produces intermediate 3D Gaussian representations. 
We showcase that L4GM that is only trained on synthetic data generalizes well on in-the-wild videos, producing high quality animated 3D assets.

Main paper: 
 introduction: 
 Animated 3D assets are essential in bringing 3D virtual worlds to life. However, these animations are time consuming to create as the procedure involves rigging and skinning of objects, and crafting keyframes of the animation -all with minimal automation in tooling. The ability to generate animated 3D assets from widely available monocular videos or simply from text would be a desirable capability for this application. This is the goal of our work. Building more advanced 4D content editing tooling, the ultimate goal of this line of research, is out of scope for this work.Past work on automatically generating animated 3D objects, which we refer to as 4D modeling in this paper, falls into different categories. The first line of work aims to faithfully reconstruct 4D objects from multiview video data, and oftentimes requires many views across time to achieve high quality [29, 5, 36] . Such data is expensive to collect which limits applicability. Another line of work instead relies on the power of video generative models. Most commonly, the video score distillation technique is used which optimizes a 4D representation, for example a 3D deformation field, by receiving iterative feedback from the video generative model. Score distillation is known to be fragile (sensitive to prompts), and time consuming (hours per prompt) as oftentimes many iterations are needed to achieve high quality results [47, 26, 72, 1] .Recently, a promising method emerged for the task of single-image 3D reconstruction. This method leverages large scale synthetic and real datasets to train a large transformer model, dubbed 3D Large Reconstruction Model (LRM) [20, 19] , to generate 3D objects represented as neural radiance fields from a single image in a single forward pass -thus being extremely fast. We build on top of this idea to achieve fast and high quality 4D reconstruction.We present L4GM, the first 4D Large Reconstruction Model (Figure 2 ), which aims to reconstruct a sequence of 3D Gaussians [22] from a monocular video, in a feed-forward fashion. Key to our method, is a new large-scale dataset containing 12 million multiview videos of rendered animated 3D objects from Objaverse 1.0 [11] . Our model builds on top of LGM [49] , a pre-trained 3D Large Reconstruction Model that is trained to output 3D Gaussians from multiview images. We extend it to take a sequence of frames as input and produce a 3D Gaussian representation for each frame. We add temporal self-attention layers between the frames in order to learn a temporally consistent 3D representation. We upsample the output to a higher fps by training an interpolation model that takes two consecutive 3D Gaussian representations and outputs a fixed set of in-betweens. L4GM is trained on our multiview video dataset with per-timestep image reconstruction losses by rendering the Gaussians in multiple views.We showcase that although only trained on synthetic data, the model generalizes well to in-the-wild videos, e.g., videos generated by Sora [34] and real-world videos in AcitivityNet [12] . On the video-to-4D benchmark, we achieve state-of-the-art quality while being 100 to 1,000 times faster than other approaches. L4GM further enables fast video-to-4D generation in combination with a multiview generative model, e.g., ImageDream [53] .2 Related Work 
large 3d reconstruction models: 
 Reconstructing 3D representations from posed images typically requires a lengthy optimization process. Some works have proposed to greatly speed this up by training neural networks to directly learn the full reconstruction task in a way that generalizes to novel scenes [65, 55, 54, 57] . Recently, LRM [20] was among the first to utilize large-scale multiview datasets including Objaverse [11] to train a transformer-based model for NeRF reconstruction. The resulting model exhibits better generalization and higher quality reconstruction of object-centric 3D shapes from sparse posed images in a single model forward pass. Similar works have investigated changing the representation to Gaussian splatting [49, 68] , introducing architectural changes to support higher resolution [61, 44] , and extending the approach to 3D scenes [6, 7] . Methods such as LRM can generalize to input images supplied from sampling multiview diffusion models which enables fast 3D generation [24] .Recent works have made impressive progress in reconstructing dynamic 3D representations from multiview video inputs [28, 29] . However, for single-view video inputs, the problem of dynamic reconstruction becomes ill-posed and requires hand-crafted or data-driven priors. Good results have been achieved when targetting specific domains using shape templates [62] . Template-free methods typically require accurate depth inputs and cannot fill in portions of the object that are occluded or not visible [14, 60] .Recently, a few works have attempted to extend feed-forward generalizeable novel view synthesis to the challenging setting of dynamic monocular videos. DYST [43] introduces the synthetic 4D Dyso dataset which they use to train a transformer for generalizeable dynamic novel view synthesis but their model lacks an explicit 3D representation of the scene and camera. PGDVS [70] extends the generalizeable 3D NVS model GNT [54] to dynamic scenes but relies on consistent depth maps obtained through per-scene optimization. Other methods [50, 4] leverage pretraining but still require some amount of test-time finetuning to achieve acceptable novel view synthesis. Several works tackle generalizerable human shape reconstruction but rely on human template meshes [31, 23] .Dreamfusion [35] introduced the score distillation framework for text-to-3D shape generation. Such per-object optimization methods have also been extended to the 4D domain [47] where they leverage video diffusion models [46, 3, 30] . Some methods combine the guidance from video diffusion models with multiview [45] and single-image diffusion models [41] to boost 3D consistency and individual frame quality [72, 1, 26] , and they utilize different forms of score distillation [25, 66, 56] .By utilizing image conditional diffusion priors such as Zero123 [27] and ImageDream [53] as guidance, this approach has also been applied to the image-conditional setting [71] and video-conditional 4D generation setting by Consistent4D [21] and DreamGaussian4D [39] . GaussianFlow [15] introduces a Gaussian dynamics based representation which supports them to add optical flow estimation as an additional regularization to SDS. STAG4D [67] and 4DGen [64] use diffusion priors to sample additional pseudo-labels from "anchor" views to expand the set of reference images for imageconditional SDS and as direct photometric reconstruction loss.To accelerate the generation process, some works eschew the use of score distillation guidance altogether. For reconstruction from monocular video input, Efficient4D [33] and Diffusion 2 [63] utilize a two stage approach. In the first stage they craft schemes for sampling multiview videos conditioned on the input video. Standard optimization-based reconstruction is then used for stage 2. However, this optimization process can still take on the order of tens of minutes. In this work we directly train a feed forward 4D reconstruction model instead.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Appendix I Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: 
 Our L4GM builds on the success of single-image 3D reconstruction models [20, 19, 49] , specifically the Large Multi-View Gaussian Model (LGM) [49] . LGM accepts a set of multiview images of an object and directly outputs a 3D reconstruction of the object, represented by a set of Gaussian ellipsoids P . Each Gaussian ellipsoid is represented by 14 parameters, including a center z ∈ R 3 , a scaling factor s ∈ R 3 , a quaternion rotation q ∈ R 4 , an opacity α ∈ R, and a color featurec ∈ R 3 . The multiview images J = {J v } V v=1 are taken from V camera poses O = {O v } V v=1. These camera poses are encoded as image embeddings via Plücker ray embeddings and concatenated to the RGB channels of the multiview images as input to the model. They are fed through an asymmetric U-Net [42] yielding V 14-channel image feature maps, where each of the pixels will be interpreted as the parameters of a 3D Gaussian.When only a single input image is given, an image-conditional multiview diffusion model such as ImageDream [53] is first used to generate plausible completions for the missing multiview images. These generated views are then fed to LGM for reconstruction.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
our method: 
 Given a monocular video of a dynamic object, denoted as I = {I t } T t=1 where T is video length, our objective is to rapidly reconstruct an accurate 4D representation of the object. Our approach is grounded in two conceptually simple yet impactful insights.Our inspiration stems from video diffusion models. Recent advances in video generation have highlighted the benefits of first pretraining on image data and then extending and finetuning the model on video datasets to effectively model temporal consistency [3, 16, 59, 2] . Similarly, recognizing the scarcity of 4D data, we want to leverage a pre-trained Large Multi-View Gaussian Model (LGM) [49] that operates on images and has been extensively trained on a large-scale 3D dataset of static objects. This strategy leverages the robustness of pre-trained models to effectively train a 4D reconstruction model with limited data.Secondly, in constrast to most existing methods [28, 29] that are required to use multiview videos for 4D reconstruction, we found that utilizing a single set of multiview images at the initial timestep is sufficient. We can obtain these multiview images easily by leveraging multiview image diffusion models to expand the first frame of the view. By adding temporal self-attention layers, our model capitalizes on the initial multiview input by propagating and adapting this information across subsequent timesteps (subsection 4.2). This approach significantly reduces the computational complexity and challenges typically associated with generating consistent multiview videos, while still enhancing the quality of the reconstruction, as our results demonstrate. Thus, in the following, we introduce L4GM, a model that processes a monocular video to output a set of 3D Gaussians for each timestep, denoted by P = {P t } T t=1 , where each P t is a set of 3D Gaussians at time t. L4GM is an extension of a pretrained 3D LGM, enhanced with temporal self-attention layers for dynamic modeling. We generate V multiview images based on the first frame of the input monocular video. These generated views, along with the input video, are fed into L4GM to reconstruct the entire 4D sequence. We also explore further finetuning L4GM into a 4D interpolation model, allowing us to generate 4D scenes at a higher FPS than the monocular input video, offering smoother and more detailed motion dynamics within the 4D reconstructions.Similar to the single-image scenario of LGM, we use ImageDream [53] to generate four orthogonal views conditioned on the initial frame I 1 . We denote J 1 as the set of generated multiview images taken from camera poses O at the initial time step t = 1. We would like our generated multiview images to contain three extra views that are orthogonal to the input first frame of the original video.However, often none of the viewing angles of the generated multiview images match the input frame I 1 . An example can be found in Appendix Figure 8 . To address this, we first use the 3DLGM to reconstruct an initial set of 3D Gaussians, P init , from the generated multiview images, and render this reconstruction from views that are orthogonal to I 1 as desired. We provide exact details in Appendix E.LGM into a 4D Reconstruction Model Model Architecture. We adopt the asymmetric U-Net [42] structure from the pretrained LGM as backbone. To align with LGM's input specifications, we replicate the generated multiview images in J 1 at t = 1 (except I 1 ) across all other time steps to construct a T × V grid (see left side of Figure 2 for an example). For simplicity, we assume the camera in the reference monocular video is static and only the object is moving so we also copy the camera poses O (except O 1 ) across time steps. Similar to LGM, these poses are then embedded using Plücker ray embeddings [48] . We concatenate this camera embedding with the RGB channels of the input images. The concatenated inputs are reshaped into the format (B T V) H W C and fed into the asymmetric U-Net, where B is batch size.As illustrated in the middle section of Fig 2 , each U-Net block within L4GM consists of multiple residual blocks [18] , followed by cross-view self-attention [52] layers. To maintain temporal consistency across different timestamps, we introduce a new temporal self-attention layer following each cross-view self-attention layer. These temporal self-attention layers treat the view axis V as a batch of independent videos (by transferring the view axis into the batch dimension). After processing, the data is reshaped back to its original configuration. In einops [40] notation, this process looks as:EQUATIONwhere x is the feature, B H W C are batch size, height, width, and the number of channels.The output of the U-Net consists of 14-channel feature maps with shape B ×T ×V ×H out ×W out ×14.Each of the 1 × 14 units is treated as the set of parameters of a per-pixel Gaussian. We concatenate these Gaussians along the view dimension V to form a single set of Gaussians for each timestamp, resulting in T sets of 3D Gaussians, {P t } T t=1 . This collection forms our final 4D representation.Loss Functions. Besides the input camera poses O, we select another set of camera poses O sup for multiview supervision. We train the model with a simple reconstruction objective on the video rendering of the output 4D representations from camera poses O ∪ O sup , as detailed in Appendix B.In practice, one may want to apply L4GM on long videos and obtain temporally smooth 4D outputs. To this end, L4GM also enables autoregressive reconstruction which processes videos in an autoregressive fashion, one chunk of T frames after another. We additionally train a 4D Interpolation Model to upsample the 4D representation to a higher fps. Details can be found in Section A.Autoregressive Reconstruction (Figure 3 , left). Our base model is designed to accept a monocular video of fixed length T . For long videos that exceed T , we partition the video into chunks of size T which we process sequentially. We first apply L4GM to the initial T frames to generate the first set of T Gaussians. Subsequently, instead of generating multiview images for the first frame of the next chunk using a multiview diffusion model, we render the last set of Gaussians from four orthogonal angles to obtain new multiview images. These newly rendered multiview images, along with the next set of video frames, are then used to generate the next set of Gaussians. The process is repeated until all frames of a long video have been reconstructed. We empirically observe that such an autoregressive reconstruction method can be repeated more than 10 times without a significant drop in quality.4D Interpolation Model (Figure 3 , right). As our model does not track Gaussians across frames, directly interpolating Gaussian trajectories is not feasible [29, 58] . Hence, we develop an interpolation model that operates in 4D, fine-tuned on top of L4GM. Similar interpolation methods have been used successfully in the video generation literature [3] . As shown in Figure 3 , the input to the 4D Interpolation Model consists of two sets of multiview images and the interpolation model is trained to produce additional intermediate sets of Gaussians. It leverages the weight-average of the RGB pixels between the multiview images for the newly created intermediate frames. The 4D Interpolation Model then outputs the corresponding sets of Gaussians. In practice, we insert two additional time frames. 
objaverse-4d dataset: 
 Dataset Collection To collect a large-scale dataset for the 4D reconstruction task, we render all animated objects in Objaverse 1.0 [11] .Out of 800,000 objects, only 44,000 have animations. Since each object can have multiple associated animations, we obtain a total of 110,000 animations. All of the animations are 24 fps.The dataset consists mostly of animations on rigged characters or objects, e.g., "a T-Rex walking" or "a windmill spinning". The motions include diverse scenarios such as dynamic locomotions, fine-grained facial expressions, and smoke effects. Interestingly, there are a considerable amount of deforming motions in the dataset, since many animations feature a fantasy style -thus, even rigid real-world objects are deformable. See Appendix Figure 7 and supplementary video for examples.t 6 ExperimentsRGB interpolated frames 4D Interpolation Model L4GM … … 𝒥 ! 𝐼 " 𝐼 # 𝐼 $ 𝑃 " 𝑃 # 𝑃 $ 𝑃 ! 𝐼 % L4GM … … 𝐼 $&! 𝐼 $&" 𝐼 "$'! 𝑃 $'! 𝑃 $'" 𝑃 "$'! 𝑃 $ 𝐼 ( 𝒥 $ render 𝑃 ) 𝑃 )&# render render 𝒥 ) 𝒥 )&# 𝑃 ) 𝑃 )&# 𝑃 )&! 𝑃 )&" 
implementation details: 
 We rendered the dataset with Blender and the EEVEE engine [9] . We used fixed camera intrinsics and lighting as detailed in the appendix. For L4GM, we downsample the clips to 8 FPS and train the model for 200 epochs. In training, we set T = 8 and use 4 input cameras and 4 supervision cameras. During inference, we used T = 16, which we empirically show to work well for longer videos in section 6.3. Each forward pass through L4GM takes about 0.3 seconds, while generating sparse views requires about 2 seconds. For training the interpolation model, we use the 24 FPS clips without downsampling and fine-tune L4GM for another 100 epochs. The 4D interpolation model takes 0.065 seconds to interpolate between every two frames (see Appendix for details).We focus our evaluations on video-to-4D reconstruction. Although L4GM can also be used for text-to-4D or image-to-4D synthesis by taking text-to-video or image-to-video outputs from video generative models as input, existing text-to-4D [26, 1, 46] and image-to-4D [71] approaches typically rely on score distillation and are orders of magnitudes slower than L4GM, preventing meaningful comparisons. Hence, we concentrate on video-to-4D. Results are presented in the following paragraphs.Quantitative Evaluation. We evaluate our L4GM on the benchmark provided by Consistent4D [21] . It consists of eight dynamic 3D animations of 4 seconds in length, at 8-FPS. A video from one view is used as input and 4 videos from other viewpoints are used for evaluation. Three metrics are computed: 1) Perceptual similarity (LPIPS) between the generated and the ground truth novel views, 2) the CLIP [38] image similarity between the generated and the ground truth novel views, and 3) the FVD [51] against the ground truth novel views , which measures video quality. We also report the runtime. We reconstruct the video at the original frame rate without using interpolation model. The results are shown in Table 1 . L4GM outperforms existing video-to-4D generation approaches on all quality metrics by a significant margin, while being 100 to 1,000 times faster. These examples are taken from the Activi-tyNet [12] and Consistent4D [21] datasets. As shown in the figure, L4GM produces highquality, sharp renderings while exhibiting strong temporal and multiview consistency.We compare our visual results to DG4D [39] , STAG4D [67] , and OpenLRM [19] . DG4D and STAG4D are optimization-based approaches that take 10 minutes and 2 hours on 64-frame videos, respectively. OpenLRM is an opensource work reproducing LRM [20] that reconstructs 3D shapes from single-view images. We run OpenLRM on every video frame to construct a 3D sequence. We collect 24 evaluation videos from Emu [16] , Sora [34] , Veo [10] , and ActivityNet [12] , covering both generated videos and real-world videos.For our approach, we use T = 16 and use the interpolation model. Some qualitative comparisons are presented in Figure 5 . Notably, a significant improvement from our approach is a higher 3D resolution. Optimization-based approaches use only thousands of Gaussians to keep the optimization tractable, while our feed-forward approach can easily reconstruct more than 60,000 Gaussians per frame at a dramatically faster speed.We further conduct a user study based on qualitative comparisons, and the results are shown in Table 2 .Our approach is the most favorable on all evaluation criteria including overall quality, 3D appearance, 3D alignment with input video, motion alignment with input video, and motion realism. More details about the evaluation dataset and the user study are in Appendix G.We carry out a variety of ablation studies. For training models in the ablation study, we only keep animations from high-quality objects in GObjaverse [37] , which accounts for ≈25% of the data.3D Pretraining. Without 3D pre-taining, i.e. without initializing from LGM [49] , our model fails to converge (using the same training recipe). The explanation is likely that without large-scale pre-training on static 3D scenes our Objaverse-4D dataset is insufficient for L4GM to not only learn temporal dynamics but also its 3D understanding from scratch. Moreover, starting from a fully random initializion may also contribute to training instabilities. Note that when reducing the model size to the "small" LGM variation [49] , the model starts to converge. However, as shown in Frozen LGM. In this experiment, we freeze the layers of the original LGM model and only train the temporal attention layers. As shown in Table 6 (b), the model improves faster at the beginning of the training but converges to a lower PSNR. This suggests that end-to-end fine-tuning of L4GM, including the non-temporal layers of the 3D LGM, is preferable for the 4D reconstruction task.Temporal Attention. Without temporal attention, the model falls back into a 3D reconstruction model, while receiving asynchronous multiview inputs. The model does a surprisingly good job using only the 3D information, but it still converges to a lower PSNR, as shown in Table 6 ( 6 (c). When the groundtruth multiviews are provided, the quality slightly drops when using longer video length. When reconstructing autoregressively, the quality decreases with more autoregressive runs. A shorter video length will start with a higher quality, but the quality drops faster than a longer video length because more self-reconstructions are required. In practice, we select T = 16, which offers a balanced performance for different video lengths.Time Embedding. Here, we explore adding a time embedding to L4GM so that the model is aware of the ordering of the frames. Timestamps are encoded by a sinusoidal function and added to the camera embedding. However, the PSNR did not improve after adding the time embedding. We speculate that the image frames from the input video are already giving sufficient information about the temporal relation between the timesteps. Therefore, we do not train with temporal embedding to add more flexibility to the model at inference time.4D Interpolation. Finally, we show a comparison of using vs. not using the 4D interpolation model on an 8-FPS video from the Consistent4D dataset in the supplementary video. Notably, the 4D interpolation model can successfully improve the framerate beyond the input video framerate. 
conclusions: 
 We presented L4GM, the first large reconstruction model for dynamic objects. It produces dynamic sequences of sets of 3D Gaussians from a single-view video. The model leverages prior 3D knowledge from a pretrained 3D reconstruction model, and learns the temporal dynamics from a synthetic dynamic dense-view 4D dataset, Objaverse-4D dataset, which we collect. L4GM can reconstruct long videos and uses learned interpolation to achieve high framerates. We achieve orders of magnitude faster inference times than existing 4D reconstruction or text-to-4D methods. Moreover, our model generalizes well to in-the-wild real and generated videos. Our work is an early attempt at AI tooling for 4D content creation, with many challenges remaining. For example, for making this technology really useful for professionals we need to develop more advanced human-in-the-loop 4D editing capabilities.Broader Impact. L4GM allows fast 4D reconstruction from in-the-wild videos, which is useful for various graphics and animation applications. However, this model should be used with an abundance of caution to prevent malicious impersonations. The model is also trained on non-commercial public datasets and is for research-only purpose.[72] Y. Zheng, X. where the frame length L significantly exceeds T . We first apply L4GM to the initial T frames to generate the first set of T Gaussians {P t } T t=1 . Subsequently, we render the last Gaussian P T from four orthogonal angles to obtain new generated multiview images J T = {f (P T , ∆θ)} ∆θ∈{0 • ,90 • ,180 • ,270 • } . The rendered multiview images J T , along with frames I = {I t } 2T -1 t=T , are used to construct the next set of T -1 Gaussians {P t } 2T -1 t=T +1 . This process is repeated until all L frames have been reconstructed. We find that this autoregressive reconstruction method can be repeated more than 10 times without a significant drop in quality (see Figure 3 , left).4D Interpolation Model. Since our model does not track Gaussians across frames, interpolating Gaussian trajectories is not feasible [29, 58] . Hence, we developed an interpolation model that operates directly within the 4D space, fine-tuning it on top of L4GM. As demonstrated in Figure 3 , right side, the input to the 4D Interpolation Model consists of two sets of multiview images, J i and J i+3 . During training, we render J i and J i+3 at 8 FPS from an asset in our dataset. During inference, J i and J i+3 are rendered from reconstructed sets of gaussians P i and P i+3 . To create the T × V input grid, we insert two time steps between the input views by simply calculating the weighted-average on the RGB pixels between the multiview images J i and J i+3 , resulting in a total of 4 × V images. The 4D Interpolation Model then outputs the corresponding four sets of Gaussians, which are supervised using the ground truth data at 24 FPS. During inference, we render the reconstruction results into multiview videos and feed them into the interpolation model to produce a sequence of Gaussians at a three times higher framerate. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: section 7Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Appendix B Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Dataset. We render the dataset with the EEVEE engine in Blender [9] , which requires three days with 200 GPUs. For random cameras, we select a random elevation from [-5 • , 60 • ]. For the fixed camera setting, we rendered from 16 views equally distributed at 0 • elevation looking at the origin. We set the camera radius to 1.5 to align with LGM so that we can leverage its 3D pretrain. The camera FOV is set to 49.1 • . We use the lighting from the Objaverse-XL codebase 1 while fixing the strength to the mean value. Following Blattmann et al. [2] , we also employ motion filtering. Specifically, we downsample the 4D animation clips of Objaverse-4D dataset to 2-FPS and compute the optical flow magnitude averaged on videos rendered from four orthogonal fixed 0-elevation cameras. A histogram of the optical flow magnitude is shown in Table 3 . We only keep the animation clips with an average optical flow magnitude larger than 0.15, resulting in 51K 4D animations with 28K different objects, 246K 1-second clips, and 12M videos rendered from 48 camera poses.Model. We use the pretrained LGM model [49] released in the official code2 , which has 6 downsampling blocks with channels [64, 128, 256, 512, 1024, 1024] 3 : Optical flow magnitude histogram on Objaverse-4D dataset.Loss Functions. Following LGM, we use a combination of an LPIPS [69] loss and an MSE loss on RGB images, and an MSE loss on segmentation masks to supervise the reconstruction,EQUATIONEQUATIONEQUATIONwhere f, g are Gaussian volume rendering functions for RGB and alpha masks, I O t and α O t are the image and mask rendered from camera pose O at time t, and λ is a loss weight.Training. For training L4GM, we downsample the videos to 8 FPS and set T = 8, V = 4. We apply the grid distortion augmentation [49] to non-reference views to improve the model robustness. We train the model with one 8-frame clip per GPU on 128 80G A100 GPUs. In each epoch, we iterate through all animations and sample a one-second clip from it regardless of the animation length. The model is trained for 200 epochs, which takes about one day. For ablation models, we reduce the dataset size to the 25% GObjaverse subset and reduce the number of GPUs to 32 correspondingly while keeping other settings unchanged. For training the interpolation model, we fine-tune L4GM with 4 frames per GPU on 64 80G A100 GPUs for 100 epochs.Inference. We test on a 16G RTX 4080 Super GPU. We set the video length to T = 16 Each forward pass through L4GM takes about 0.3 seconds while generating sparse views requires about 2 seconds, including ImageDream generation, LGM reconstruction, and azimuth selection. The 4D interpolation model takes 0.065 seconds to interpolate between every two frames. For example, for a 10-second 30-FPS video, we can reconstruct the input video in 15 FPS and interpolate the result to 45 FPS in 15 seconds, consisting of 2 seconds for sparse view generation, 3 seconds for reconstruction, and 10 seconds for interpolation. Video segmentation time is not included.We show example training data in Figure 7 .We present the detailed metric for every test sample on the Consistent4D benchmark in Table 4 . After getting P init , we render P init from a series of azimuth angles θ ∈ {-180 • , ..., 180 • } and select the azimuth that best aligns with the input frame. This is determined by θ align = argmin θ ||f (P static , θ) -I 1 || 2 2, where f is a Gaussian volume rendering function. The final utilized sparse views, J 1 , are defined as {f (P static , θ align + ∆θ)} ∆θ∈{0 • ,90 • ,180 • ,270 • } . An illustration is shown in Figure 8 .In this ablation study, we predict a canonical Gaussian and a deformation field represented by HexPlane. An additional decoder decodes the output from the U-Net middle block into 6 planes. The decoder is also equipped with cross-view and temporal attention. The canonical Gaussian then goes through the deformation field to produce T sets of 3D Gaussians. The decoder has channels [1024, 128, 128] . We show an illustration in Figure 9 . We collect 24 evaluation videos from multiple sources, including 1) 10 videos from Emu video [16] ; these are generated 4-second 16-FPS videos. 2) 7 videos from Sora [34] ; these are generated long 30-FPS videos. 3) 5 videos from Veo [10] ; these are generated long 30-FPS videos. 3) 2 videos from ActivityNet [12] ; these are real-world long videos of weight lifting.We segment the foreground object with SAM-Track [8] and then crop and rescale the video to 512x512. Since the optimization-based baseline methods are very memory intensive, we trim all videos to 4 seconds and downsample all 30 FPS videos to 15 FPS, so that all videos have 64 frames.We conducted human evaluations (user studies) through Amazon Mechanical Turk to assess the quality of our generated 4D scenes, comparing them with DG4D [39] , OpenLRM [19] , STAG4D [67] and performing ablation studies. We used the 24 examples evaluation dataset as described in Section 6.2. We rendered both baselines and our dynamic 4D scenes from similar camera perspectives and created similar videos. We first asked the participants to watch the reference input video and then asked them to compare the two videos with respect to 5 different evaluation axes and indicate a preference for one of the methods with an option to vote for 'equally good' in a non-forced-choice format. The 5 categories measure overall quality, 3D appearance quality, as well as motion alignment to reference video and motion realism.For a visual reference, see 10 for a screenshot of the evaluation interface. In all user studies, the order of video pairs (A-B) was randomized for each question. In all user studies, each video pair was evaluated by five participants, totaling 120 responses for each of the baseline comparisons. We selected participants based on specific criteria: they had to be from English-speaking countries, have an acceptance rate above 95%, and have completed over 1000 approved tasks on the platform. H More Qualitative ResultsWe show more qualitative comparisons with state-of-the-art approaches on both generated (Figure 11 ) and real-world (Figure 12 ) videos.We show more qualitative results on the Consistent4D dataset in Figure 13 .We show some limitations of L4GM in Figure 14 . Our model can not handle motion ambiguity well. For example, in some walking motions, the model can successfully align with the reference view but the leg motion is not natural from other views. The model also cannot reconstruct multiple objects well, particularly when they occlude each other. Finally, the model fails to reconstruct objects from an ego-centric viewpoint, since the model assumes input views to be taken from 0 • elevation. Ablation results on the Consistent4D benchmark. In Table 5 , we evaluate ablation models and interpolation models on the Consistent4D. Note that the quantitative results can be noisy since the benchmark only has 7 test samples. Comparing b) to a), we confirm that the pretrain plays a critical role. Comparing f) to a), we observe that the 4D representation is crucial. To evaluate the 4D interpolation model, we downsample the evaluation videos' framerate by 3, reconstruct downsampled videos with a), and then interpolate the 4D reconstruction with our interpolation model. Notably, g) achieves an on-par performance to a), which suggests the effectiveness of the interpolation model. Full attention. We explore replacing the temporal attention in L4GM with full attention. Full attention computes the self-attention on all images regardless of time and view, while temporal attention only computes the self-attention on images under the same viewpoint. As a result, full attention requires more computing. We show a memory-time analysis on a training iteration in Table 6 , implemented with Efficient Attention. c) requires a larger memory usage and a longer processing time than b). In Figure 15 , We further show a PSNR plot that compares temporal attention with full attention, trained on the GObjaverse subset. They achieve almost identical PSNR curves. Considering both the computation cost and empirical results, temporal attention would be a better design choice. New evaluation on the GObjaverse subset We provide quantitative results using the GObjaverse subset as training data in Table 5 line h. We manually verified that these Consistent4D testing samples are not part of the GObjaverse subset. The new results remain state-of-the-art and show no significant difference from the numbers reported in the main paper. Justification: section 1Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/allenai/objaverse-xlhttps://github.com/3DTopia/LGM 


Decision: The work received positive feedback from reviews, highlighting the impressive results, clear writing and acknowledging that this is the first work presenting a feed forward approach for 4D reconstruction from video. Slight concerns remained with respect to technical novelty. All in all though, the reviewers believe that this paper is of value to the community, due to the currently relevant topic and state of the art results.

I agree and follow with my accept recommendation.

Congratulations to the authors!

Reviews: review: paper_summary: paper_summary: This paper proposes a model for 4D reconstruction from a single video, building upon dynamic 3D gaussians and LGM architecture [49] previously applied to static 3D scenes. By processing generated multi-view images (derived from the first frame using prior method) alongside the video, the model outputs 4D Gaussians to reconstruct the video dynamics. The model was trained on a synthetic multi-view video dataset (Objaverse) and shows qualitative generalization to real-world images. Based on the good performance and extensive ablation studies, I recommend accepting this paper.
strength_weakness: strength_weakness: - Claiming that model generalises “extremely well” to in-the-wild lacks empirical support (apart from cherry-picked qualitative results) and likely not true due to training assumptions (e.g. masks, static camera at 0 degree elevation). One possible evaluation to substantiate the claim would be a comparison with other works such as HyperNeRF. 
- Since the code is not available, the details about the architecture and training are not sufficient for reproducing experiments and should be expanded. E.g. I could not find how exactly 32 heldout views are sampled neither in supplementary nor in the main paper. 
- Minor: The architecture and training are not new and relies heavily on prior techniques, including a multi-view image generator. This perhaps aligns work more closely with the computer vision and engineering community (e.g., CVPR)
questions: questions: - Could you provide more information about various axis of failure cases and the generalization limits to in-the-wild videos? 
- Could you provide more details about "grid distortion" mentioned on line 594? The referenced paper also lacks clarity on this.
- Given the use of 128 80GB A100 GPUs, could you provide details on training the model with fewer resources? (particularly memory use)
limitations: limitations: Authors have acknowledged the limitations, however, the claim about generalizing "extremely well" on in-the-wild videos is unsubstantated and likely not true due to training assumptions (e.g. masks, static camera at 0 degree elevation).
paper_summary: paper_summary: This work introduces a novel framework for generating animated 3D objects from single-view videos. The proposed framework employs a feed-forward approach, thus eliminating the need for computationally expensive optimization. The core idea is to create a large-scale synthetic multi-view video dataset and train a corresponding 4D LGM model, which is initialized from the 3D LGM [49] and enhanced with temporal self-attentions. Additionally, an interpolation network is introduced to improve the frame rate further. Overall, this method achieves superior performance in producing 4D assets, both in terms of efficiency and reconstruction quality.
strength_weakness: strength_weakness: Technical contributions

- Despite its impressive performance, this work's technical contributions are somewhat limited. The two main contributions are temporal self-attention and the 4D interpolation network.
- The temporal self-attention can be seen as a direct extension of the multi-view self-attention proposed in [49].
- While the 4D interpolation network can speed up inference, its overall performance does not seem to have a substantial impact, as demonstrated in the supplementary material video.

Having said that, I agree that a straightforward solution to a new problem should be recognized, and the above weakness is relatively minor.
questions: questions: Some Questions:

- Regarding the temporal self-attention layer, why is only the time dimension considered in the self-attention mechanism rather than incorporating both multi-view and temporal information (full attention)? An analysis focusing on efficiency and performance would be valuable to understand the rationale behind this design choice.
- Why is the ablation study performed using PSNR (a reconstruction-based metric) instead of the metrics provided in Table 1?
- It appears that only qualitative comparisons are offered for the interpolation network. Could the performance be evaluated using the metrics shown in Table 1?
- Given that this work is an original contribution to the feed-forward generation of 4D assets, the community would greatly benefit from the release of the dataset and code to enable the reproduction of the work. Are there any plans to release these resources?
limitations: limitations: - This work also necessitates a "synthetic" multi-view video dataset for training. According to the implementation details, generating this dataset takes approximately 3 days using 200 GPUs. While this is easier to obtain than a real dataset, it still demands significant computational resources. Including a more comprehensive discussion on this in the limitations section would be beneficial.
- The proposed framework is claimed to generalize extremely well to in-the-wild videos (Abstract). However, there are only a few samples provided (4 samples in Figures 1 & 4), and many of these are in an animated style, with the exception of the left example in Figure 4. This left example displays various artifacts on the produced objects, such as blurred arms. Given the current evidence, this claim appears to be inadequately supported, and it is recommended to either include more samples or lower the claim accordingly.
paper_summary: paper_summary: This paper utilizes rendering of animated objects from objverse(-xl) to extend lgm into 4D generation. Specifically, L4GM uses four orthogonal images of an object and the object's monocular dynamic video to obtain 3D Gaussians at each moment, enhancing the consistency between different moments through temporal self-attention layers. Subsequently, the smoothness of generated actions is further improved through a 4D Interpolation Model. L4GM leads in generation speed and evaluation metrics in the task of video-to-4D.
strength_weakness: strength_weakness: The paper discusses extensively how to use dynamic datasets for pre-training, which is also a very important part of this work and will have a significant impact on the community. Whether this dataset is open source is also extremely important for evaluating this work, but the paper does not mention this point.
questions: questions: 1. In section 5, it is mentioned that some objects come with animations. Does this mean the objects already have predefined animations?
2. In the supplementary materials, I noticed that some generated 4D objects have limited motion range and motion rationality, which is a common issue in other 4D generation works. Since evaluating these is a resource-intensive task, are there any possible evaluation methods that could be discussed? This could be part of future work, and once an evaluation method is established, the motion quality of 4D generation could potentially be further improved.
limitations: limitations: The main issue lies in the amplitude and controllability of the animations. Is there a potential direction for this in the future?
paper_summary: paper_summary: This paper proposed L4GM, an efficient large 4D Gaussian reconstruction model to produce animated objects from videos by a single feed-forward. L4GM leverages ImageDream and LGM to achieve multiview images of the first frame as the input. The overall model is built upon the pre-trained LGM with cross-view and temporal attention blocks, which could be easily modified by rearranging the feature dimension. Besides, this paper proposed autoregressive reconstruction and 4D interpolation, largely improving the overall smoothness of the generated 4D. To train L4GM, the authors rendered animated objects from Objaverse, and conducted sufficient experiments to verify the effectiveness.
strength_weakness: strength_weakness: 1. The main concern is the novelty. Although the overall pipeline is convincing, this paper is more like an extension of LGM for 4D generation. Most techniques are very straight-forward, such as temporal and cross-view attention, and multi-view synthesis by ImageDream+LGM.

2. Another concern is the setting of repeating the multiview images from the initial timestep as inputs for other frames. It works more like seeking temporary relief rather than a solid solution. For example, this repeating suffers from solving 4D generation with the motion of turning around, whose multiview images would contain conflict to the inputs. Why not use ImageDream+LGM to achieve more multiview inputs? Moreover, more quantitative and qualitative ablation studies should be considered for this setting, such as no-repeating views, repeating views, and multi-timestep-multi-view inputs respectively.
questions: questions: 1. Maybe there is a typo of temp. embed. in Figure 6(b), which should be time embed?
2. Why there are 12M videos at all? The authors said that 110k animations are captured with 48 views. So the overall videos should be about 5M.
3. Since L4GM is trained with animated objects from Objaverse,  the comparison of Consistent4D in Table 1 is not convincing enough. To the best of my knowledge, most objects in the Consistent4D test set are from Objaverse too. Please refer to https://github.com/yanqinJiang/Consistent4D/blob/main/test_dataset_uid.txt for more details.
limitations: limitations: The authors have discussed limitations in the supplementary.

Comments: title: Thanks to Reviewer UkTx
Thanks to Reviewer 2k2z
Thanks for the rebuttal
Thanks to Reviewer hQSp
comment: Thank you for your response.
Thank you for the response. We are glad to have your concerns clarified.
Thank you for raising the score. We appreciate your suggestions and will include them in the revision.
Thank you very much for the additional experiments and comments. These have addressed my concerns. I am maintaining my positive stance towards the current manuscript.
Thanks for the rebuttal. The rebuttal addressed most of my concerns. I raise my score to 5. Importantly, the limitation and related discussion about repeating views should be included in the revision.
We are glad to hear that our response addressed your concerns. Thank you for raising the score.
The author addressed my concerns. I agree that the animation is determined by the video. Thanks a lot. I will raise my score from borderline accept to weak accept.