GenXD: Generating Any 3D and 4D Scenes
Authors: Yuyang Zhao
Chung-Ching Lin
Kevin Lin
Zhiwen Yan
Linjie Li
Zhengyuan Yang
Jianfeng Wang
Gim Hee Lee
Lijuan Wang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Generation; 4D Generation; Diffusion Models
Primary area: generative models

Abstract: Recent developments in 2D visual generation have been remarkably successful. However, 3D and 4D generation remain challenging in real-world applications due to the lack of large-scale 4D data and effective model design. In this paper, we propose to jointly investigate general 3D and 4D generation by leveraging camera and object movements commonly observed in daily life. Due to the lack of real-world 4D data in the community, we first propose a data curation pipeline to obtain camera poses and object motion strength from videos. Based on this pipeline, we introduce a large-scale real-world 4D scene dataset: CamVid-30K. By leveraging all the 3D and 4D data, we develop our framework, GenXD, which allows us to produce any 3D or 4D scene. We propose multiview-temporal modules, which disentangle camera and object movements, to seamlessly learn from both 3D and 4D data. Additionally, GenXD employs masked latent conditions to support a variety of conditioning views. GenXD can generate videos that follow the camera trajectory as well as consistent 3D views that can be lifted into 3D representations. We perform extensive evaluations across various real-world and synthetic datasets, demonstrating GenXD's effectiveness and versatility compared to previous methods in 3D and 4D generation.

Published as a conference paper at ICLR 2025
GEN GENERATING ANY 3D AND 4D SCENES
Yuyang " Chung-Ching Lin Kevin Lin Zhiwen Yan Linjie Li Zhengyuan Yang". Jianfeng Wang Gim Hee Lee*, Lijuan Wang National University of Singapore, https: /tt/////ennee x d. d.g.tbub..ii io
Microsoft Corporation
Single View 3D
Gen XD
Single View 4D Multi- View 4D
Multi-View 3D
Figure I: GenA'D unified model for high-quality 3D and 4D generation from any number of condition images. By controlling the motion strength and condition masks, GenA'D support various application without any modification. The condition images are shown with star icon and the time dimension is illustrated with dash line.
ABSTRACT
Recent developments in 2D visual generation have been remarkably successful. However, 3D and 4D generation remain challenging in real-world applications due to the lack of large-scale 4D data and effective model design. In this pa- per, propose to jointly investigate general 3D and 4D generation by leveraging camera and object movements commonly observed in daily life. Due to the lack of real-world 4D data in the community, we first propose data curation pipeline to obtain camera poses and object motion strength from videos. Based on this pipeline, introduce large-scale real-world 4D seene dataset: Cam Vid-30K. By leveraging all the 3D and 4D data, we develop our framework, Gen which allows to produce any 3D or 4D scene. We propose multiview-temporal mod- ules, which disentangle camera and object movements, to seamlessly learn from both 3D and 4D data. Additionally, GenA'DD employs masked latent conditions support variety of conditioning views. GenA'D can generate videas that follow the camera trajectory as well as consistent 3D views that can be lifted into 3D representations. We perform extensive evaluations across various real-world and synthetic datasets, demonstrating Gen ' effectiveness and versatility compared to previous methods in 3D and 4D generation.

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
Generating 2D visual content has achieved remarkable success with diffusion (Rombach et al., 2022; Betker et al., 2023; Esser et al., 2024; Blattmann et al., 2023) and autoregressive modeling (Tian et al., 2024; Sun et al., 2024; Kondratyuk et al., 2023; Luo et al., 2024), which have already been used in real-world applications, benefiting society. In addition to 2D generation, 31D content generation is also of vital importance, with applications in video games, visual effects, and wearable mixed reality devices. However, due to the complexity of 3D modeling and the limitations of 3D data, 3D content generation is still far from satisfactory and is attracting more attention. In this paper, we focus on the unified generation of 3D and 4D content. Specifically, static 3D content involves only spatial view changes, referred to as 3D generation in this paper. In contrast, dynamic 3D content includes movable objects within the scene, requiring the modeling of both spatial view and dynamic (temporal) changes, which we term 4D generation.
Most previous works (Liu et al., 2023b; Shi et al., 2023; Zhao et al., 2023; Xie et al., 2024; Tang et al., 2024; 2023) focus on 3D and 4D generation using synthetic object data. Synthetic object data are typically meshes, allowing researchers to render images and other 3D information (e.g., normals and depth) from any viewpoint. However, object generation is more beneficial to specialists than to the general public. In contrast, scene-level generation can help everyone enhance their images and videos with richer content. As result, recent works (Gao et al., 2024; Wu et al., 2024b) have explored general 3D generation (both scene-level and object-level) in a single model, achieving impressive performance. Nonetheless, these works focus solely on static 3D generation, without addressing dynamics. In this paper, we propose a unified framework for general 3D and 4D generation, enabling the generation of images from different viewpoints and timesteps with any number of conditioning images (Fig. 1).
The first and foremost challenge in 4D generation the lack of general 4D data. In this work, we propose Cam Vid-30K, which contains upproximately 30K 4D data samples. 4D data require both mulli-view spatial information and temporal dynamics, so we turn to video data to obtain the necessary 4D data. Specifically, we need two key attributes from the video: the camera pose for each frame and the presence of movable objects. To achieve this, we first estimate the possible movable objects in the video using a segmentation model and then estimate the camera pose using keypoints in the static parts of the scene. While successful camera pose estimation ensures multiple views, we also need to ensure that moving objects are present in the video, rather than purely static scenes. To address this, we propose an object motion field that leverages aligned depth to estimate true object movement in the 2D view. Based on the object motion field, we filtered out static scenes, resulting in approximately 30K videos with camera poses.
In addition, we propose unified framework, Gen to handle 3D and 4D generation within single model. While there are similarities between 3D and 4D data in terms of their representation of spatial information, they differ in how they capture temporal information. Therefore, 3D and 4D generation can complement each other through the disentanglement of spatial and temporal informa- tion (see Appendix. G for details). To achieve this, we combine both 3D and 4D data during model training. To disentangle the spatial and temporal information, we introduce multiview-tempon modules in Gen each module, we use a--ussng to merge spatial and temporal information for 4D data, while removing temporal information for 3D data. Previous works (Xu et al., 2024; Voleti ct al., 2024) typically use a fixed number of conditioning images (e.g., the first image). However, single-image conditioning can be more creative, whereas multi-image conditioning offers greater consistency. As a result, we implement masked latent conditioning in our diffusion model. By masking out the noise in the conditioning images, GenA'DD can support any number of input views without modifying the network. With high-quality 4D data and a 4D spatio-temporal generative model, GenA'D achieves significant performance in both 3D and 4D generation using single or multiple input views. Our contributions are summarized as follows:
We design data curation pipeline for obtaining high qualltuality 4D data with movable ob- jects from videos and annotate 30,000 videos with camera poses. This large-scale dataset, termed Cam Vid-30K, will be made available for public use.
We propose 3D-4D joint framework, GenADD, which supports image-conditions 3D and 4D generation in various settings (Tab. 1). In GenYDD, the temporal layer is introduced to disentangle and fuse multi-view and temporal information.
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison among the settings of previous works.
3D Generation
40 Generation
Method
Object
Scene
Single View
Multi-View
Object
Scene
Single View
Multi-Viee
IM-3D
X
X
ReammDreamerr
ReconFusion
/
/
X
X
CAT3D
/
/
Animatel 24
X
CameraCul
x
K
X
X
J
SV4D
X
CamCo
X
V
MotionCtrl
X
X
Gen.tDD (Our))
V V
Using the proposed Cam Vid-30K along with other existing 3D and 4D datasets, GenA'D achieves performance comparable to or better than previous state-of-the-art and baseline methods in single-view 3D object generation, few-view 3D scene reconstruction, single- view 4D generation, and singlc/multivview 4D generation.
2 RELATED WORK
3D Generation. Before the emergence of large-scale 3D data, early works (Melas-Kyriae al., 2023; Poole et al., 2022) distill the knowledge from 2D diffusion models for text- and image-based 3D generation. Later, with the development of 3D data, 3D generation has been mainly explored in two directions: multi-view priors and feew-fo models. Multi-view priors (Shi et al., 2023; Liu al., 2023b;e; Long et al., 2024; Gao al., 2024) generate multi-view images and other features (e.g., normal maps and depths) based on camera embeddings, and then train a 3D representation using the generated samples or distill from generative priors. Feed-forward models (Hong et al., 2023; Liu et al., 2024b; 2023a; Tang et al., 2024; Hu et al., 2024) directly predict NeRF (Hong al., 2023), 3D Gaussians (Tang et al., 2024; Tochilkin al., 2024), meshes (Liu et al., 2023a; 2024b) from single or multi-view images. Compared to multi-view priors, more efficient but produce lower quality results. In this paper, we follow the paradigm of multi-view priors.
models are
4D Generation. Similar to 3D generation, early 4D generation works (Zhao et al., 2023; Singer et al., 2023; Ling et al., 2024; Ren et al., 2023) distill 2D video generation models into 4D rep- resentation. Due to the variability and complexity require long optimization time. Later, by leveraging animated 3D mesh data (Deitke et al., 2024), researchers render multi-view videos and use them to train 4D diffusion models (Liang et al., 2024) and feed-forward models (Ren al., 2024). Although these models achieve good multi-view and video quality, they only focus on object-centric synthetic scenarios rather than entire scenes. This limitation due to the lack of scene-level 4D data and the requirement for both multi-view static and dynamic information in these models. In this paper, we introduce a large-scale dataset of scene-level 4D data and address the challenge general 4D generation.
multi-view videos, these methods typically
Camera-controlles Video Generation. In the real world, videos contain not only object motion but also camera movement. Therefore, controlling camera movement in video generation has also garnered attention in the community (Wang et al., 2024b; Xu et al., 2024; He et al., 2024). Mo- tionCtrl (Wang al., 2024b) and CameraCtrl (He et al., 2024) introduce a branch to encode camera information from multi-view 3D data and integrate it into frozen video generation model. How- ever, due to the limitations this integration approach, these methods cannot generate videos that align well with the camera pose. CamCo (Xu et al., 2024) annotates some 4D data similar to ours and fine-tunes the entire video generation model using this data. However, due to limitations in camera pose quality and diversity, CamCo struggles to handle large camera movements.
CAMVID-30K
The lack of large-scale 4D scene data limits the development of dynamic 3D tasks, including but not limited to 4D generation, dynamic camera pose estimation, and controllable video generation. To address this, introduce high-quality 4D dataset in this paper. First, we estimate the camera poses

---

Published as a conference paper at ICLR 2025
Object
Tracking
Scale & Shift
Reletive Depth
SIM Sparse Death
Object Motion Strength
Tracks
Projection
(A) Estimation
(b) Object Motion Estimation
Figure 2: The pipeline for Cam Vid-30K data curation, including (a) camera pose estimation and (b) object motion estimation. We first leverage mask- based SfM (masks are overlayed to images in (a) for visualization) to estimate camera pose and reconstruct 3D point clouds of static parts. Then relative depth is aligned with the sparse depth and project the tracking keypoints to consecutive frame for object motion estimation.
using a Structure- Prtncture-from-Motiin (SfM) based method, then filter out data without object movement using the proposed motion strength. The pipeline is illustrated Fig.
3.1 CAMERA POSE ESTIMATION
The camera pose estimation is based SfM, which reconstructs 3D structure from their projections series of images. SfM involves three main steps: (1) feature detection and extraction, (2) feature matching and geometric verification, and (3) 3D reconstruction and camera pose estimation. In the second step, the matched features must be on the static part of the scene. Otherwise, object movement will be interpreted as camera movement during feature matching, which can impair the accuracy of camera pose estimations.
To address this. Particle-SIM (Zhao al., 2022) separates moving objects from the static back- ground using a motion segmentation module, and then performs SiM on the static part to estimate camera poses. However, it is extremely difficult accurately detect moving pixels when the camera itself is moving, and we empirically observe that the motion segmentation module in Zhao et al. (2022) lacks sufficient generalization, leading false negatives and incorrect camera poses. To obtain accurate camera poses, it is essential to segment all moving pixels. In this case, false positive error more acceptable than false negative. To achieve this, we use an instance segmen- tation model (Cheng et al., 2022) to greedily segment all pixels that might be moving, The instance segmentation model
more generalizable than the motion segmentation module in Zhao et al. (2022), particularly on training categories. After segmenting the potentially moving pixels, we esti- mate the camera pose with Particle-SIM (Zhao et al., 2022) to obtain camera information and sparse point clouds (Fig. 2(a)).
3.2 OBJЕCT MOTION ESTIMATION
Unravel Camera and Object Motion. While instance segmentation can accurately separate objects from backgrounds, it cannot determine whether the object itself is moving, and static objects neg- atively impact motion learning. Thus, we introduce motion strength to identify true object motion and filter out videos with only static objects.
Since camera movement and object motion are both present in videos, 2D-based motion estimation methods (e.g., optical flow) cannot accurately represent true object motion. There two ways to capture true object motion: by measuring motion in 3D space or by projecting motion in videos to the same camera. Both approaches require depth maps aligned with the camera pose scale. The
4

---

Published as a conference paper at ICLR 2025
Motion=0.1 Motion=2
Figure 3: Examples for object motion estimation. The motion strength is multiplied by 100. In the first example, the girl is dancing, together with the camera moving. In the second example, the camera is zooming in (red rectangle for better illustration) but the object is static. In this case, the motion strength is much smaller.
sparse depth map can be obtained by projecting 3D point clouds Pworld onto the camera view:
Peamera = R- Pworld + t, vu, = K (Xe/Zes Yo/Ze.1)²
(1)
where Peamera (Xe, Ye, Ze) denotes the coordinates of point cloud in the camera space. R and denote the rotation and translation to transform from world space to camera space. K is the camera intrinsics. With the projection equation, the depth value dsmm on the image pixel (u, v) can be obtained by dsmm( (u, u) = Ze.
As shown in Fig. 2(a), since only features in the static parts matched during 3D reconstruction, we can only obtain sparse point clouds for the static regions. However, depth information in the dynamic parts is crucial for estimating motion. To address this, we leverage a pre-trained relative monocular depth estimation model (Yang et al., 2024) to predict the relative depth of each frame dret E [0, 1]. We then apply a scale factor a and a shift 3 to align it with the SfM sparse depth:
a median (dsm:)/mmdiann///// htt///mdddaannaaaa (dre))
3 median 0r dra).
daligned = a due + B, where median(-) denotes the median value, and daligned is the dense depth map aligned with SIM depth scale.
Object Motion Field. With the aligned depth datign. we can project dynamic objects in frame into 3D space, providing a straightforward way to measure object motion. As shown in Fig. 2(b), if the object (e.g., the man in the green shirt) is moving, there will be displacement in the projected 3D point clouds. However, since SIM operates up to a scale, measuring motion directly in 3D space can lead to magnitude issues. Therefore, we project the dynamic objects into adjacent views and estimate the object motion field.
Specifically, we first need to find matching points in the 2D video. Instead of using dense rep- resentations like optical flow, we sample keypoints for each object instance and use video object segmentation (Cheng et al., 2023) and keypoint tracking (Doersch et al., 2023) in 2D videos to es- tablish matching relationships. Each keypoint is then projected into adjacent frames. The keypoint (u,, v1)" in the i-th frame is first back- projeceeoettddd into world space to obtain the 3D keypoint kp.:
kpp = R. (Z, K (u4, U1 1)" - -t+).
(3)
where Z; datigned (u,, Uz) the depth value in the aligned dense depth map. Then the 3D keypoint is projected to j-th frame with the projection equation (Eq. 1) to obtain the 2D projected keypoint (uij, "Ty)² Similar to optical flow, we represent the displacement of each 2D keypoint on the second camera view as object motion field:
(Auzj, Av.y)" = ((u, w/5/ (13 - H)))/H)²
(4)
where H and W denotes image height and width.
With the motion field for each object, we can estimate the global movement of an object by averaging the absolute magnitude of the motion field. For each video, the motion strength Y is represented by the maximum movement value among all the objects. As shown in Fig. 3, when the camera is moving while the object remains static (second example), the motion strength is significantly smaller compared to videos with object motion. Using motion strength, obvious object movement. The motion strength value also serves as a good indicator of the scale of object movement, which is used in the temporal layer enable better motion control (Sec. 4.1).
further filter out data that lacks
5

---

Published as a conference paper at ICLR 2025
Denoising UNet
Encosda Diffuee
Spatial
Multi-View
Temporal Conv
Spatial
Multi-View Self Artn
Temporal Self Selt-Attn
Conv
Canv
Self-Attn
Object Motion Strength y
MultiView-Tempora ResBlock
MultWView-Temporal Transformer
Figure 4: The framework of GenA'D. We leverage mask latent conditioned diffusion model to generate 3D and 4D samples with both camera (colorful map) and image (binary map) conditions. In addition, multiview-temmen temporlll modules together with o-fusing proposed to effectively disentangle and fuse multiview and temporal information.
4 GENAtD
4.1 GENERATIVE MODEL
Since most scene-level 3D and 4D data are captured via videos, these data lack explicit represen- tations (e.g., meshes). Therefore, we adopt an approach that generates images aligned with spatial camera poses and temporal timesteps. Specifically, we incorporate the latent diffusion model (Rom- bach et al., 2022) into our framework, introducing additional multiview-temporil layers, including multiview-temporal ResBlocks and multiview-temporal transformers, to disentangle and fuse 3D and temporal information.
Mask Latent Conditioned Diffusion Model. Latent diffusion model (LDM) (Rombach et al., 2022) is leveraged in GenA'D to generate images of different camera viewpoint and time together. LDM first encode an image/video into a latent code z with VAE (Kingma, 2013) and diffuse the latent with gaussian noise to obtain 24- Then denoising model co(-) is leveraged to estimate the noise and reverse the diffusion process with conditions:
LLOM := E [||e co (2,, t, 1
(5)
where c is the condition used for controllable generation, which is commonly text or image.
GenA'D generates multi-view images and videos with camera pose and reference image, and thus requires both camera and image conditions. Camera conditions are independent for each image, either conditioned targeted. Therefore, it is easy append it to each latent. Here, opt for Plücker ray (Plücker, 1828) as camera condition:
o x 6
where 3 and d 2 denote the camera center and the ray direction from camera center to each image pixel, respectively. Therefore, Plücker ray is a dense embedding encoding not only the pixel information, but also the camera pose and intrinsic information, which is better than global camera representation.
The reference image condition more complex. GenA'D aims to conduct 3D and 4D generation with both single and multiple input views. The single view generation has less requirement while the multi-view generation has more consistent results. Therefore, combining single and multi-view generation will lead to better real-world application. However, previous works (Blattmann et al., 2023; Voleti et al., 2024; Liu et al., 2023b) condition images by concatenating condition latent to the target latents and by incorporating CLIP image embedding (Radford et al., 2021) via cross attention. The concatenation way requires to change the channel of the model, which is unable to process arbitrary input views. The CLIP embedding can support multiple conditions. However, both ways

---

Published as a conference paper at ICLR 2025
cannot model the positional information of multiple conditions and model the information among the input views. In view of the limitations, we leverage the mask latent conditioning (Gao et al., 2024; Jain et al., 2024) for image conditions. As shown in Fig. 4, after encoding with VAE encoder, the forward diffusion process is applied to the target frames (2nd and 3rd frame), leaving the condition latent (1st frame) as usual. Then the noise on the two frames estimated by denoising model and removed by the backward process.
The mask latent conditioning has three main benefits. First, model can support any input views without modification on the parameters. Second, for sequence generation (multi-view images or video), we do not need to constraint the position of the condition frame since the condition frame keeps its position in the sequence. In contrast, many works (Blattmann et al., 2023; Xu et al., 2024; Chen et al., 2023) requires the condition image at a fixed position in the sequence (commonly the first frame). Third, without the conditioning emhedding from additional models (Radford et al., 2021), the cross attention layers used to integrate conditioning embedding can be removed, which will greatly reduce the number of model parameters. To this end, we leverage mask latent conditioning approach for Gen.YD.
MultiView-Tempon Modules. As GenA'D aims to generate both 3D and 4D samples within single model, we need to disentangle the multi-view information from the temporal information. We model these two types of information in separate layers: the multi-view layer and the temporal layer. For 3D generation, no temporal information is considered, while both multi-view and temporal information are required for 4D generation. Therefore, illustrated in Fig. 4, we propose a- fusing strategy for 4D generation. Specifically, we introduce a learnable fusing weight, a, for 4D generation, with a set for 3D generation. Using the a-fusing strategy, Gen can preserve the multi-view information in the multi-view layer for 3D data while learning the temporal information from 4D data.
a-fusing can effectively disentangle the multi-view and temporal information. However, the motion less controllable without any cues. Video generation models (Zhou et al., 2022; Blattmann al., 2023) use FPS or motion id to control the magnitude of motion without considering the camera movement. Thanks to the motion strength Y in CamVid-30K we can effectively represent the object motion. Since the motion strength is constant, we combine it with the diffusion timestep and add to the temporal resblock layer, as illustrated in ValliViee-Teeprra ResBlock of Fig. 4. With the sulltiview-tempore modules, Gen can effectively conduct both 3D and 4D generation.
4.2 GENERATION WITH 3D REPRESENTATION
GenXD can generate images with different viewpoints and timesteps using one or several condition images. However, to render arbitrary 3D-consistent views, we need to lift the generated samples into a 3D representation. Previous works (Wu et al., 2024b; Shi et al., 2023; Zhao et al., 2023) commonly optimize 3D representations by distilling knowledge from generative models. Since GenA'D can generate high-quality and consistent results, we directly use the generated images to optimize the 3D representation. Specifically, we utilize 3D Gaussian Splatting (3D-GS) (Kerbl al., 2023) and ZiR-NeRF (Barron et al., 2023) for 3D generation, and 4D) Gaussian Splatting (4D-GS) (Wu et al., 2024a) for 4D generation. More details can be found in Appendix. H.
5 EXPERIMENT
5.1 EXPERIMENTAL SETUP
Datasets. Gen is trained with the combination of 3D and 4D datasets. For 3D datasets, we lever- age five datasets with camera pose annotation: Objaverse (Deitke et al., 2023), MVImageNet (Yu et al., 2023), Co3D (Reizenstein et al., 2021), Re10K (Zhou et al., 2018) and ACID (Liu et al., 2021). Objaverse is a synthetic dataset with meshes, and we render the 80K subset (Tang et al., 2024) from 12 views following (Liu et al., 2023b). MVImageNet and Co3D are video data record- ing objects in 239 and 50 categories, respectively. Re10K and Acid are video data that record real-world indoor and outdoor scenarios. For 4D datasets, we leverage the synthetic data Objaverse- XL-Animation (Deitke al., 2024; Liang et al., 2024) and our Cam V0K. For the Objaverse- XL-Animation, we use the subset filtered by Liang et al. (2024), and re-render the depth and images by adding noise to the oribit camera trajectory. With the ground truth depth, estimate the object
7

---

Published as a conference paper at ICLR 2025
MotionCrrl
Camera Trajectoyy
CameraCurl
Ours Single View
Single View Condition
Ours 3 Views
Multi-View Condition
Figure 5: Qualitative comparison with camera conditioned video generation methods. GenA'D can generate video vell-aligned with camera trajectory and containing realistic object motion. (Please refer to supplementary video for better illustration.)
Table 2: 4D scene generation.
Table 4D object generation.
Method
FID
FVD 4.
Method
Time
CLIP-1
MotionCtrl (Wang et al., 2024h)
118.14 14
1464.08
Zero-1 1-10--VV V (Liu et al., 2023h)
hrs
79.25
CameraCirl (He et ul., 2024)
138.64
1470.59
RealfFusionVV V (Melas-Kyy) Moolas-Kyrizi et ul., 2023)
hrs
80,26
Animate 124 (Zhao et al., 2023)
Gen.lDD (Single View) Gen.IDD (3 Views)
101.78
1208 93
55.64
490.50
GenAIDD (Single View)
min
90.32
motion strength as Sec. 3.2, and then filter out data without obvious object motion. Finally, we get 44K synthetic data from Siaaverse-LL-AAimattion and 30K real-world data from Cam Vid-3KK00K,
Implementation Details. GenA'D is partially initialized from Stable Video Diffusion (SVD) pre- trained model (Blattmann et al., 2023) for fast convergence. Specifically, both the multi-view layer (multi-view convolution and multi-view self-attention) and temporal layer (temporal convolution initialized from the temporal layer in SVD, and the cross -attenattnnionn
and temporal self-attention) layers in SVD are removed. GenAYD trained three stages. We first train the UNet only with 3D data for 500K iteration and then fine-tune it with both 3D and 4D data for 500K iterations in single view mode. Finally, GenA'DD trained with both single view and multi-view mode with all the data for 500K iteration. The model is trained on 32 A100 GPUs with batch size 128 and resolution 256x 256. AdamW (Loshchilov & Hutter, 2019) optimizer with learning rate 5 x 10 is adopted. In the first stage, data are center cropped square. In the final stage, we make the images square by either center crop or padding, leading to GenX'D working well on different image ratio.
5.2 4D GENERATION
4D Scene Generation. In this setting, videos with both object and camera movement are required for evaluation. Therefore, we introduce the Cam-DAVIS benchmark for 4D evaluation. Specifically, use annotation pipeline proposed in Sec. 3 to get the camera poses for videos in DAVIS (Per- azzi et al., 2016) dataset. Then we filter the data and get 20 videos with both accurate camera poses and obvious object movement. The camera trajectories of Cam-DAVIS the training data and thus are good evaluation for the robustness to the camera movement.
-utto--distributiinn from
We compare Gen with the ooun-sourced camera conditioned video generation methods, Mo- tionCtrl (Wang et al., 2024b) and CameraCirl (He et al., 2024), on FID (Heusel et al., 2017) and FVD (Unterthiner al., 2018) evaluation metries. We use Stable Video Diffusion (Blattmann al.,
8

---

Published as a conference paper at ICLR 2025
Noodl Vere
Newal View
3D408
Gen.DD (3D-GS)
Zip-NeR
GenAD (Zip-NoRF)
Figure 6: Qualitative comparison of few-view 3D recon- struction.
Figure 7: Multi-view videos gen-
erated by Gen.YD.
Table 4: Quantitative comparison of few-view 3D reconstruction both in-distribution (Re10K)
and out-of-distributiin (LLFF) datasets.
Re10K SSIM
LLFF SSIM
Method
PSNR1
LPIPS.
PSNR1
LPIPS.
Zip- Zip-NeRF (Barron el al., 2023)
20.58
0.729
0.382
14.26
0.327
0.613
Zip-NeRF GemX'D
25.40
0.858
0.223
19.39
0,556
0.423
3D-GS (Kerbl et al., 2023)
18.84
0.714
0.286
17.35
0.489
0.335
3D-GS + GenXY
23.13
0.808
0.202
19.43
0.554
0.312
2023) the base model for both previous methods and generate the videos with the camera tra- jectory and the first frame conditions. As shown Tab. 2, using first view as condition, Gen.'DD outperforms CameraCtrl and MotionCtrl in terms of both metrics significantly. In addition, with views as conditions (first, middle and last frames), GenAD outperforms previous works by a large margin. Such results demonstrate the strong generalization ability of GenAtD on 4D) generation. In Fig. 5, we compare the qualitative results of the three methods. In this example, MotionCtrl cannot generate obvious object motion and the video generated by CameraCtrl is neither 3D nor tempo- ral consistent, Instead, our single view conditioned model can generate smooth and consistent 4D video. With conditioning views, Gen can generate quite realistic results.
4D Object Generation. We evaluate the performance on 4D object generation following Zhao al. (2023). Since GenA'D only leverages image condition instead of image-text condition as Animate124 (Zhao et al., 2023). we compare the optimization time and CLIP image similarity in Tab. 3. Instead of optimizing dynamic NeRF with score distillation sampling (SDS) (Poole al., 2022), GenAYD directly generates 4D videos of the orbit camera trajectory and uses such videos to optimize the 4D-GS (Wu et al., 2024a). With the effective generated videos and 4D representation, our method can be much faster than Animate 124 (4 minutes v.s. 7 hours). In addition, the semantic drift problem mentioned Zhao et al. (2023) is well addressed in GenX'D since we use the condition image for 4D generation. The results on 4D scene and object generation demonstrate the superiority of Gen in generating 3D and temporal consistent 4D videos. Furthermore, Gen able to generate synchronized multi-view videos from fixed view video. One qualitative example is shown in Fig. 7, and the implementation details are provided Appendix. D.
5.3 3D GENERATION
Few View 3D Generation. For few-view 3D reconstruction setting, we evaluate GenX'D on both in-distribution (Re10K (Zhou et al., 2018)) and out-of-distribution (LLFF (Mildenhall et al., 2019)) datasets. We select 10 scenes from Re10K and the scenes LLFF, and 3 views in each scene are used for training. The performance is evaluated with PSNR. SSIM and LPIPS metrics on the rendered test views. As generative model, GenA'D can generate additional views from sparse input views and improve the performance of any reconstruction method. In this experiment, we leverage two baseline methods: Zip-NeRF (Barron et al., 2023) and 3D-GS (Kerbl et al., 2023). The two baselines methods for many-view reconstruction, and thus we adjust the hyperparameter for better few-view reconstruction (more details in Appendix. H). As shown in Tab. 4, both Zip-NeRF and 3D-GS can be improved with the generated images from Gen and the improvement is more significant with the Zip-NeRF baseline. Specifically, the PSNR on Re 10K (in-distribution) and LLFF cout-of-distribution increased by 4.82 and 5.13. The qualitative comparison is illustrated

---

Published as a conference paper at ICLR 2025
Motion=0
Mation
Camera Trajectory
Motion=2.0
Single View Condition
Figure 8: Qualitative evaluation on the influence of motion strength. (Please refer to supplemen-
tary video for better illustration.)
Table 5: Ablation studies on motion disentangle.
Re10K SSIM
LLFF SSIM
Cam-DAVIS
Method
PSNR
LPIPS!
PSNRT
LPIPS}
FIDI
FVDJ
w.o. Motion Disentangle
20.75
0.635
0.362
16.89
0.397
0.560
122.73
1488.47 47
Gen.tDD
22.96
0.774
0.341
17.94
0.463
0,546
101.78
1208.93
in Fig. 6. With the generated views, the floaters and blurs reduced in the reconstructed scene. We also evaluate the performance on single view generation setting in Appendix. B.
5.4 ABLATION STUDY
In this section, we conduct the ablation study of multiview-tempar modules. The ablation study evaluated on the quality of the generated diffusion samples on few view 3D and single view 4D generation settings (Tab. 5). More ablation studics are conducted in Appendix.
Motion Disentangle (a-fusing). The camera movement and object motion are entangled in 4D data. To enable high quality generation in both 3D and 4D, GenAtD introduces multiview-temport modules (Sec. 4.1) to learn the multi-view and temporal information separately, and then fuse them together with a-fuusig. For 3D generation, the a is set to 0 to bypass the temporal module while the learned during training for 4D generation. Removing the a-fussng will result in all 3D and 4D data passing through temporal modules, which will result in the model being unable to disentangle object motion from camera movement. The failure of disentanglement will adversely affect both 3D and 4D generation (Tab. 5).
Effectiveness of Motion Strength 7 The motion strength can be used effectively control the magnitude of the object's movement. As shown in the second to last row of Fig. 8, increasing the motion strength can increase the speed of the car, As result of these observations, we can conclude that it is important to learn object motion and that the object motion field and motion strength in our data curation pipeline can accurately represent true object motion.
CONCLUSION
In this paper, we investigate the general 3D and 4D generation with diffusion models. To enhance the learning of 4D generation, we first propose data curation pipeline to annotate camera and ob- ject movement in the videos. Equipped with the pipeline, the largest real-world 4D scene dataset, Cam Vid-30K, is introduced in this paper. Furthermore, leveraging the large-scale datasets, we pro- pose GenA'D to handle general 3D and 4D generation. Gen utilize multiview-tempae to disentangle camera and object movement and is able support any number of input condition views by mask latent conditioning. Gen can handle versatile applications and can achieve com- parable or better performance in all settings with model.
modules
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was partially carried out during Yuyang Zhao's internship at Microsoft. This work is also supported by the National Research Foundation (NRF) Singapore, under its NRF-Investigatorshipp Programme (Award ID. NRF-NRFI09-0008),
Additionally, Yuyang would also like to express his deepest gratitude to Dr. Peng Qi for her unwa- vering love, patience, and inspiration throughout this journey and would like to ask her: Peng, will you marry me?
ETHICS STATEMENT
In this paper, we introduce a 4D dataset, CamVid-30K, and a generative model for general 3D and 4D generation. Vid-3OK is curated from existing public video datasets (Nan et al., 2024; Bain et al., 2021; Miao et al., 2022), and we additionally estimate the camera poses and object motion. Cam Vid-30K adheres the licenses and agreements of the original video datasets, and does not raise any new ethical concerns. While we ensure compliance with the licenses of the curated datasets and advocate for responsible use, the ability generate realistic images and videos from various viewpoints raises risks related to misinformation and privacy violations. We encourage the development of tools for detecting misuse, along with responsible dissemination of the dataset and model, to balance innovation with ethical considerations.
REPRODUCIBILITY STATEMENT
The experiments in our paper mainly include the training of GenXDD and generation with 3D rep- resentation in different settings. In Sec. 5.1, we describe the 3D and 4D datasets used to train the diffusion model, together with the training configurations. In Appendix. H, introduce the back-
bone models and the implementation details for generation with 3D representation in each setting. Our curated 4I) dataset, Cam Vid-30K, and GenAtDD model will be made publicly available.
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 11
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In ICCV, 2023. 7,
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dall-e.. pdf. 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023. 2, 6, 7, 8
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-qualit video generation. arXiv preprint arXiv:2310.19511 2023. 7
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 4
Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking anything with decoupled video segmentation. In ICCV, 2023. 5
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. 2023 ieee. In CVPR, 2023.

---

Published as a conference paper at ICLR 2025
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl A universe of 10m+ 3d objects. NeurIPS, 2024. 7,
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In ICCV, 2023. 5
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022. 17
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In JCML, 2024.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405. 10314, 2024. 3,7
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 24X1: 2024. 3,8
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. 8
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.00 04.000000 2023. 3
Tao Hu, Wenhang Ge. Yuyang Zhao, and Gim Hee Lee. X-ray: A sequential 3d representation for generation. arXiv preprint arXiv:24044 14329, 2024. 3
Siddhant Jain, Daniel Watson, Eric Tabellion, Ben Poole, Janne Kontkanen, et al. Video interpola- tion with diffusion models. In CVPR, 2024. 7
Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 ('ddeg dynamic object generation from monocular video. In ICLR, 2024. 18
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat-
ting for real-time radiance field rendering. ACM TOG, 2023. 7, 7, 20
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013. 6
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang. Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: large language model for zero-shot video generation. arXiv preprint 2rXh::2312.14125, 2023. 2
Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew Davison. Esch- ernet: A generative model for scalable view synthesis. In CVPR, 2024. 18
Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Platanio- tis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. arXiv preprint 24X.:22405.16645. 2024. 3, 7.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. In CVPR,
2024. 3
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation natural scenes from a single image. In ICCV, 2021.
12

---

Published as a conference paper at ICLR 2025
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, al. One-2-3-45 Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXfv:2306.16928 2023a.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng. Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In CVPR, 2024a. 17
Minghua Liu, Chong Zeng, Xinyue Wei, Ruoxi Shi, Linghao Chen, Chao Xu, Mengqi Zhang, Zhaoning Wang. Xiaoshuai Zhang, Isabella Liu, et al. Meshformer: High-quality mesh gener- ation with 3d-guided reconstruction model. arXiv preprint arXiv:2408. 10198, 2024b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-33 Zero-shot one image 3d object. arXiv preprint arXiv:3303./1322, 2023b. 2, 3, 6, 7, 8, 18, 19
Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453, 2023c. 3, 18
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In CVPR, 2024. 3
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 8
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang. Limin Wang, and Ying Shan. Open-magvit2. An open-source project toward democratizing auto-regressive visual generation. arXiv preprint 1:Xiv:2409.04410. 2024. 2
Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360( degg reconstruction of any object from single image. In CVPR, 2023. 3, 8
Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation. arXiv preprint
2024. 17
Jiaxu Miao, Xiaohan Wang. Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. In CVPR, 2022. 11
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (ToG), 2019. 9
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint ar 2000::20007003771, 2024. 11
F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine- A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.
Julius Plücker. -enoyericcceemmmttiscchhhhee Entwicklungen, volume 2. GD Baedeker, 1828. 6
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2022. 3, 18
Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li. Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic123: One image to high-quality object generation using both 2d and 3d diffusion priors. arXiv preprint 118684 2023, 18
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML., pp. 5748-8766 PMLR, 2021. 6,
13

---

Published as a conference paper at ICLR 2025
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cate- gory reconstruction. In ICCV, 2021. 7,
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dream- gaussian4d: Generative 4d gaussian splatting. arXiv preprint wrXiv:231..71142 2023. 3, 18,
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, et al. LAgm: Large 4d gaussian reconstruction model. arXiv preprint arXiv:2406. 2024. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022. 2, 6
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, al. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. In CVPR, 2024. 19
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long. Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023. 3,
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, al. Text-to-4d dynamic scene generation. In ICML, 2023. 3
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 2.5v::2406.06555, 2024. 2
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023. 2, 17, 19
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large mieti-view gaussian model for high-resolution content creation. arXiv preprint arXiv:2402.05004, 2024. 3.7
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02000
2024.
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan Lin- Cao. Triposr: Fast 3d object reconstruction from single image. arXiv preprint arXiv:2403.0 00100.211,,1. 2024.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. 2018. 8
arXiv preprint arXiv:
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Chris- tian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint urXiv:2403.12000 2024. 2, 6
Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXhv:2312.02201, 2023. 17, 18
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. In NeurlPS, 2024a. 19
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. In SIGGRAPH, 2024b. 3,
14

---

Published as a conference paper at ICLR 2025
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024a. 7, 18, 20
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In CVPR. 2024b. 2.7
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dy- namic 3d content generation with multi-frame and multi-view consistency. arXiv preprint arXiv:2407. 17470, 2024. 2, 18
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Camco: Camera-controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.02509 2024. 3,7
Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406. 09414, 2024.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. Mvimgnet: A large-scale dataset of multi-view images. In CVPR, 2023. 7, 21
Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. In ECCV, 2024. 18
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In ECCV, 2022.
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate 124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311. 1,6033,,,, 2023, 2, 3, 7, 8, 9
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint orXiv:2211.11018 2022.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely, Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 18055 2018. 7, 9, 21
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A MORE QUALITATIVE RESULTS
Condition
Trajectory
Time
Figure 9: The visualization of the generated 4D videos. (Please refer to supplementary video for better illustration.)
Generaato
Generation
Cienestioo
Rendering
Condition
Figure 10: The visualization of the 3D generation results. The condition images are generated by FLUX.I. The first three
the generated samples from Gen following camera trajectory, and the last row the renderings from 3D-GS model trained with the generated samples. (Please refer supplementary video for better illustration.)
16

---

Published as a conference paper at ICLR 2025
Condition
IregeDream
Magic123 123
SyaeDreamer
Zero (c-o123-XL
Ours
Figure 11: Qualitative comparison single view 3D generation.
Table Quantitative comparison of image-to-3D generation on examples from Wang & Shi (2023).
Method
Model Type
Time (min)
CLIP-I
ImageDream (Wang& & Shi, 2023)
3D
120
83.77
One2345++ (Liu et al., 2024a)
3D
0.75
83.78
IM-3D (Melas-Kyriazi et al., 2024)
3D
3
91.40
Gea/tD (ours)
3D&4D
84.75
In this section, we show more qualitative results in general 3D and 4D generation with Gen In Fig. 9, GenA'D can generate 3D and temporal consistent videos following the camera trajectory in with one or multiple condition images. addition, GenA'D can also achieve high quality 3D) generation with in-the-wild images. In Fig. 10, use GenA'D to generate multi-view images following orbit and forward-facing trajectories. Fur- thermore, the generated samples are 3D consistent and lift them 3D representations (e.g.. 3D-GS). The last row of Fig. 10 is the renderings from 3D-GS model, which is optimized from the generated samples in the third
FLUX.1' to generate condition images and
B SINGLE VIEW 3D GENERATION
We evaluate GenA'DD on the object-centric single view 3D generation on Imagelream bench- mark (Wang & Shi, 2023) and GSO dataset (Downs et al., 2022). First, Wang & Shi (2023) and calculate the evaluation metric with views in the orbit trajectory. The similarity between CLIP image embedding of the reference image and rendered views adopted as evaluation metric. As shown Tab. 6, although our performance is slightly worse than IM- 3D MMlas--yriazz et al., 2024), GenA'D achieves better performance than other baselines (Liu et al., 2024a; Wang & Shi, 2023) with fast generation speed. In addition, GenA'D can handle scene-level generation and 4D generation, which cannot be achieved by other methods. Further- more, we only use basic 3D-GS model and train with the generated views. Approaches proposed by previous works (Tang al., 2023; Melas- Melas-Kyriazi al., 2024) improve the 3D quality can still be used. In Fig. 11, we compare the rendering results from GenAtD with several previous meth-
the examples in
1-schnell
17

---

Published as a conference paper at ICLR 2025
ods (Qian et al., 2023; Liu et al., 2023b; Wang & Shi, 2023; Liu et al., 2023c). The results of other methods are obtained from Wang & Shi (2023). Compared with previous methods, GenA'D the generated 3D consistent samples to optimize the 3D-GS model, instead of using distillation techniques (Poole et al., 2022). Therefore, GenA'D) can well handle the over-saturated and Janus problem faced by other methods.
Second, we further evaluate the 3D generation ability of GenA'D on Google Scanned Objects (GSO) dataset. Here, we leverage the benchmark in Kong al. (2024), which evaluates the novel view synthesis (NVS) and 3D reconstruction accuracy on 30 samples in GSO. For NVS evaluation, 25 views are generated from the generative model and are used to calculate the PSNR, SSIM, and LPIPS with the ground truth view. For 3D reconstruction evaluation, 36 frames are generated and used to optimize NeuS. Chamfer distance and volume IoU of the exported mesh are used as the metrics. As shown in Table. 7, Gen.t') achieves the best performance on both NVS and 3D reconstruction evaluation. In addition, the compared method cannot work well on scene-level 3D generation, while GenADD can handle both object-level and scene-level generation 3D and 4D.
Table 7: Evaluation of 3D object generation on GSO-30.
NVS SSIM
3D Reconstruction
Method
PSNR
LPIPS
Chamfer Dist. +
Volume IoU 1
Zerol23
18.51
0.856
0.127
Zero123-XL
18.93
0.856
0.124
EscherNet
20.24
0.884
0.095
0.0314 0.0234
0.5974 0.6770
Gen (Ours)
21.08
0.888
0.094
C VIDEO-TO-4D EVALUATION
Following STAG4D (Zeng et al., 2024), we evaluate GenA'D video-to-4D benchmark proposed by Jiang et al. (2024). The performance is evaluated with the CLIP similarity, LPIPS, and FVD between the rendered video and the ground truth video. To realize video-to-4D, first generate multi-view images for each video frame in our 3D generation mode. The multi-view images of the first frame used to optimize 3D gaussian model first, and then optimize 4DGS (Wu et al., 2024a). After optimizing 4DGS, we conduct video refinement. Specifi- cally, we render video from 4DGS, add noise to the latent, and generate the video with GenA'DD in 4D generation mode. The generated video is used to further optimize 4DGS. The results are shown in Tab. 8. GenA'D outperforms Consistent4D (Jiang 2023) and STAG4D (Zeng et al., 2024) on all the evaluation metrics. In addition, GenAtD achieves better CLIP similarity and LPIPS than SV4D (Xie et al., 2024), and comparable FVD with SV4D.
the multi-view videos
al., 2024), DreamGaussian4D (Ren al.,
Table 8: Evaluation of video-to-4D object generation on Consistent4D. t denotes using the official open-source code to do the evaluation.
Method
CLIP
LPIPS
FVDJ
Consistent4D
0.87
0.16
1133.44
DreanGaussian4D
0.92
0.15
782.94
STAG4D!
0.92
0.13
963.61
SV4D
0.92
0.118
732.40
Gen
0.93
0.113
733.25
D GENERATING MULTI- VIEW VIDEOS
GenA'D has the capability to generate multi-view videos from single video, shown in Fig. 12. Specifically, we first generate multi-view images for each frame with our model 3D generation mode to get the coarse multi-view videos. After that, for each video, select N frames (N=12
18

---

Published as a conference paper at ICLR 2025
Input Video
Novel View
Novel View
Novel View
Figure 12: Qualitative results of multi-view videos.
4D Gaussian Generated
Figure 13: Qualitative results of 4D representation scene-level 4D gneration.
aligned with our training setting) once, take three of them conditions, and add noise to the remaining frames. Then the video is fed into Gen for denoising with 4D generation mode, obtaining the refined video. The refinement process is similar video editing, which widely adopted by 3D (Tang et al., 2023) and 4D (Ren et al., 2023) generation methods.
E SCENE-LEEEL 4D REPRESENTATION
GenAtDD can generate spatially and temporally consistent 4D videos, allowing them be lifted to 4D representations. Results are shown in Fig. 13. Specifically, we used the dynamic gaussian representation from Wang et (2024a), which deforms 3D gaussians using basis trajectories. This representation is supervised by photometric, depth, and mask losses.
F CAMERA ACCURACY EVALUATION
In this section, evaluate the camera accuracy of the generated video in terms of translation error, rotation етrоr and the COLMAP success ratio. Specifically, COLMAP is applied to the generated results and the videos with all the frames registered viewed success. The normalized transla- tion error and rotation error is calculated on the success samples. As shown in Table. 9, Gen can achieves better performance both in-distribution (Re10K) and out-of-distribttion (DL3DV) data.
G ADDITIONAL ABLATION STUDIES
In this section, further analyze the effectiveness of ray camera condition and the joint training of 3D and 4D generation.
Camera Condition. Gen leverages Plücker ray camera condition. which is dense pixel- wise camera representation. Some previous works (Liu et al., 2023b; Sargent et al., 2024) convert the camera poses spherical coordinate system 1-dimension embeddings, and then integrate the
19

---

Published as a conference paper at ICLR 2025
Table Evaluation of camera controlled video generation on both in-distribution (Re10K) and out- of-distribution (DL3DV) dataset.
Re10K Trajactory RotError.
DL3DV Trajactory
Method
TransError.
Success
TransError.]
RotError.
Success
MotionCtrl
0.0996
0.2884
0.54
0.1557
0.8054
0.57
CameraCtrl
0.0871
0.2730
0.62
0.1304
0.8251
0.75
GenAtDD (Ours)
0.0759
0.2341
0.73
0.1007
0.8019
0.89
Table 10: Ablation studies camera conditioning scheme and joint training.
Re1OK SSIM
LLFF SSIM
Cam-DAVI
Method
PSNR
LPIPS|
PSNRT
LPIPS
FID
FVD
Camera CA
21.73
0.692
0.355
17.15
0.434
0.573
105.69
1331.62
w.o. 3D Data
16.38
0.604
0.465
14.98
0.400
0.587
107.74
1262.12
W.o. 4D Data
20.74
0.740
0.359
17.35
0.448
0.554
107.93
1240.57
Gen
22.96
0.774
0.341
17.94
0.463
0.546
101.78
1208.93
condition into diffusion model with cross attention. As shown in the first row of Tab. 10, we convert the 3 camera extrinsies and the focal length 1-dimension embeddings and use it with cross attention layer (Camera CA). Compared to GenA'D, Camera CA performs worse on both 3D and 4D generation. In addition, due the effectiveness of mask latent conditioned diffusion model, GenA'D do not requires cross attention in the denoising U-Net, which is more efficient.
Joint training of 3D and 4D generation. GenADD combine both 3D and 4D data into modeling training to facilitate 3D and 4D generation. In Tab. G, we conduct ablation studies on the effec- tiveness of each type of data. 3D data contains more camera position variation and more accurate camera poses. Thus, removing 3D data impairs both 3D and 4D generation, especially the gener- ated 3D samples which cannot well aligned with the camera poses. Instead, 4D data contains object motion together with spatial camera information. The lack of 4D data will impair the quality of the generated 3D samples. In addition, despite the performance drop on Cam-DAVIS is not significant in terms of FVD and FID, removing 4D data leads the model hardly generate object movement.
H DETAILS OF GENERATION WITH 3D REPRESENTATION
For few-view 3D reconstruction, we fit trajectories from training views and generate samples corre- sponding to the cameras on the trajectory. Then the training views and generated samples are used together to optimize the 3D representation (3D Gaussian Splatting (3D-GS) (Kerbl et al., 2023) and Zip-NeRF (Barron et al., 2023)). Since both of them are designed for many views reconstruction, modify the hyperparameter to fit for few-shot setting. For Zip-NeRF, the width and depth of view-dependence network is set to 32 and to avoid overfitting. The model is trained with batch size 262,144 (64x64x6)) for 1k iterations. At each iteration, patch of 64x64 are rendered and supervised with photometric and LPIPS loss. For 3D-GS, the model is optimized for 10k iterations with photometric, SSIM and LPIPS loss. The initial point clouds are obtained from the input views for the baseline and all the generated views when combining it with Gen For single-view 3D generation, set of anchor views are first generated with single view condition. and then we gen- erate more views by interpolating the camera poses of these anchor views. All the generated views are directly used to optimize a 3D-GS model for 2k iterations with photometric, LPIPS and mask loss functions 4D generation requires the modeling of dynamic, (4DGS) (Wu et al., 2024a) as backbone. Directly optimizing a 4D representation with 4D videos can be difficult to converge. Therefore, we first optimize 3D-GS as 3D single-vie view generation manner for 500 iterations and then optimize the dynamic deformation with the generated 4D video for 2k iterations with the same loss functions.
use 4D Gaussian Splatting
20

---

Published as a conference paper at ICLR 2025
I FILTERING OBJAVERSE ANIMATION DATA
We perform filtering on the high-quality subset Liang et al. (2024) (about 80K) to get the 44K dynamic data. Specifically, we evenly select keypoints on the first view of the object and then track the keypoints to get the object motion field as in Sec. 3. Based on the motion strength, we set threshold (0.009) to filter data with low motion strength. The data filtering technique can not only remove data with small motion but also get the motion strength to be used in our multi-view temporal modules.
LIMITATIONS
Gen demonstrates remarkable performance in both 3D and 4D generation. However, there are two key limitations when applied to real-world scene generation. First, despite the abundance of 31) datasets is limited. For example, complex scenes (Zhou al., 2018),
data, the diversity of typically feature simplistic camera trajectories, such forward-facing views. In contrast, datasets focused on objects (Yu et al., 2023; Reizenstein et al., 2021), often have more varied camera paths, providing richer 3D information, but the object categories generally few in number and the scene structure is quite simple. As a result, GenA\D struggles with generating 360-degree views of complex scenes from single-view conditions.
Second, in 4D generation, temporal consistency and object motion are difficult to maintain during large camera movements in real-world scenarios. This limitation arises from the nature of available video data: video data typically provides limited object motion when the camera is moving quickly, whereas large object motion is often associated with static or slightly moving cameras. With such video data, our curated 4D data do not contain many samples with both large object and camera movement.
Both limitations are primarily due to the constraints of current datasets. However, with our proposed data curation pipeline and the increasing availability of public data, GenXDD has the potential for significant improvement in future applications.
21


Reviews:
[{"review_id": "tX1EIQqCUB", "reviewer": "Reviewer_rGgP", "summary": "This paper proposes GENXD, a latent diffusion model for 3D/4D generation of objects or scenes. Specifically, it adopts masked latent conditions to support various number of input views, and the alpha-fusing mechanism allows joint training on 3D and 4D data. Considering the lack of 4D scene dataset, the authors further curated a new dataset, CAMVID-30K, by estimating camera with a SfM-based method and filtering out videos without object motion. Qualitative and quantitative results show that the proposed method generates comparable or slightly more satisfactory outputs than corresponding prior arts.", "questions": "Q1: Following W1, what are the assumption and failure cases of the proposed camera estimation?\n\nQ2: Following W3, please describe how the metric is calculated in detail for fair comparison against prior methods.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1: Sensible model design\nAlthough the masked latent conditioning is not new, the architectural modification upon SVD is sensible and allows joint training on 3D and 4D data. \n\nS2: General model for 3D and 4D generation\nThe proposed model is capable of 3D and 4D generation of both object-centric and scene-level videos, which is more general than most prior methods. On a side note, the authors should also include MotionCtrl in Table 1.\n\nS3: Good writing\nThe paper is well-written and easy to follow overall.", "weaknesses": "W1: Limitation of camera pose estimation\nThe proposed camera pose estimation relies on segmentation of all moving pixels. However, in scenarios where camera moves independently of object motion, especially when camera motion is large or objects take up a large portion of the scene, it would be challenging to estimate accurate camera pose. Does the method assume that these cases do not exist in the dataset?\n\nW2: Quality of 3D object generation\nThe results of 3D object generation seem to be of comparable or worse quality than the prior state-of-the-arts both qualitatively (Figure 10) and quantitatively (Table 6). Moreover, the quantitative evaluation is incomplete since some more recent methods (Zero123XL, Magic123, SV3D, EscherNet, etc) are missing and the metric is limited to CLIP-I only, while prior works usually report metrics like LPIPS, SSIM, Chamfer Distance, 3D IoU (on 3D object datasets like Google Scanned Objects).\n\nW3: Evaluation of 4D object generation\nAgain, the quantitative evaluation for 4D object generation is limited to the CLIP-I metric and more recent methods like STAG4D and DreamGaussian4D are missing. Also, it is unclear if the metrics in Table 3 are calculated on the training (synthesized) video frames only or on densely sampled views and timestamps. Since the proposed method optimizes 4D-GS only on one camera orbit without SDS loss, I suspect that the outputs look good on these frames/views but worse than other methods in novel views.\n\nW4: Small camera motion in 4D scene generation\nAll the presented results on 4D scene generation seem to have smaller camera motion compared to results shown in prior work like MotionCtrl. Although the results in Figure 5 and supplemental video show decent temporal consistency and motion, I’m wondering if it is limited to camera trajectories without much deviation from the input view.\n\nW5: Lack of results on motion strength control\nWhile the paper emphasizes the contribution of motion strength control, there is only one example of a simple driving scene. It would be more insightful to show more diverse motion cases to understand the effeteness and limitations of it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wqPZx24mBZ", "reviewer": "Reviewer_x1Cy", "summary": "The paper trained a video generation model that can control camera trajectory and magnitude of motion and supports multiple frame conditioning.", "questions": "The author should provide rigorous analysis of the accuracy of the camera-controll capability, and how changing alpha values affects the generated motions.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The paper shares technical details on how to annotate magnitude motion and camera poses from in the wild videos. The alpha-fusion layers for motion disentangle seems an interesting design.", "weaknesses": "First, I feel the claim of being able to perform 4D generation is an over-claim to me. 4D generation requires the capability of either directly generating 4D representations such as dynamic 3D GS, or at least generating synchronized multi-view videos like in SV4D. Neither of these capabilities were presented in the main paper. In table 1, the capability of generating synchronized videos were not discussed, and to me, this is a severe misrepresentation. It would be more appropriate for the author to rebrand their method as a motion-controllable and 3D-aware video model. \n\n2nd, although the idea of using alpha-fusion seems interesting, it is currently not properly evaluated. It did not show how changing alpha values affects the magnitude of generated motions, and it did not evaluate the camera control accuracy as other related papers did. Reporting CLIP-score and FID is not enough to reflect the accuracy of the proposed capability of the method.\n\n3rd, a minor point, I am not sure promoting the capability of taking multiple image input can be regarded as a major technical contribution, given it is already supported in prior works including CAT3D, and it is conceptually trivial to be implemented in most video generation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3nh3XuH7Kx", "reviewer": "Reviewer_nZJ1", "summary": "1. This paper aims to jointly generate 3D and 4D objects and scenes with camera control. \n2. This paper proposed multiview-temporal modules that disentangle camera and object movements and thus can learn from both 3D and 4D data. The proposed approach employs masked latent conditions to support a variety of conditioning views.\n3. They construct a dataset CamVid-30K that consists of high-quality 4D data with camera poses for model training\n4. Extensive experiments show that the proposed method can achieve comparable or better results than baselines in 3D/4D object/scene generation.", "questions": "1. In the top case of Figure 10, the results from the proposed method appear off-center, possibly due to an inappropriate object-to-image occupancy ratio in the input images. Adjusting this ratio might improve the alignment of the results.\n2. If the learnable fusion weight, alpha, is set to 1, would it enable video generation based on the first frame? With alpha at 1, only the outputs from the temporal modules would contribute.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is the first to generate any 3D and 4D scenes with camera control and an arbitrary number of condition frames.\n2. The proposed multiview-temporal modules with alpha-fusing enable separate multi-view and temporal information and effectively conduct both 3D and 4D generation.\n3. The paper constructs a new dataset for 4D scene generation. The dataset and the data curation pipeline potentially benefit the following video generation with camera control and 4D generation.\n4. The paper is well-written and easy to follow.", "weaknesses": "**Experiments**:\n\n1. In the experiment of 4D object generation, some relevant references and comparisons are missing, such as Consistent4D [1] and STAG4D [2].  Since these works are open-source, it would strengthen the paper to include these baselines or clarify why they are not suitable for comparison. They take single-view video as input, which should be applicable for this work.\n2. In Table 3, it would also be beneficial to report temporal consistency metrics (e.g., FVD), as temporal consistency is critical for 4D object generation.\n\n**Minor Points:**\n\n1. Clarifying the selection process for the 44K dynamic data in Objaverse-XL would be helpful. According to Diffusion4D [Liang et al. (2024)], ~323K dynamic objects were collected. For instance, what filters were applied in this work? Will the selected dynamic objects be publicly available? Adding these details in the Appendix would enhance transparency.\n2. Some technical details are missing: What is the maximum number of frames the model supports? Additionally, in Table 3, Zero-1-to-3 and RealFusion were originally designed for 3D reconstruction—how were they adapted for 4D generation in this work?\n\n[1] Jiang, Yanqin, et al. \"Consistent4d: Consistent 360 {\\deg} dynamic object generation from monocular video.\" ICLR 2024.\n\n[2] Zeng, Yifei, et al. \"Stag4d: Spatial-temporal anchored generative 4d gaussians.\" ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FJa2b0BP82", "reviewer": "Reviewer_oiM6", "summary": "In the paper the authors propose two main contributions: a curated dataset for 4D generation model learning, named CamVid-30K; and a model trained to generate in 3D or 4D given arbitrary condition images, named GenXD.\n\nAuthors proposed a detailed pipeline on how to combine existing techniques to curate a 4D scene datasets for model training. Including instance segmentation modules for static / dynamic decomposition, static structure-from-motion to recover camera parameters and sparse depth map, relative depth align with sparse depth map for spotting the dynamic object and introducing a motion strength factor as additional condition.\n\nAuthors proposed a new model GenXD to train on this dataset combining with other object-centric 3D/4D datasets and 3D scene datasets. They further design a $\\alpha$-fusing strategy to better disentangle the spatial and temporal information in the data source. Experiments across various benchmark show impressive performance of the proposed method.", "questions": "1. The statement in section 3 that \"when the camera is moving while the object remains static, the motion strength is significantly smaller compared to videos with object motion\" seems not so easy to understand. I assume the authors mean that this is the common case for natural captured video where cameras are still or moving in a slow motion?\n\n2. Does the $\\alpha$ needs to be explicitly set during training / inference? For example let network itself output the weight when dealing with 4D content and explicitly set it as 0 when dealing with 3D content. If so then it would be interesting to see given same conditions (more complicated than Figure 7) what would model outputs for different $\\alpha$. Like given multi-frames of a static scene but telling model do 4D generation and given multi-step frames with dynamic objects and force model to do static 3D generation.\n\n3. It's kind of confusing that the 5.4 ablation study is for model training or just inference after training? If it's after training, than the results in table 5 is somehow not so useful as it's trained with $\\alpha$ but in inference time not allowed to use it, which would certainly lead to performance drop.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The data curation pipeline for transforming existing videos into trainable 4D dataset is quite useful, and the proposed curation pipeline and the CamVid-30K should be beneficial to the field.\n\n2. Combining all source and sub-tasks' data (object/scene, 3D/4D) is fundamentally useful and a model trained on mixture of data should have better generalization ability. The proposed $\\alpha$ parameter seems can be understood as an explicit control to switch between 3D and 4D generation given same conditions.\n\n3. The results are promising and generally good. And extensive evaluations on multiple benchmarks show the effectiveness of the proposed method.", "weaknesses": "There are some minor errors or confusing points in the paper. I'll list some here and some in the following questions section.\n\n1. In L253, \"The keypoint $(u_i, v_i)^T$ in the $i$-th frame is first back-projected into world space to obtain the 3D keypoint $kp_i$\". I agree here the $kp_i$ should be in world space, but according to Eq.(3) seems it's in the camera space? From my perspective the Eq.(3) is transforming image-space coordinates to camera-space coordinates, missing the step of transforming to world coordinates.\n\n2. In all the figures with camera trajectory visualization, the legends and axis notations are very small and impossible to tell the actual information, also the trajectory only lies in a small region in the plot. I suggest authors remove the axis notations if they are too small, and zoom in to show the trajectory in a more detailed way.\n\n3. In section 5.2 4D object Generation, it seems unfair to say \"results in our method being $100\\times$ faster\", as the efficiency comes from using a different underlying 3D representation comparing to other methods, which are orthogonal to the proposed method. I think here using CLIP similarities for comparison is reasonable. Showing speed is fine but shouldn't be used as comparison.\n\n4. I think in general the paper is with good results. But according to the task of the proposed method, I expect to see more scene level 3D or 4D generation results, including larger camera trajectories and failure examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)