Optimizing 4D Gaussians for Dynamic Scene Video from Single Landscape Images
Authors: In-Hwan Jin
Haesoo Choo
Seong-Hun Jeong
Park Heemoon
Junghwan Kim
Oh-joon Kwon
Kyeongbo Kong
Venue: ICLR 2025 Poster (2025)
Keywords: Dynamic Scene Video
4D Gaussian
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: To achieve realistic immersion in landscape images, fluids such as water and clouds need to move within the image while revealing new scenes from various camera perspectives. Recently, a field called dynamic scene video has emerged, which combines single image animation with 3D photography. These methods use pseudo 3D space, implicitly represented with Layered Depth Images (LDIs). LDIs separate a single image into depth-based layers, which enables elements like water and clouds to move within the image while revealing new scenes from different camera perspectives. However, as landscapes typically consist of continuous elements, including fluids, the representation of a 3D space separates a landscape image into discrete layers, and it can lead to diminished depth perception and potential distortions depending on camera movement. Furthermore, due to its implicit modeling of 3D space, the output may be limited to videos in the 2D domain, potentially reducing their versatility. In this paper, we propose representing a complete 3D space for dynamic scene video by modeling explicit representations, specifically 4D Gaussians, from a single image. The framework is focused on optimizing 3D Gaussians by generating multi-view images from a single image and creating 3D motion to optimize 4D Gaussians. The most important part of proposed framework is consistent 3D motion estimation, which estimates common motion among multi-view images to bring the motion in 3D space closer to actual motions. As far as we know, this is the first attempt that considers animation while representing a complete 3D space from a single landscape image. Our model demonstrates the ability to provide realistic immersion in various landscape images through diverse experiments and metrics. Extensive experimental results are https://cvsp-lab.github.io/ICLR2025_3D-MOM/.

Published as a conference paper at ICLR 2025
OPTIMIZING 4D GAUSSIANS FOR DYNAMIC SCENE VIDEO FROM SINGLE LANDSCAPE IMAGES
In-Hwan Jin' Haesoo Choo
Seong-Hun Jeong'
Heemoon Park
Junghwan Kim
Oh-joon Kwon
Kyeongho Kong't It
Pusan National University
PPukyong National University
BBsan Munhwa Broadcasting Corporation
*Korea University 'DM Studio
ABSTRACT
To achieve realistic immersion in landscape images, fluids such as water and clouds need to move within the image while revealing new scenes from various camera perspectives. Recently, a field called dynamic scene video has emerged, which combines single image animation with 3D photography. These methods use pseudo 3D space, implicitly represented with Layered Depth Images (LDIs). LDIs separate single image into depth-based layers, which enables elements like water and clouds to move within the image while revealing new scenes from different camera perspectives. However, landscapes typically consist of contin- uous elements, including fluids, the representation of 3D space separates a land- scape image into discrete layers, and it can lead diminished depth perception and potential distortions depending on camera movement. Furthermore, due to its implicit modeling of 3D space, the output may be limited videos in the 2D do- main, potentially reducing their versatility. In this paper, we propose representing complete 3D space for dynamic scene video by modeling explicit representa- tions, specifically 4D Gaussians, from a single image. The framework is focused on optimizing 3D Gaussians by generating multi-view images from a single im- age and creating 3D motion to optimize 4D) Gaussians. The most important part of proposed framework is consistent 3D motion estimation, which estimates com- mon motion among multi-view images to bring the motion in 3D space closer actual motions. As far as we know, this is the first attempt that considers anima- tion while representing complete 3D space from single landscape image. Our model demonstrates the ability to provide realistic immersion in various landscape images through diverse experiments and metrics. Extensive experimental results are
lab. late-llab..ghbbbb. github. 1 P.D/CCLRO ICLR2025_3D- 3DMMMM/
INTRODUCTION
When observing real-world landscape, do you perceive the movement of the water and clouds? Can you discern how the speed of motion differs between distant mountains and nearby leaves as your perspective changes? If you notice these phenomena, it's because the actual landscape exists in a 4D space. Not only has significant research been conducted on creating realistic 3D videos by adding parallax to video based on depth information (Duong et al. 2019). but also to provide realistic immersion from static 2D images. Additionally. the synthesis of texture and structures in occluded regions can be enabled by 3D photography (Mildenhall al., 2021 Wiles et al. 2020) Tucker & Snavely 2020 Niklaus et 2019; Shih et 20200 Kopf et al. 2019), thereby allowing parallax effects from a single image. Among them, dynamic visual effects can be infused into static
images by single-image animation (Chuang et al. 2005 Thou & Cheng 2015 Endo et al. 2019
Logacheva et al. 2020 Holynski et al. Fan et al. 2023) which enable beaebles/i fluid motion.
Recently, field known as dynamic scene video (Li ct al. 2023 Shen et al. 2023) has emerged, which creates videos with natural animations from specific camera perspectives using a combina-
tion of single image animation and 3D photography. These methods utilize Layered Depth Images (LDIs) (Shih et al.) 2020 Kopf et al.] 2019 2020 Wang et al.] [2022] Tuusiaai et al.) 2018). which are created by dividing single image into multiple layers based on depth, represent pseudo
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2025
3D space. However, there are limitations when attempting to discretely separate most elements, including fluids, in a continuous landscape, and 3D space cannot be fully represented this way. Con- sequently, distortions can be observed, the depth perception of the space be diminished with camera movement. Therefore, the achievement of complete 4D space virtualization through explicit representation, rather than relying LDIs, is necessary.
The recently emerged 3D Gaussian splatting (Kerbl et al. 2023) offers high-quality and efficient real-time rendering by representing scenes in 3D space through multiple 3D Gaussians using explicit representation. This research has also been expanded to 4D Gaussians (Luiten et al. 2023 Yang
al.
2024; Shaw et al. 2023 Huang et al. 2024 Liang et al., 2023] Ling et al., [023] Yin al.
2023
Ren et al. (202)). with time axis added to 3D Gaussians to represent the structure and
appearance of 3D objects over time. Among these research, some researchers (Wu et al.
2024 Yang
et al.
2024 Huang et al., 2024) focused on modeling the changes in position, rotation, and scaling of each 3D Gaussian over time to achieve more natural motion in 4D Gaussians.
In this paper, propose method to represent complete 3D space for dynamic scene video by modeling 4D Gaussians from a single image. As shown Fig. [ the proposed framework consists of three step: (1) 3D Gaussians Optimization, (2) Consistent 3D Motion Estimation, and (3) 4D Gaussian optimization. First, to optimize the 3D Gaussians, we generate multi-view RGB images from single imput image. At this stage, we 2D image into a 3D space. Subsequently, we aim to optimize 4D Gaussians by moving the regions corresponding to the motion areas of the optimized 3D Gaussians. To achieve this, create multi- view motion mask and then optimize 3D motion within 3D space. Finally, through the proposed 3D motion, we calculate the changes in position, rotation, and scaling of the Gaussians over time. As result, we can optimize 4D Gaussians that maintains consistency across multiple views.
3D point cloud created by lifting the pixels of
The most important part of our framework is optimization of 3D motion. Our goal is to move the 3D Gaussians. which requires motion within the 3D space. However, motion estimation directly within 3D space, such as with 3D point clouds or 3D Gaussians, is not only unexplored in existing research but also poses highly challenging task due to the lack of dedicated datasets for this purpose. An alternative approach is to utilize existing off tff-tff-sheff 2D motion estimation models (Holynski et al.) 2021]. Fortunately, motion estimation for 2D images has been extensively studied which allows the acquisition of motion from multi-view images to be easy. Therefore, we can easily achieve 3D motion by unprojecting the estimated 2D motion into the 3D domain using depth map.
However, when the estimated motion is applied to the 3D space, we found that the motion consis- tency across the multi-view images is not maintained. This inconsistency arises because the motion estimated independently for each view. Consequently, artifacts in the 4D Gaussians can be caused by the movement of the Gaussians with the current 3D motion, potentially resulting in distortion in the rendered dynamic scene video. To address this issue, we propose a 3D Motion Optimization Module (3D-MOM). This model aims to optimize 3D motion to generate consistent motion across multi-view images. Specifically, arbitrary 3D motion is modeled and then optimized through the loss between the unprojected 3D motion and the estimated 2D motion.
Our module also benefits from utilizing 4D Gaussians to represent a fully 3D space with animation. Gaussian splatting offers differentiable volumetric model that enhances depth perception, visual fidelity, and facilitates faster training times. Furthermore, the output in the form of Gaussian in the 3D domain can offer even greater versatility compared to the previous limitation to the 2D domain. From these advantages, our module creates realistic immersive dynamic scene video that ensure high visual quality for various landscape images.
In essence, our main contributions include:
We propose a complete 3D space virtualization method for dynamic scene video, which is the first attempt to consider animation in 3D space. To achieve this, we optimize 4D Gaussians from a single landscape image to provide more immersive dynamic scene video.
To animate 3D Gaussians, we estimate 2D motion from the multi-view images, and then unproject it back into the 3D domain to generate 3D motion. In this regard, we propose 3D-MOM as a method to find a consistent motion among multi-view images.
Our framework uses Gaussian-basse spatial representation to model a complete 3D space. This approach provides not only realistic depth perception through volumetric representa- tion but also highly practical utility with explicit outputs.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 NOVEL VIEW SYNTHESIS FROM SINGLE IMAGE
Novel view synthesis involves generating 3D scenes from existing images for new camera perspec-
tives. Recent advances, particularly Neural Radiance Fields (NeRF) (Mildenhall et al. (02T). have
enabled lifelike 3D scene generation from multi-view images. Research (Wiles et al. 2020 Weick-
ert, 1999 Tucker & Snavely 2020; Niklaus et al. 2019; Shih et al. 2020; Kopf et al. 2019 Wang
jet al. (022)) has also focused on synthesizing 3D scenes from single images by mapping them to point clouds and using inpainting for rendering new views. However, these methods face challenges in maintaining content consistency between frames, particularly for complex scenes. Layered depth
images (I.IIs) (Shade et al. 1998. Tulsiani al. 2018 Shen al. 2023 Li al. 2023) have been proposed to efficiently represent 3D scenes by dividing images into depth-based layers, but these approaches often result in depth perception issues or artifacts. VividDream (Lee et 2024) extends this idea to 4D scene generation using stable video diffusion, though it lacks explicit 3D motion modeling. Therefore, continuous 3D virtualization across various angles is crucial. Our pro- posed method addresses these challenges, offering seamless novel view synthesis with full 3D space virtualization.
2.2
SINGLE IMAGE ANIMATION
Single image animation generates dynamic visual effects from static images, with focus on animat- ing landscapes containing fluid elements like clouds and water. Early research (Chuang et al.
2005
separated layers manually create looping video textures, while later studies (Jhou & Cheng
2015)
used physical modeling to animate vapor-like objects. Recent udvancements in deep learning have
led to motion prediction within single images (Endo et al., 2019 Logacheva al.) 2020 Holyn-
ski et al. 2021). transforming them into animated video textures. Among these, Holynski et al.
(Holynski et al. 202T) produced realistic looping animations using Eulerian flow fields, effectively
approximating fluid motion. This approach continues to be expanded in subsequent research (Maha-
patra & Kulkarni 2022; Fan et al.) 2023 Mahapatra et al. 2023 Li et al. 20233 Shen et al. 2023).
Recent methods, such Text2Cinemagraph (Mahapatra et al. (2023), also Eulerian flow to sim-
ulate impressive fluid motion in single images. Additionally, 3D Cinemagraphy (Li et al. 2023)
and Make-Ir-4D (Shen al., (202)) employ joint learning of animation and novel view synthesis to generate natural fluid animations in 3D space. We apply Fulerian flow to fluids to achieve natural animations when generating 3D motion optimize 4D Gaussians.
2.3 4D GAUSSIAN SPLATTING
explicit represen- tation to address the performance issues the implicit NeRF approach (Mildenhall et al. 202I). Research has expanded to include time dimension on 3D Gaussians that capture dynamic changes over time. Luiten et al. (Luiten et al.) 2023) implement frame-by-free training to define position and rotation per timestep. which shows promising tracking results despite some consistency issues. Yang et al. (Yang et al. (024)) added deformation field using canonical 3D Gaussians and MLPs, but faced longer training times. Wu al. (Wu al.. 2024) attempted reduce this by replacing multi-resolution hexplane and a lightweight MLP, though it struggled with accurate deformation predictions. Huang et al. (Huang et al. (024) focused on sparse control points and represented the 4D scene through interpolation. Alternatively, Li et al. (Li et al. 2024) enhanced motion expressive- ness by integrating temporal features into 3D Gaussians. Despite these advances, training generally relies on multi-view images. Our work, however, optimizes 4D Gaussians from single image to generate and refine 3D motion.
The Gaussian Splatting introduced by Kerbl et al. (Kerbl al., 2023)
3 PROPOSED METHOD
In this section, we provide an overview of our framework, as shown in Fig. Our proposed pipeline consists of three main stages: 1) Generation of multi-view RGB images from a single image to optimize 3D Gaussians /Scc.B.I.. 2) Estimation of 3D motion in order to move the 3D Gaussians. Since the 2D motion maps estimated from the multi-view RGB images lack consistency in the
3

---

Published as a conference paper at ICLR 2025
(a) 3D Gaussians Optimization
(b) Consistent 3D Motion Estimation
(c) 4D Caussians Optimization
Motion
+
Optimization Module
Depth Escimation Model
Depth Map
Motion Mask
Gaussians
Motion
MLP
RGB Point Cloud
Medion Mask Point Cloud
Position
Rotation
Scaling
Deformed 3D Ganasiana
Rendered Multiview RGB Image
Rendered Multi-iew
Morion Mask
Frune
Model
Frame
Motion
Sec. 3.1
Map
Sec.
Sec.
Figure I: The overview of our pipeline. Our goal is to optimize 4D Gaussians to represent a com- plete 3D space, including animation, from single image. (a) A depth map is estimated from the given single image, and it is converted into a point cloud. For optimizing the 3D Gaussians, multi- view RGB images rendered according to the defined camera trajectory. (b) Similarly, multi-view motion masks are rendered for the input motion mask. These are utilized to estimate multi-view 2D motion maps along with the rendered RGB images. 3D motion is obtained by unprojecting the estimated 2D motion into the 3D domain. In this context, the proposed 3D Motion Optimization Module (3D-MOM) ensures consistent 3D motion across multi-views. Gaussians and generated 3D motion, 4D Gaussians are optimized for changes in position, rotation, and scaling over time.
(c) Using the optimized 3D
3D space, we propose 3D Motion Optimization Module (3D-MOM) to estimate consistent 3D motion (Sec. (.2). 3) Warping the 3D Gaussians using the estimated 3D motion and optimizing the 4D Gaussians (Sec. 3.3)
3.1 MUL IMAGE GENERATION FOR 3D GAUSSIANS OPTIMIZATION
Recently introduced 3D Gaussian splatting (Kerbl al. 2023 uses explicit representation to de- pict scenes in 3D space with numerous 3D) Gaussians. This method, as a differentiable volumetric representation, provides depth perception from multi-views. We use 3D Gaussians represent complete 3D space from a single landscape image and optimize 4D Gaussians (Wu et al., 2024
Luiten et al. 2203 Yang et al. 2024, Shaw et al. 2023 Huang et al. 2024 Liang al., 2023
Ling et a [2023] Yin et 2023 Ren et al. 2023) by incorporating motion into the represented 3D
space. To achieve this, we need generate multi-vies view RGB images from single image to optimize
the 3D Gaussians.
3.1.1 POINT CLOUD GENERATION
To generate multi-view RGB images, it is necessary to convert 2D image into 3D space and project according to various camera parameters. To achieve this, we first generate a point cloud from a single image. Specifically, we a monocular depth estimation model to estimate the depth map D R! x HxW for the input image I p3x HxW Subsequently, we unproject the input image and the depth map into 3D space create the point cloud P/:
Pı = (X,, C.)) = 42-3 (,))
(1)
where X, and C, 3D coordinates and the RGB values for each point, respectively. (2-2(:) is the function to lift pixels from the RGB image to the point cloud, and K and Eg are the camera intrinsic matrix and the extrinsic matrix of input image I.
4

---

Published as a conference paper at ICLR 2025
3.1.2 MULIT-VIEN IMAGE RENDERING AND 3D GAUSSIANS OPTIMIZATION
Inspired by prior works (Chung et al. 2023), we project the initial point cloud onto 2D plane image I, according to specific camera extrinsic parameters E, render multi-view RGB images as follows:
i, = 0 K, E,).
(2)
Starting with the center camera view point Eg. we continue the rendering process until all the cam- eras have been traversed. To handle holes introduced during rendering, we apply simple linear in- terpolation, ensuring efficient and high-quality multi-view image generation. These generated multi-
view images act as the ground truth (GT) for the multiview loss.
Using these rendered images, we initialize and optimize 3D Gaussians to represent the scene vol- umetrically. Following (Kerbl et al. 2023). we optimize the Gaussian parameters by balancing LI and SSIM losses:
L L1 X(IsSIM - L1),
where LSSIM enhances structural similarity while L1 minimizes pixel-level discrepancies. This en- sures accurate volumetric representation and prepares the Gaussians for subsequent motion and temporal modeling.
In Sec. 3.1, we have streamlined the discussion of 3D Gaussian optimization to focus its role in bridging the gap between 3D scene generation (Chung et al. [0233] Ouyang et al. (202)) and our dynamic scene video. To enable the novel task of 4D scene generation, we designed Section 3.1 to closely align with prior 3D scene generation works. Further details on 4D scene generation will be Application: 4D Scene Generation.
provided in Sec. 4.5
3.2 CONSISTENT 3D MOTIONS ESTIMATION
To animate still 3D Gaussians, we need to estimate the corresponding motion field. However, direct 3D motion generation from a point cloud or 3D Gaussians is very challenging. Fortunately, various research has been conducted in the field of single-image animation (Chuang et al.] 2005 Jhou &
Cheng, 2015 Endo et al. 2019: Logacheva et al. 2020 Holynski et al. 2021; Mahapatra & Kulka- mi [002] Fan et al. 2023; Mahapatra et al. 2023) which enable the estimation of 2D motion in provided. Therefore,
areas where masks
propose novel approach that leverages existing off-the-shelf animation models to estimate 2D motion from multi-view images. Subsequently, we unproject this 2D motion into the 3D domain estimate 3D motion.
3.2.1 VULW 2D MOTTON ESTIMATION
Before estimating motion for multi-view images, it is necessary to generate a motion mask indicating the areas in each image where animation will be applied. Initially, we take an initial motion mask Mo IR 1x as input. Subsequently, similar the existing equations (1) and (2). we generate
point cloud for the initial motion mask and project it to a specific camera view point E, to create the motion mask M; as follows:
M. = 23-22 (Mo. D;K, Ko): E.).
(4)
To estimate motion maps from multi-view images at a subsequent time point, we adopt existing methods such as those by Holynski al. (Holynski et al. 2021). which estimate the motion of fluids like water and clouds from a single image. Specifically, we use Eulerian flow EF to represent the vector field that indicates the movement of pixels in the image, follows:
F--1441-) EF(),
where Fr-c-1(-) (-) represents the motion change from time time + and the Eulerian flow indicates that the amount of motion change between frames moves at constant speed and direction over time. We estimate multi-view motion maps from generated multi-view images and motion masks using an existing 2D motion estimation model based on this Eulerian flow. This process can be formulated follows:
F =F M.),
5

---

Published as a conference paper at ICLR 2025
Iteration
Treration
Iteration N
Learsoble Potnt (f.y Flaed Paind (e..4)
ho
Projected Motion (Fon E.)
Projected Motion Os-- K.E.)
Projected Motion oy- (P.ot K.E.)
L F. Sonn Pant Boll.
min > IP, 2o (Pann
L F. 4o Pant B.lll
$ Listimated 2D Motion
Estimaaeed 2D Moiion E
$ Estimated 2D Motion
Figure 2: 3D Motion Optimization Module. To maintain consistency of motion across multi-views 3D motion is defined from the point cloud and projected into 2D images using camera parameters. The LI loss between the projected 2D motion and the estimated 2D motion map the ground truth is computed, minimizing the sum of losses for multi-view to optimize the 3D motion.
where i, represents a multi-view image. Therefore, F, denotes the motion map of the multi-view image I,
However, since the multi-view motion maps F, are estimated independently for each multi-view image, there may be phenomenon known as Motion Ambiguity. This occurs when the motion values at the same position in 3D space do not match when unprojected. Using these inconsistent motion maps directly for 4D reconstruction would result artifacts and unnatural motion dynamic regions. To address this, propose the 3D-MOM. which leverages multi-view 2D motion maps to estimate consistent 3D motion, thereby extending traditional 2D motion methods into the 3D domain.
3.2.2 3D MOTION OPTIMIZATION MODULE
We prioritize achieving 3D motion consistency by parametrically modeling 3D motion. The modeled 3D motion reprojected into 2D space, and its error is calculated relative to the estimated multi- view 2D motion maps. By optimizing this error while applying consistency constraints, our approach ensures that the resulting 3D motion is both consistent and robust, even though may be sub-optimal in general 3D motion estimation tasks.
As illustrated in Fig. we use the point cloud P, located at coordinates (Ir, y. 2) as the starting point. Then, we define the coordinate difference between (,',3/,=)) =') and (ar, y, 2) as the 3D motion, FAD- This represents the movement, or motion, of the coordinate points in 3D space. Afterwards, we project the 3D motion information through camera parameters K and E, into 2D image space, rendering it as (u, v;) in the 2D space. Subsequently. the LI loss computed between this projected motion (u, v:) and the motion map F, derived from the 2D motion estimation model. Using the multi-view 2D motion F, as the ground truth, we compute the loss 2D space and optimize the 3D coordinate (æ', ,//22 2²) accordingly. This process be formulated as follows:
N
min
F 33-2 ( K. Ex)l1
1=0
where
FAD (d. (x,
(8)
N represents the total number of multi view images generated along the camera trajectory. To mini- mize (7) we updated FaD through Stochastic Gradient Descent, ultimately obtaining consistent 3D motion space.

---

Published as a conference paper at ICLR 2025
3.3 4D GAUSSIANS OPTIMIZATION
Recently, the research in 3D Gaussian is expanded to 4D Gaussians (Luiten et al.
2023 Yang et al.
2024 Wu al., 2024 Shaw et al.
2023 Huang al. 2024 Liang al.) 2023 Ling et al. 2023
Yin et al. 2023 Ren et al. 2023 to represent the structure and appearance of 3D over time changes.
In our framework, we adopt 4D-GS (Wu et al. 2024) ensure efficient training time and quality. However, our unified framework is not constrained to a specific 4D Gaussian model and is com- patible with various models. As the field advances, it enables the rapid generation of high-fidelity dynamic scene videos. To predict the deformation of 3D Gaussians specifically. 4D-GS (Wu et al., 2024) utilize a Spatial-Temme -empirll Structure Encoder. This enables the effective modeling 3D Gaus- sian features using multi-resolution HexPlane and a tiny MLP. Subsequently, through a multi-head Gaussian deformation decoder, we calculate the deformation of position, rotation, and scaling (Ax,
År, As), representing each as follows:
(x'r',,)) (x Ax, + A As),
where x, r, are initial position, rotation, and scale, respectively.
3.3.1 3D MOTION INITIALIZATION
In this case, to represent the animation, the most dominant component is the deformation of the position. This corresponds to the change in the mean values of each Gaussian, which is initializa- tion with the previously estimated 3D motion. Therefore, by adding the motion information to the position in advance, we can modify the equation as follows:
(x',''s') = (x + F3D Ax', A As).
(10)
The remaining deformations, aimed at preventing distortion of the estimated 4D Gaussians, can also be sufficiently learned through weak supervision. Therefore, we jointly train the first frame's s multi-view images I obtained earlier and the 2D animation results of the input image to estimate the remaining deformations. For this purpose, we draw inspiration from prior research (Mahapatra al.] 2023) (Fan et al. (023) and utilize a video generation model to obtain the animation. In particular, they utilize the Fulerian flow field, which represents changes in motion over time moving at a con- stant speed and direction, create realistic animated looping videos for fluids, as demonstrated by Holynski et al. (Holynski et al. 202T.
3.3.2 TWO-STAGE TRAINING
Animating videos of the entire multi-view images time-consuming and redundant due to signif- icant overlap in areas. To address this, we devised a learning method that can accurately captures motion across the entire spatial and temporal domains while reducing time. Inspired by the training of prior work (Li et al.) 2023), Shen et al.) 2023), we applied two-stage training approach across all viewpoints and animated videos. First, we fixed the time at 0 and trained a 3D Gaussian for the entire viewpoints. In the second stage, trained 4D Gaussians across the entire temporal axis using video frames of sampled viewpoints. To the best of our knowledge, this is the first work to validate the effectiveness of two-stage training approach for 4D Gaussian learning, demonstrat- ing its capability to efficiently reconstruct spatiotemporal dynamics. Through this process, we can efficiently obtain 4D Gaussians that consider motion by applying natural animation for fluids from optimized 3D Gaussians. The experimental results for metrics and runtime can be found in Table of Appendix.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
4.1.1 BASELINE MODEL.
To evaluate the effectiveness of our approach in the context of dynamic scene video, we compared with two state-of-the-aar models. 3D.Cinemagraphy (Li et al. 2023) utilizes LDIs for 3D rep- resentation and jointly produces animations apply the parallax effect in realistic dynamic scene
7

---

Published as a conference paper at ICLR 2025
(0
(6)
Fremm: Framm: 30 Frome
Figure 3: Qualitative Results. (a) 3D Cinemagraphy (L et al. 2023), (b) Make-It-4D (Shen al.
2023 (c) DynamiCrafter (Xing et al. 2025), (d) Motion-12V Shi et al. 2024). and (e) Ours.
Table 1: Quantitative Results. We measured three metrics to quantitatively compare the results of the baseline model and the proposed method. The results showed that the proposed model outper- formed in all metrics. Furthermore, the user study evaluated the performance of each model, with the proposed model demonstrating the highest level of performance across all criteria.
Metrics
User study (%)
Method
PSNR
SSIM
PIQE
Immersion
Realism
Stuctural Consistency
Quality
DynamiCrafter Xing et 2025
14.98
0.81
0.23
24.58
Motion-12V (Sh st 2074
14.38
0.80
0.31
8.40
Cinemagraphhy
17.30
0.83
0.17
8.93
31,87
31.87
28.75
30.31
Make-It-4D Shell et air. 2023
16.98
0.81
0.20
8.30
11.87
10.31
8.43
9.06
20.57
0,90
57.81
62.81
60.25
video. Similarly, Make-It-4D (Shen et al. 2023) uses LDIs for 3D representation but employs pre-trained diffusion model for inpainting fill occluded areas, achieving wider camera poses similar to long-range views. In these studies, the output is limited to the 2D domain in the form of images. contrast, our model outputs in the 3D domain as Gaussian forms, which are projected to 2D according to camera trajectories for comparison of results.
4.1.2 IMPLEMENTATION DETAILS
We manually generate the input masks using the LabelMe tool to designate the motion the imput image. During multi-view image generation, we employ ZoeDepth (Bhat et al., 2023) for depth estimation, which provides normalized depth values. While these values lack absolute scale, they are sufficient for constructing a 3D point cloud when combined with intrinsic and extrinsic camera parameters. Intrinsic parameters determined based on the input image size using stan- dard conventions, and extrinsic parameters are initialized as an Identity Matrix to define the camera trajectory. We establish a rendering trajectory with about 30 camera viewpoints for point cloud ren- dering. To estimate flow from single image, we utilize Holynski et al. (Holynski et al., 2021) and the pretrained single-image animation model from SLR-SES (Fan al., 2023), selected after evaluating various options for their ability to produce consistent looping animations and temporal coherence. Our 3D motion optimization module is trained for about 200 iterations with a batch size of 30 using the SGD Optimizer. We set the initial learning rate 0.5 and then decayed it exponen- tially. All models were tested to ensure compatibility with the proposed framework. We conduct all experiments single NVIDIA GeForce RTX 3090 GPU.
in
4.1.3 EVALUATION DATASET
Following (Li et al. 2023), evaluated method and the baselines using the validation set from Holynski et al. (Holynski et al., (02T). The validation set consists of 162 samples of ground truth video clips, captured from static camera viewpoint across 27 different scenes. For evaluation,
8

---

Published as a conference paper at ICLR 2025
Projected
View
Motion
w/o 3D Motion
Motion
Motion
w/ 3D Motion
3D Motion
Frame 20
Frame 40
(a)
Figure 4: (a) Visual comparison of the 2D motions estimated from multi-view images and projected 3D motion estimated via 3D-MOM. (b) Rendered image of 4D Gaussians trained using viewpoint videos generated from 2D motion and 3D motion, and the optical flow between the resulting frames.
4D Gaussian
3D Motion
w/o Motion Initialization
3D Motion Initialization
Figure 5: The effect of 3D motion initialization is applied to 4D Gaussians training. The left part shows the trained result frames and the projected 3D motion. The right part compares the results with and without 3D motion initialization by extracting the optical flow from the 4D Gaussians results at the same time and viewpoints.
rendered the ground truth videos from novel viewpoints using 4 different camera trajectories from 3D-Cinemagraphy (Li al. 2023), resulting in 240 ground truth frames for each sample.
4.1.4 EVALUATION METRICS
The generated videos are compared with the ground truth frames at the same time and from the same view. We use Peak Signal-to-Noise Ratio (PSNR). Structural Similarity Index Measure (SSIM) (Wang et al. 2004),, and Perceptual Similarity (LPIPS) (Zhang et al. (018) as reference metrics, and Perception- based Image Quality Evaluator (PIQE) (Venkatanath et al. 2015) as the non-reference evaluation metric for our study. We introduced mask for the motion area to separately measure the moving regions. Additionally, in the ablation study, we utilized Lodd-Poin-EEmees Error (EPE), which average L2 distance between flows and is commonly used for measuring optical flow, to evaluate the effect of our 3D motion.
4.2 QUANTITATIVE RESULTS
In Table we show the quantitative results of our method compared to other baselines on reference and non-reference metrics. Our approach outperforms the other baseline all metrics in the text view generation. In particular, our method achieved the highest scores in PSNR, SSIM, and LPIPS, indicating that the generated views are high fidelity and perceptually similar the ground truth views. Furthermore, we demonstrated that our proposed method outperforms existing methods on non-reference metrics by locally measuring the extent of noise and distortion in images using PIQE /Venkatanath et al.] 2015). Additionally, we conduct user study on the generated videos, confirming that our model not only outperform in quantitative metrics also surpasses user experiences across four visual aspects.
4.3 QUALITATIVE RESULTS
In Fig. present qualitative comparison results with other baseline methods and ceffusiio--bseddd methods. this case, our proposed model, as an explicit representation, projected to 2D video for comparison of results. The process of separating the input image into L.DIs in 3DDCiinemagraphy

---

Published as a conference paper at ICLR 2025
[Li et al. 2023 leads to artifacts animated regions and fails to provide natural motion which results in reduced realism. Similarly, Make-It-4D (Shen et al. 2023) also utilizes LDIs to represent 3D for multi-view generation, which results in lower visual quality. Additionally, due unclear layer separation, objects appear fragmented or exhibit ghosting effects, where objects seem to leave behind afterimages. Likewise, DynamiCrafter (Xing al., (025) and Motion-I2V (Shi al. 2024). though capable of producing cinemagraphy. encounter challenges in accurately rendering the desired views due limited capabilities in view manipulation. In contrast, the proposed model represents complete 3D space with animations, providing less visual artifact and high rendering quality from various camera viewpoints. Therefore, our method provides more photorealistic results compared to others for various input images.
4.4
ABLATION STUDY
We encourage readers to visit the project page to explore our comprehensive video results.
4.4.1 3D MOTION OPTIMIZATION MODULE
Independently estimated 2D motion from multi-view images can result in different motion values for the same region in 3D space. Directly using these 2D motions to animate viewpoint videos can fail to train 4D Gaussians to represent natural motion. The estimated motions are projected to the center point through depth map for the same position measurement. Without the 3D Motion Optimization Module, the estimated flows show significant differences for the same positions, with an EPE of 0.152. On the other hand, with the 3D Motion Optimization Module, the consistency across entire viewpoints is outstanding, demonstrating almost negligible variance with an EPE of 0.003.
Fig. shows the visualized results of 2D motion and projected 3D motion. Similarly, it indi- cates that 3D motion represents motion information in 3D space, which ensures consistency when projected different viewpoints. We animated viewpoint videos using each motion and trained 4D Gaussians on multi-view videos. However, as shown in Fig. 4 (b), the rendered video of 4D Gaus- sians and estimated optical flow have the lack of motion consistency in the viewpoint videos caused unnatural movements.
4.4.2 EFFECT OF 3D MOTION INITIALIZATION
To verify the significance of 3D motion in training 4D Gaussians. we compared the results with and without 3D motion initialization. When applying animation fluids, we observed repeated patterns. Fig. shows rendered 3D motion and estimated optical flow from rendered video of each 4D Gaus- sian model. These results demonstrate challenges of accurately learning natural motion from videos without 3D motion initialization. When applying weakly supervised learning to deformation field by 3D motion, it accurately represents the motion from the generated multi-view videos.
4.5 APPLICATION: 4D SCENE GENERATION
To demonstrate the scalability of our framework, we extended its application to 4D scene generation by integrating it with existing 3D Scene Generation algorithms, such ViewCrafter (Yu et al. (024) and LucidDreamer (Chung et al. 2023). Experimental results, detailed in Appendix B validate the effectiveness of our method in generating consistent and high-fidelity 4D scenes.
CONCLUSION
This paper proposes method to model 4D Gaussians from a single landscape image to represent fluid-like motion within 3D space. Our approach efficiently estimates motion from multiple views using a 2D motion estimation model, and 3D-MOM optimizes the loss between 3D and 2D motion to produce consistent motion across views. The framework is applicable to various images, including those with complex depth variations, and its effectiveness has been validated through extensive experiments.
10

---

Published as a conference paper at ICLR 2025
6 REPRODUCIBILITY
The code used in our research is available full on the GitHub page where detailed usage instructions also provided. Masks and labeling for motion areas can be generated using the labelme program. The Holinsky dataset (Holyn- ski al., 202T). which was utilized in the research, is an existing public dataset that allows for experimentation on quantitative results. Furthermore, the data collected for qualitative results and further analysis have also been made publicly available. a result, the findings of this paper are fully reproducible, and additional video results are uploaded at the provided link.
comb
cvsp- aab/-/aa//// ab/ ICLR2025_ __-
7
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-0045615)) and regional broadcasting development support project funded by the Foundation for Broadcast Culture.
REFERENCES
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.122555 2023.
Jongwoo Choi, Kwanggyoon Seo, Amirsaman Ashtari, and Junyong Noh. Stylecinegan: Landscape cinemagraph generation using pre-trained stylegan. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7872-7881, 2024.
Yung-Yo Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H Salesin, and Richard Szeliski. Animating pictures with stochastic motion textures. ACM SIGGRAPH 2005 Papers, pp. 853-860, 2005.
Jacyoung Chung, Suyoung Lee, Hycongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:231 1.3384, 2023.
Dinh Trieu Duong, Minh Le Dinh, Byeungwoo Jeon, and Xiem HoangVan. A novel fusion method for 3d-tv view synthesis using temporal and disparity correlations. IEIE Transactions on Smart Processing & Computing, 8(6):431-44 2019.
Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama. Animating landscape: self-supervised learning of decoupled motion and appearance for single-image video synthesis. arXiv preprint 1XXY:19100001122 2019.
Siming Fan, Jingtan Piao, Chen Qian, Hongsheng Li, and Kwan-Yee Lin. Simulating fluids in real-world still images. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 15922-15931, 2023.
Aleksander Holynski, Brian L Curless, Steven M Seitz, and Richard Szeliski. Animating pictures with eulerian motion fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5810-5819, 2021.
Yi-Hua Huang, Yang- Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc- gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4220 4230, 2024.
Wei-Cih Jhou and Wen-Huang Cheng, Animating still landscape photographs through cloud motion creation. IEEE Transactions on Multimedia, 18(1):4-13, 2015.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
Johannes Kopf, Suhib Alsisan, Francis Ge, Yangming Chong, Kevin Matzen, Ocean Quigley, Josh Patterson, Jossie Tirado, Shu Wu, and Michael F Cohen. Practical 3d photography. Proceedings of CVPR Workshops, volume 1, pp. 2, 2019.

---

Published as a conference paper at ICLR 2025
Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michin Michael Frahm, Shu Wu, Matthew Yu, et al. One shot 3d photography. ACM Transactions Graphics (TOG), 39(4):76-1, 2020.
Yao-Chih Lee, Yi-Ting Chen, Andrew Wang, Ting-Hsuan Liao, Brandon Y Feng. and Jia- Bin Huang, Vividdream: Generating 3d scene with ambient dynamics. arXiv preprint arXiv:2405.20334, 2024.
Xingyi Li, Zhiguo Cao, Huiqiang Sun, Jianming Zhang, Ke Xian, and Guosheng Lin. 3d cinemag- raphy from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4595 4605, 2023.
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen-Phuoe, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint arXiv:2312.11456 2023.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaus- sians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:23/2. 137633 2023.
Elizaveta Logacheva, Roman Suvorov, Oleg Khomenko, Anton Mashikhin, and Victor Lempitsky. Deeplandscape: Adversarial modeling of landscape videos. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pp- 256-272. Springer, 2020.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint crXiv:2308.09713. 2023.
Aniruddha Mahapatra and Kuldeep Kulkarni. Controllable animation of fluid elements in still im- ages. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3667-3676, 2022.
Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Ying Lee, Sergey Tulyakov, and Jun- Yan Zhu. Text-guided synthesis of culerian cinemagraphs. ACM Transactions on Graphics (TOG), 42(6): 1-13, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-11)) 106, 2021.
Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 3d ken burns effect from single image. ACM Transactions on Graphics (ToG), 38(6): 2019.
Hao Ouyang, Kathryn Heal, Stephen Lombardi, and Tiancheng Sun. Text2immersion: Generative immersive scene with 3d gaussians. arXiv preprint arXiv:2312.09242, 2023.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference computer vision, pp. 12179-12188 1888 2021.
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dream- gaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142, 2023.
Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In Pro- ceedings of the 25th annual conference on Computer graphics and interactive techniques, pp- 231-242, 1998.
Richard Shaw, Jifei Song, Arthur Moreau, Michal Nazarczuk, Sibi Catley- Helisa Dhamo, and Eduardo Perez-Pellitero. Swags: Sampling windows adaptively for dynamic 3d gaussian splatting. arXiv preprint 1321:22312.13308, 2023.
12

---

Published as a conference paper at ICLR 2025
Liao Shen, Xingyi Li, Huiqiang Sun, Juewen Peng, Ke Xian, Zhiguo Cao, and Guosheng Lin. Make- it-4d: Synthesizing a consistent long- long-term term dynamic scene video from a single image. In Proceed- ings of the s1 ACM International Conference Multimedia, pp. 17777877, 2023.
Xiaoyu Shi, Zhaoyang Huang, Fu- Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image- to-video generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Meng-Li Shih, Shih-Yang
Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context- aware layered depth inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8028-8038, 2020.
Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 551-560, 2020.
Shubham Tulsiani, Richard Tucker, and Noah Snavely. Layer-structured 3d scene inference via view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 302-317, 2018.
Narasimhan Venkatanath, D Praneeth, Maruthi Chandrasekhar Bh, Sumohana S Channappayya, and Swarup Medasani. Blind image quality evaluation using perception based features. In 2015 twenty first national conference on communications (NCC), pp. 1-6. IEEE, 2015.
Qianqian Wang, Zhengqi Li, David Salesin, Noah Snavely, Brian Curless, and Janne Kontkanen. 3d moments from duplicupl photos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3906-3915, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Joachim Weickert. Coherence-enhancing diffusion filtering. International journal of computer vi- sion, 31:111-1211 1999.
Olivia Wiles. Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from single image. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7467-7477, 2020.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 20310-20320, 2024.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In European Conference Computer Vision, pp. 399 417. Springer, 2025.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with apatial-tempont consistency. arXiv preprint arXiv:2312.17225, 2023.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewerafter: Taming video diffusion models for high-fieelity novel view synthesis. arXiv preprint arXiv:2400) 2409..2048, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 586-595, 2018.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL RESULTS ON "IN-THE-WILD" DATASET AND BASELINES
We verified our results qualitatively and quantitatively using Holinsky al. (Hllynkki
202
which is commonly used for validation in the field of Dynamic Scene Video. Additionally, to com- pare the performance of our method with baseline models, we use our "in-the-wild dataset, which collect of global landmarks from online sources. Fig. 6 demonstrate that our model outper- forms baseline models by producing more realistic and stable videos across a variety of complex scenarios.
(b)
(c)
(6)
Fmaee
Fram=20 20
Fmmr 40
Prane 6ll
Franc
Frame 20
Frame 401
Frane
Fiane 1
Frano 20
Fame 40
Frunc 60
Figure 6: "in-the-wild" dataset Results. (a) 3D Cinemagraphy Shen et al. 2023), (c) Ours
(Li al. 2023). (b) Make-II-4D
14

---

Published as a conference paper at ICLR 2025
Frume
Frune 201
Frame 40
Frame
Frame
Frame 20
Frame 40
Frume 68
Frame
Frame
Frame
Frame 60
(9
Framc
Figure 7: "in-the-wild" dataset Results. (a) 3D Cinemagraphy (Li al. 2023). (b) Make-It-4D Shen et al. (202)). (c) Ours
B APPLICATION: 4D SCENE GENERATION
Our framework is designed to seamlessly incorporate 3D scene generation models, facilitating straightforward spatial expansion. Fig. 8 shows the results of incorporating the 3D Scene Gener-
15

---

Published as a conference paper at ICLR 2025
ation model, LucmeDreamer (Chung al. 2023), into method. LucidDreamer converts single images into point clouds and progressively fills empty areas with an inpainting model, enabling spatiotemporal expansion when incorporated into our framework. This incorporation enables the creation of videos with more natural motion and expansive views.
Additionally, we conduct comparisons with recent work on the 4D Scene Generation model, Vivid- Dream (Lee et al. 2024). Unlike our approach, VividDream directly generates multi-view videos through the T2V model without utilizing motion estimation for temporal expansion. Fig. com- pares the results of our framework, incorporating LucidDreamer (Chung et al. 2023), with those of VividDream (Lee et al., 2024). Our framework also integrates Viewcrafter Yu al. 2024). which enhances multi-view video generation, conditioned on point fusion pass. We render the comparisons using the closest matching images and cameras available, as the code and data were not disclosed. Since VividDream (Lee et al. (024) generates videos independently from multiple views, this approach results in motion ambiguity that leads to blurred reconstructions in fluid scenes, failing to accurately capture various motions. In contrast, our method estimates consistent 3D motion based on 2D motion, subsequently generating videos that achieve high-quality video with more natural motion.
videos in single dif-
Input Image
Frame
Frame 30
Frame 60
Frame 90
Ours LucidDreamer
Ours
Input Image
Frame 0
Frame 30
Frame 60
Frame 90
Ours LucidDreamer
Ours
Figure 8: Results of Our framework with LucidDreamer (Chung et al. 2023
16

---

Published as a conference paper at ICLR 2025
Input Image
Frame 0
Frame 30
Frame 60
Frame 90
Ours + ViewCraft
Ours LucidDreamer
DreidDream
Figure 9: Comparison results of Ours and VividDream (Lee al. 2024)
C
SINGLE IMAGE ANIMATION MODEL
Input Image
SLR-SFS
Text2Cinemagraph
StyleCineGAN
Figure 10: Comparison of rendered videos from 4D Gaussians trained by three different single image animation models. We optimized each 2D flows with 3D-MOM and trained the 4D Gaussians for comparison. By evaluating the visual results of the rendered videos, we can assess the effectiveness of the three single image animation models our franework.
In our model, it is crucial to utilize single image animation model that precisely estimate 2D mo- tion from Multi-view images and generate multi-view videos by accurately reflecting 3D motion. Fig. shows the results of trained 4D Gaussians using animated videos by different single image animation models, SIR-SFS (Fan et al. 2023). Text2Cinemagraph (Mahapatra et al. 2023) and StyleCineGAN (Choi al. 2024). The Fulerian flows estimated by each model enable the genera- tion of consistent 3D motion through 3D-MOM. which facilitates the restoration of natural motion in 4D scene. Additionally StyleCineGAN (Choi et al. (024)) can generate natural videos not only of fluids like water but also of clouds and smoke, allowing for the reconstruct various motions when utilized our framework. The results can be seen Fig. II As the field of single image anima- tion expands, we are able to generate consistent 3D motion from various 2D motions, allowing for expansion across a wider variety of scenes.
17

---

Published as a conference paper at ICLR 2025
Input Image
Frame
Frame 13
Frame 30
Frame 45
Ours StyleCineGAN
Figure 11: Results of Our framework with StyleCineGAN (Choi et al. 2024)
Input Image
Frame
Frame 30
Frame 60
Deformable-
Ours
Ours 4DGS
Ours Ours + Deformable-31
4DGS
Figure 12: Results of Our framework with Deformable-3D Yang al. 2024
D 4D GAUSSIAN SPLATTING
Since our framework utilizes 4D Gaussians to model complete 3D spaces with motion, the expres- siveness of the model itself significantly influences the quality of the final results. Fig. shows the results of implementing the Deformable-3D (Yang et al. 2024 within our framework. Compared to previous results using 4D-GS (Wu et al. 2024), it reconstructs low-fidelity 4D scenes and gener- ates videos with reduced realism. Therefore, by utilizing 4D-GS (Wu et al. 2024). framework capable producing more immersive Dynamic Scene Videos. This experiment demonstrates the adaptability of model, and as advancements made the field of 4D Gaussians, the perfor- mance of our framework also improves.
18

---

Published as a conference paper at ICLR 2025
E
EFFECT OF TWO-STAGE TRAINING
w/o Two-Stage Training
w/ Two-Stage Training
Figure 13: Effect the two-stage training on 4D Gaussians. Comparison of our model with and without two-stage training. The left part shows results trained with videos from all viewpoints, right part shows results trained with videos from sampled viewpoints.
Table 2: Comparison metrics and runtime with and without applying two-stage training. Through two-stage training, significantly reduced the overall runtime while quantitatively demonstrating that the results are comparable to those trained with videos from all viewpoints.
Metries
Runtime
Method
PSNR
SSIM
LPIPS
M.PSNR
M.SSIM
M.I.PIPS
Stepl
Step2
w/o stat running
18.17
0.91
0.18
22.10
0.96
0.08
1h 3m 39s
51m 34s
2-stage ranning
To achieve faster and more stable results with our algorithm, separated the 4D Gaussians learning process by viewpoints and time axis. In step 1, we trained 3D Gaussians using all viewpoints, and step 2, we trained 4D Gaussians using videos from sampled viewpoints.
Fig. 13 shows the results of training 4D Gaussians with animated videos for all viewpoints, while the bottom shows the results of two-stage training approach trained only three viewpoint videos. This demonstrates that our training method produces results almost identical to those obtained by training with videos from all viewpoints. Additionally, on sample validation set, our method not only maintains high performance but also achieves significant efficiency improvement.
shown Table 2 which was evaluated
over 30 times faster in generating videos and reduces the training time for the 4D Gaussians by third. It demonstrates an optimal balance between speed and accuracy.
19

---

Published as a conference paper at ICLR 2025
F LONG FRAME RESULTS
Recent diffusion-basse T2V models capable of simultaneously generating annlle images and cinemagraphy, similar to Dynamic Scene Videos, have emerged (Shi et al. 2024). (Xing et al,) 2025). However, these models experience sharp increase in computational load with the number of frames, limiting them to a maximum of 30 frames per inference and requiring lengthy inference times for each new view. In contrast, our framework
reconstruct long durations using explicit 4D Gaussians, allowing for the creation of novel view videos shorter time and at a lower cost. Fig. 14 demonstrates that our framework can produce long videos maintaining natural motion and high- fidelty, capable of generating up to 330 frames. The high compatibility of our framework ensures that the field of 4D Gaussians advances, the performance of framework also improves, enabling the production of longer Dynamic Scene Videos.
Frame
Frame
Frame 60
Frame 90
Frame 120
Frame 150
Frame 180
Frame 210
Frame 240
Frame 270
Frame 300
Frame 330
Frame 0
Frame 30
Frame 60
Frame 90
Frame 120
Frame 150
Frame 180
Frame 210
Frame 240
Frame 270
Frame 300
Frame 330
Figure 14: Qualitative results of the long frame experiment on our framework
20

---

Published as a conference paper at ICLR 2025
G DETAILED PROCESS OF MODEL SELECTION FOR OUR FRAMEWORK
In this appendix, we provide an in-depth examination the experimental process and the decision- making rationale involved in selecting the specific components of our framework. Our objective was to ensure a robust capability for high-fidelity 4D scene reconstruction, necessitating systematic ex- perimentation and domain-specific insights. For depth prediction models, we evaluated ZoeDepth (Bhat et al. 2023). noted for its zero-shot accuracy and broad adaptability, and DPT (Ranftl et al.) 2021). which utilizes advanced transformer technology for dense prediction tasks. ZoeDepth (Bhat er al. 2023) was selected due to its superior robustness and generalization across varied scenarios, aligning with our framework's s requirements. In the realm of 2D motion estimation, we tested mod- els including Holynski et al. (Holynski et al. (02II) approach for animating pictures with Eulerian motion fields and techniques for controllable animation of fluid elements in still images. Holynski al. (Holynski et al.) 2021) method was chosen for its ability generate consistent, looping imations, particularly effective for natural phenomena such as fluids and clouds. Regarding video generation, our evaluations included Text2Cinemagraph (Mahapatra et al. 2023). SLR-SFS((ann et al. 2023), and StyleCineGAN (Choi et al. 2022)), with SLR-SFS (Fan et al. 2023) being selected for its excellent balance between computational efficiency and temporal coherence, essential for our framework's 's integration needs. Lastly, for 4D Gaussian optimization, we compared 4D-GS (Wu al. 2024) and Deformable 3D Gaussian Splatting (Yang et al. (2044), opting for 4D-GS(Wu al., 2024 due to its efficiency in modeling spatiotemporal dynamics while maintaining high fidelity. This methodical approach to component selection ensures that our framework not only effective but also adaptable, paving the way for future enhancements and broad applications in dynamic scene reconstruction.
21


Reviews:
[{"review_id": "e8pSGlDKAW", "reviewer": "Reviewer_TeT9", "summary": "This paper presents a framework to form the 4D Gaussians representation for dynamic video generated from the single landscape image. To be more specific, the author builds the point cloud based on the single depth estimation. By reprojecting it to the multi view images, the static 3D Gaussians can be optimized. Then, the author proposed consistent 3D motions estimation module to lift the predicted 2D motions into the 3D space and further achieve the consistent motion in the 3D space with further optimization. Finally, the motion and 3D guassian can be used to optimize the 4D Gaussians as the final representation. The experiment conducted on the dataset from Animating Pictures with Eulerian Motion Fields demonstrating that the results surpasses several baseline methods.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experiment shows the performance surpass the baselines on the public dataset.\n2. The paper tries to solve the multi view consistency issue in the 3D generation by leveraging minimizing the reprojection error for the motion prediction. It is a good to attempt for this common challenging issue in this field.", "weaknesses": "1. The Sec. 3.1 for 3D Gaussians generation seems to just follow the previous work, Luciddreamer. Please correct me there is any additional novel effort for this part.\n2. Reprojecting the point cloud from the single view image to different views always leads to the holes, distortion and some other artifacts. A common idea to incorporate the generative model to fulfill the missing information. But it seems the author does not include anything related to this point. Please include more discussion if it is not necessary.\n3. Minimizing the reprojection error for the predicted motions seems to be feasible in Sec. 3.2, However, this optimization is still based on the 2D information and can hardly achieve the 3D consistency. For example, in Structure from Motions, such method is always used refine the estimated camera poses and the position for the point clouds. But it does not consider the inter relationship between each 3D positions, so I am not sure that the 3D consistency can be achieved with this method.  \n4. Please provide more implementation details of the 3D Motion Optimization Module. It seems to be vague now.\n5. The Sec. 3.3 for 4D Gaussians generation seems to just follow the previous work. Please correct me there is any additional novel effort for this part.\n6. The experimental results will be more solid and comprehensive if more dataset and baselines can be included. For example, the author mentions that simply using animation based method cannot achieve satisfying results, so similar baselines can be included.\n7. The paper presentation can be more compact if the part of just following the previous work can be shortened while emphasizing the novel part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pkFnfunW1e", "reviewer": "Reviewer_aadJ", "summary": "This paper proposes a method based on 4DGS to represent landscape 4D scenes from a single landscape image. It adapts Gaussian Splatting techniques for content with fluidity in the landscape images from the following aspects: \n-  3DGS methods require sparse points as initiations, while this paper uses existing depth prediction models to lift 2D pixels.     \n- This paper proposes a module named 3D Motion Optimization to generate 3D animations with off-the-shelf 2D motion methods.\nIt also conducts experiments over multiple datasets to show quantitative and qualitative comparisons over existing methods.", "questions": "1) What is the max temporal length the proposed method can achieve with good quality?\nIt has potential value if this paper can handle a long temporal range (with generated content). \n\n2) Can authors provide more technical insights? \nThis paper builds upon a lot of exciting techs (depth prediction model, 2D motion models). I would like to know if the authors selected these methods by experiments or random or with insights. If this paper can improve this part, it can benefit future work in this field.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Application to a new domain\n\nThis paper builds a solution to the problem of modeling landscape 4D scenes. \n\n2. Adpation of 3DGS / 4DGS\n\nThis paper adapts 3DGS from the initializations and 4D GS with motion prior.   \n\n3. Comparison to existing methods \n\nThe method shows better results compared to the existing methods.", "weaknesses": "1) The problem of generating Gaussians from landscape images with fluid content is useful but seems a bit too specific/narrow. If this paper can handle general 4D content, it would have a very large impact. \n2) Lack of theoretical innovation for ICLR. To address the task of fluid motion, this paper uses constant speed and direction. As 3DGS is a particle-like model, the theoretical contribution should be greater if this paper can present or show different physical-based models for fluids. \n3) Presentaiton of figures. Figure 2’s motion content is hard to see. Too many small arrows and small squares hurt the quality of the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GMj6glClBk", "reviewer": "Reviewer_NyCT", "summary": "This paper introduces a novel approach to optimize 4D Gaussian Splatting representations using single landscape images. By leveraging Eulerian motion fields, it effectively generates animations of dynamic scenes like waterfalls, beaches, and rivers. The authors also present the 3D-MOM method, which addresses frame inconsistency to ensure smooth animations. Experimental results demonstrate the method’s state-of-the-art quality.", "questions": "**Animating Pictures with Eulerian Motion Fields:** This technique primarily supports fluid motion, such as water and smoke. How does it handle other types of motion, like wind or more complex movements? I am curious about the potential future improvements of this method. Could it be adapted for human or animated motions in the future?\n\n **Handling $M_0$:** How is $M_0$ obtained? Is SAM or another method used?\n\nOverall, I think the demo and the presentation of the paper are good. My main consideration lies in weakness 2,3,4.\n\nBesides, I just think working on landscape images is quite small (not quite confident).  if the authors can demonstrate this pipeline could be applied in 4D scene generation, I would like to highly recommend accepting this paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose a novel method to animate 3D scenes with the flow-based method. It is quite novel and can fit the landscape generation properly.\n\n2. The results seem pretty good in certain demos and demonstrate the SOTA quality. \n\n3. 3D-MOM method is a good balance  when merging different 2D motion in multi-view images.", "weaknesses": "1. **Section 3.1 Citation:** It seems that a citation for 4D Gaussian Splatting (Wu et al., CVPR 2024) is missing in Section 3.1.\n\n2. **Comparison with VividDream:** I recommend that the authors include discussions or comparisons with VividDream (Lee et al., ArXiv 2024). Using a single image for initialization may not be robust enough, so it would be beneficial to expand the method to accommodate larger 3D scenes. Consider the motivations of viewcrafter/VividDream4D, as merely supporting \"zoom in\" functionality is insufficient. It’s essential to address additional capabilities like \"turn left/right\" or \"zoom out/up/down.\"\n\n3. **Multiview Generation and Novel Content:** Since only one image is used as input, the authors suggest that warping techniques can generate multiview images, but this approach may lack **novel content** and leave many empty regions. How do the authors handle these issues? If not resolved, I recommend using viewcrafter to enhance quality. Additionally, how are rendering paths chosen? Please provide more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRVbbV0omD", "reviewer": "Reviewer_WRaw", "summary": "This paper presents a novel framework for generating realistic 4D dynamic videos from a single landscape image. The approach uses a three-stage optimization process: first, it optimizes 3D Gaussians by generating multi-view images from the input; second, it estimates consistent 3D motion across views using the 3D Motion Optimization Module (3D-MOM); and third, it optimizes 4D Gaussians by applying the estimated motion to model changes in position, rotation, and scaling over time. This technique outperforms existing methods, such as 3D Cinemagraphy and Make-It-4D, by producing better qualitative results with reduced artifacts. The use of Gaussian Splatting contributes to improved PSNR and sharper video effects, as demonstrated in the supplementary videos.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Reasonable Pipeline: \n\nThe choice of Gaussian splatting offers better depth approximation compared to Layered Depth Images (LDI), leading to fewer artifacts in novel view synthesis.\n\n+ Natural 3D Motion Consistency: \n\nThe use of 3D consistent motions derived from 2D motion guidance results in more natural and coherent dynamics in the generated videos.\n\n+ Improved Performance: \n\nThe method demonstrates superior PSNR and visually appealing results compared to the approaches by Shen et al. and Li et al.", "weaknesses": "It has several limitations, where my main concerns are the implementation details and its evaluations.\n\n+ Gaussian Splatting Optimization:\n\nIn sec. 3.1, the explanation for both (a) camera settings and (b) multi-view optimization is unclear. Monocular depth estimation typically outputs normalized depth or disparity, which loses the depth scale. Additionally, a single image usually lacks intrinsic or extrinsic camera parameters. How were these parameters obtained?\n\nIn Eq. 3, what serves as the ground truth (GT) for multiview loss? A single image cannot provide multiple views for supervision unless additional views are available. However, the paper claims to only use a single image as input. How is Eq. 3 implemented in this case?\n\n+ Motion Estimation:\n\nIn sec. 3.2, there is some ambiguity regarding the motion estimation process. Are you using 2D optical flow? While projecting 3D motion into a 2D image would result in 2D optical flow, which offers rich motion data, the implementation is unclear.\n\nFor various views, such as in Fig. 1, it seems semantic masks (e.g., for water or fluid) are used. First, how do you get these masks?  Second, if you render a novel view with most black regions and it to the motion estimation model, how do you get a reasonable motion estimation result? The paper lacks details on this critical point.\n\n+ 4D Gaussian Training Process:\n\nIn sec. 3.3, the training process for 4D Gaussians is confusing. In pipeline 1 and 2, the method seems to already estimate 3D Gaussians and their motions. This suggests that you could simply deform the Gaussians to generate a video. However, training 4D Gaussians requires a video, but the input is a single 2D image.\n\nLine 343 mentions using a video generation model to produce a pseudo-video (is this using OpenSORA?), which is then used to create a 4D Gaussian with initialized motions and 3D points, supervised by the input image and generated video. Is this correct?\n\n+ Heavy Reliance on Pretrained Models:\n\nIf I understand correctly, the method seems to rely heavily on pretrained models. Specifically, it seems to (a) use video generation models to create a video, (b) apply depth and motion models to estimate initial depth and motion, and (c) lift these into 4D for dynamic novel view synthesis (NVS). This implies the core technique is lifting generated 4D content from 2D guidance, which could limit the novelty of the approach.\n\n+ Applicability to Specific Motions:\n\nThe method appears limited to handling fluid motions, such as water.\n\n+ Evaluation Metrics:\n\nPSNR may not be a meaningful metric for evaluating motion generation. A user study could provide more insightful feedback. Nevertheless, I acknowledge that the method shows better performance than the baselines, as demonstrated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)