GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation
Authors: Quankai Gao
Qiangeng Xu
Zhe Cao
Ben Mildenhall
Wenchao Ma
Le Chen
Danhang Tang
Ulrich Neumann
Venue: Accepted by TMLR ()

Abstract: Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be obtained efficiently by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to handle by existing methods. The common color drifting issue that occurs in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality in extensive experiments demonstrates the effectiveness of our method. As shown in our evaluation, GaussianFlow can drastically improve both quantitative and qualitative results for 4D generation and 4D novel view synthesis.

Published in Transactions on Machine Learning Research (06/2025)
Gaussian Splatting Gaussian Dynamics for 4D Content Creation
Quankai Gao University of Southern California
quankaig@usc.edu
Qiangeng Xu Google
giangenaz@igoonee.cmm
Zhe Cao Google
Zhecao wwgoooo..goooc.ocoom com
Ben Mildenhall Google
bmild@google.com
Wenchao Ma
wmm53900@psu.col edu
Pennsylvansa State University
Le Chen Max Planck Institute for Intelligent Systems
le. chen chen@ttucingen..ppy
Danhang Tang Google
dankangtang@ggoogge.coom
Ulrich Neumann
uneumann&uusc.eek
University of Southern California
Reviewed on OpenReview: https: openreview. net/ forum? id= XBL 7w15rt0
(a) 4D Generation
(b) 4D Novel View Synthesis
Figure 1: We propose GaussianFlow, a dense 2D motion flow created by splatting 3D Gaussian dynamics,
which significantly benefits tasks such as 4D generation and 4D novel view synthesis. (a) Based on monocular videos generated by Lumiere Bar-Tal al. (2024) and Sora Brooks al. (2024), our model can generate 4D Gaussian Splatting fields that represent high-quality appearance, geometry and motions. (b) For 4D novel view synthesis, the motions in generated 4D Gaussian fields smooth and natural, even in highly dynamic regions where other existing methods suffer from undesirable artifacts.
Abstract

---

Published in Transactions on Machine Learning Research (06/2025)
Creating 4D fields of Gaussian Splatting from images or videos is challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the imput videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be obtained efficiently by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to handle by existing methods. The common color drifting issue that occurs in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality in extensive experiments demonstrates the effectiveness of our method. As shown in our evaluation, GaussianFlow can drastically improve both quantitative and qualitative results for 4D generation and 4D novel view synthesis. Project page: litps:: https://zeerg-owe //wer//-veeeiiidd...mm wttpthubb.o//.uusseeee Flow. wtw.github.ioo/
1 Introduction
4D dynamic content creation from monocular multi-view videos has garnered significant attention from academia and industry due to its wide applicability in virtual
reality, digital games, and movie industry. Studies (Li et al.] 2022; Pumarola el al. 2021 Park et al. 2021a]b) (b model 4D scenes using 4D dynamic Neural Radiance Fields (NeREs) and optimize them based on input multi-view or monocular videos. Once optimized, the 4D field can be viewed from novel camera poses at preferred time steps through volunetric rendering. A more challenging task is generating 360 degree 4D content based on uncalibrated monocular videos or synthetic videos generated by text-to-video or image-to-video models. Since monocular input cannot provide enough multi-view cues and unobserved regions are not supervised due to occlusions, studies (Singer et al. 2023; Jiang et al.) 2023 Zhao et al. 2023 optimize 4D dynamic NeRFs by lever- aging generative models to create plausible and lemporally consistent. 3D structures and appearance. The optimization of 4D NeRFs requires volumetric rendering, which makes the process time-consuming. And real-time rendering of optimized 4D NeRFs also hardly achieved without special designs. A more efficient alternative is to model 4D radiance fields by 4D Gaussian Splatting (GS) (Wu et al.) 2023 Luiten et all, 2023), which extends 3D Gaussian Splatting (Kerbl et al., 2023) with temporal dimension. Using the efficient rendering of 3D GS, the lengthy training time of 4D Radiance Field can be drastically reduced (Yang et al. 2023c Ren et al. 2023 and rendering can achieve real-time speed during inference.
The optimization of 4D Gaussian fields takes photometric loss as the major supervision. As a result, the scene dynamics are usually under-constraint. Similarly to 4D NeRFs (Li ct al. 2023 Park et al., 2021a
Pumarola et. al.) 202T), the radiance properties and the lime-varying spatial properlies (location, scales, and orientations) of Gaussians are both optimized to reduce the photometric Mean Squared Error (MSE) between the rendered frames and the input vidco frames. The ambiguities of appearance, geometry, and dynamics have been introduced in the process and become prominent with sparse-view or monocular video input. Per- frame Score Distillation Sampling (SDS) (Tang et al. 2023) reduces the appearance-geomette ambiguity to some extent by involving multi-view supervision in the latent domain. However, both monocular photometric supervision and SDS supervision do not directly supervise scene dynamics.
To avoid temporal inconsistency caused by fast motions, Consistent4D (Jiang et al. 2023) leverages a video interpolation block, which imposes a photometric consistency between the interpolated frame and the generated frame, at the cost of involving more frames pseudo ground truth for fitting. Similarly, AYG (Ling et al. 2023) uses the text-to-video diffusion model to balance motion magnitude and temporal consistency with preset frame rate. 4D NeRF model [Li et al. 2023) has proven that optical flows in reference videos are strong motion cues and can significantly benefit scene dynamics. However, for 4D GS, connecting 4D Gaussian motions with optical flows has the following two challenges. First, a Gaussian motion is in 3D space, but it is its 2D splat that contributes to rendered pixels. Sccond, multiple 3D Gaussians might contribute to the same pixel in rendering, and each pixel's flow does not equal to any one Gaussian's 's motion.
2

---

Published in Transactions on Machine Learning Research (06/2025)
To overcome these challenges, we introduce novel concept, Gaussian flow, bridging the dynamics of 3D Gaussians and pixel velocities between consecutive frames. Specifically, we assume that the optical flow of each pixel in the image space is influenced by the Gaussians that cover it. The Gaussian flow of each pixel is considcred to be the weighted sum of these Gaussian motions in 2D. To obtain the Gaussian flow value on cach pixel without losing the speed advantage of Gaussian Splatting, we splat 3D Gaussian dynamics, including scaling, rotation, and translation in 3D space, onto the image plane along with its radiance properties. As the whole process is end-to-end differentiable, the 3D Gaussian dynamics can be directly supervised by matching Gaussian flow with optical flow on input video frames. We apply such flow supervision to both 4D content generation and 4D novel view synthesis to showcase the benefit of our proposed method, especially for contents with rich motions that are hard to handle by existing methods. The flow-guided Guassian dynamics also resolve the color drifting artifacts that are commonly observed in 4D Generation. We summarize our contributions as follows:
We introduce a novel concept, Gaussian flow, that first bridges the 3D Gaussian dynamics to resulting pixel velocities, enabling flow supervision for Gaussian Splatting based representations. Matching Gaussian flows with optical flows, 3D Gaussian dynamics can be directly supervised.
The Gaussian flow can be obtained by splatting Gaussian dynamics into the image space. Following the tile-based design by original 3D Gaussian Splatting, we implement the dynamics splatting in CUDA with minimal overhead. The operation to generate dense Gaussian flow from 3D Gaussian dynamics is highly efficient and end-to-end differentiable.
With Gaussian flow to optical flow matching, our model drastically improves over existing Gaussian Splatting based-methods, especially on scene sequences of fast motions. Color drifting is also resolved with our improved Gaussian dynamics.
2
Related Works
3D Generation. 3D generation has attracted tremendous attention with the progress of various 2D 3D-
aware diffusion models [Liu et al. 2023b Rombach et al. 2022 Shi et al.
2023c and large 2023b Liu et al.
vision models Radford et al. (2021]; Jun & Nichol (2023): : Nichol et al. (2022). Thanks to the availability of
2022). object-level large-scale multi-view image datasets (Deitke et al. 2023 Yu et al. 2023 Downs et al.)
multi-view cues can be encoded in generative models and are used for generation purpose. Pioneered by DreamFusion (Poole et al., 2022) that first proposes Score Distillation Sampling (SDS) loss to lift realistic content from 2D to 3D via NeRFs, 3D content creation from text or image input has flourished. This progress Wang et al. 2024c;
includes approaches based on online optimization (Tang et al.] 2023 Lin et al. 2023
Raj et al. 2023) and feedforward methods (Hong et al. 2023 Liu el al. 2023a 2024
Xu et al. 2023 Wang
et al. 2023c) with different representations such as NeRFs Mildenhall et al. (2021).
triplane Chan et al. 2023). 3D generation 2023b) and 3D-aware diffusion models (Liu et al. 2023bb as SDS supervision. Not limited to high-quality rendering, studies (Sun)
2022
Chen et al.) 2022 Gao et al. 2023) and 3D Gaussian Splatting (Kerbl et al.
becomes more -ielt consistent by involving multi-view constraints (Shi et al.,
et al. 2023 Long et al. 2023) also explore enhancing the quality of generated 3D geometry by incorporating
normal cues.
4D Novel View Synthesis and Reconstruction. By adding timestamp as an additional variibille recent 4D methods with different dynamic representations such as dynamic NeRF (Park et al. 2022a b Li et al.,
2021 Wang et al.
2023a
Li et al. 2022: Tretschk et al. 2021 Gao et al. 2021), dynamic triplane Fridovich-
Keil et al. (0023)
Cao & Johnson (2023 ; Shao et al. (0023) and 4D Gaussian Splatting Huang et al.
[2024b; Wu et al.
(2023); Yang et al. (002cch: Lin et al. (0024) Yang et al. (002);; Duan et al. (2024)
Guo ct al. (2024) are proposed to achieve high quality 4D motions and scene contents roconstruction from
either calibrated multi-view or uncalibrated RGB monocular video inpuls. Some works (Newcombe et al.,
2011 2015, Zollhöfer et al.) 2014) reconstruct rigid and non-rigid scene contents with RGB-D sensors, which help to resolve 3D ambiguities by involving depth cues. Different from static 3D reconstruction and novel view synthesis, 4D novel view synthesis consisting of both rigid and non-rigid deformations is notorionsly
challenging and ill-posed with only RGB monocular inputs. Some progress [Li et al. 2021
Gao et al.
3

---

Published in Transactions on Machine Learning Research (06/2025)
Image Space
Canonical Gaussian space
Image Space
F2
14
j
X1
0
Xj,02
(0,0)
MiAz
jo
j
flow = 3j,2² X12
X1,12
5
Mac
(0,0)
pltzz
f4
flow = X1,22" Xtz
f
t=t
t=t2
flowf
flowf 1.t
1:3D to 2D Projection :DD/2D Gaussian
image plane
We illustrate our GaussianFlow formulation using two Gaussian primitives. Flow Composition (Left): the two Gaussians, indexed by and j are shown at two consecutive time steps 4 and l2. The Gaussian flow on the yellow pixel 2t, is defined as the weighted sum of the shift contributions from all Gaussians and j) covering the pixel. The weighting factor utilizes alpha composition weights the same as in the volume rendering. The Gaussian flow of the entire image can be obtained efficiently by splatting 3D Gaussian dynamics and rendering with alpha composition. Flow from Single Gaussian (Right): Between two consecutive frames, the yellow pixel It, will be pushed towards _t, - Zi,t, by the 2D Gaussian i's motion g + i'2 where we denote this shift contribution from Gaussian as flow 32 We can track dt, Gaussian by normalizing it lo canonical Gaussian space as &, and unnormalize it to image space to obtain 22, A similar process applies for Gaussian j.
Figure 2:
2021
involves temporal priors and motion cues (e.g. optical to better regularize temporal photometric consistency and 4D motions. One of recent works (Wang) 2023a) provides an analytical solution for flow supervision on deformable NeRF without inverting the backward deformation function from the world coordinate to the canonical coordinate. Several works (Yang)
Tretschk et al. 2021 Wang et al. 2021 2025
flow)
et al.
et al.
2021 2023a b) explore object-level mesh recovery from monocular videos with optical flow.
4D Generation. Similar to 3D generation from text prompts or single images, 4D generation from text prompts or monocular vidcos also relics on framo- by-famo multi-view eues from pro-trained diffusion models. Besides, 4D generation methods yet always rely on either video diffusion models or video interpolation block to ensure the temporal consistency. Animate124 (Zhao et al. 2023). 4D-fy (Bahmani et al., 2023 and of the earliest works Singer et al. (2023) use dynamic NeRF's as 4D representations and achieve temporal consistency with text-to-video diffusion models, which can generate videas with controlled frame rates. Instead of using dynamic NeRF, Align Your Gaussians (Ling et al. 2023) DreamGaussian4D (Ren et al. 2023) and L4GM Ren et al. 20244) generate vivid 4D contents with 3D Ganssian Splatting, but again relying on video diffusion models for free frame rate control. Without the use of text-lo-video diffusson models, Consistent<4 (Jiang et al. 2023) achieves coherent 4D generation with an off-the-shelf video interpolation model (Huang et al. 2022]. Our method benefits 4D Gaussian representations by involving flow supervision and without the need of specialized temporal consistency networks.
3 Methodology
To better illustrate the relationship between Gaussian motions and corresponding pixel flow in image space, first. recap the rendering process of 3D Gaussian Splatting and then investigate its 4D case.
4

---

Published in Transactions on Machine Learning Research (06/2025)
3.1 Preliminary
3D Gaussian Splatting. From set of initialized 3D Gaussian primitives, 3D Gaussian Splatting aims to recover the 3D scene by minimizing the photometric loss between input m images {4 and rendered images hm For cach pixel, its rendered color C is the weighted sum of multiple Gaussians' colors c; in depth order along the ray by point- based a- a-blending as in Eq.
C
Tiaa
(1)
with weights specifying as
i-1
04 o3e 0 -xx- (x-u.)) and II - (g)))).
(2)
where E [0, Ma E [221, and E; E R2 are the opacity, 2D mean, and 2D covariance matrix of i-th 2D Gaussian projected from 3D space, respectively. And x is the intersection between a pixel ray and i-th Gaussian. As shown in Eq- the relationship between a rendered pixel and 3D Gaussians is not bijective.
3D Gaussian Splatting in 4D. Modeling 4D motions with 3D Gaussian Splatting be done frame-by- frame via either directly multi-view fitting (Luilen et al. 2023 moving 3D Gaussians with a time-variant deformation field (Ling et al. 2023 Ren et al. 2023) or parameterixing 3D Gaussians with time (Yang et al. 2023cp. While with monocular inputs, Gaussian motions are under-constrained because different Gaussian motions can lead to the same rendered color, and thus long-term persistent tracks are lost (Luiten et al. 2023). Though Local Rigidity Loss (Tuiten et. al. 2023 Ling et al., 2023 is proposed to reduce global freedom of Gaussian motions, it sometimes brings severe problems due poor or challenging initialization and lack of multi-view supervision. As shown in Fig. 0 3D Gaussians initialized with the skull mouth closed
are hard to be split when the mouth open with Local Rigidity Loss.
3.2 GaussianFlow
We consider the full freedom of each Gaussian motion in 4D field, including 1) scaling, 2) rotation, and 3) translation at each lime step. As time changes, Gaussians covering the queried pixel at / t= i will move to other places at t = t2, as shown in Fig, leftt. To specify new pixel location X;, at t f= t2, we first project all the 3D Gaussians into 2D image plane as 2D Gaussians and calculate their motion's influence on pixel shifts.
Flow from Single Gaussian. To track pixel shifts (flow) contributed by Gaussian motions, we let the rel- ative position of pixel in a deforming 2D Gaussian remain the same. This setting preserves the mahalanobis distance between the pixel locations under two consecutive time steps and the 2D Gaussian unchanged. Ac- cording to Eq. 2 this preservation will grant the pixel with the same radiance and a contribution from the 2D Gaussian, albeit the 2D Gaussian is deformed.
The pixel shift (flw) is the image space distance of the same pixel at two time steps. We first calculate the pixel shift influenced by a single 2D Gaussian that covers the pixel. As shown in Fig. 2 rightt, we can find pixel x's location at to by normalizing its image location at t1 to canonical Gaussian space and unnormalizing it to the image space in tz:
1) normalize. A pixel Xt, following i-th 2D Gaussian distribution can be written as Xt, ~ N(p E And in i-th Gaussian coordinate system with 2D mean μ4,t, R2x1 and 2D covariance matrix E,,, E 2 After normalizing the i-th Gaussian into the standard normal distribution, we denote the pixel location canonical Gaussian space as
8, -B11 (Xe, (...)):
(3)
which follows Σig; BB,,, BY,, &, N(O,T) and I E R2x2 is identity matrix.
5

---

Published in Transactions on Machine Learning Research (06/2025)
2) unnormalize. When 12, the new location along with the Gaussian motion denotes X1,12 the image plane.
X1.tz
Bi St, + Mi,to
(4)
and Di,r Bi, B Xt, (ui,t 2.,,)) Eq. and Eq. preserve the Mahalanobis distance between the tracked pixel and the 2D Gaussian, leading to consistent value of (see 2 for this pixel over consecutive time steps. The pixel shift contribution from each Gaussian therefore can be calculated as:
G flow Xi.t2 Xt1
(5)
Flow Composition. In the original 3D Gaussian Splatting, pixel's color is the weighted sum of the 2D Gaussians' radiance contribution. Similarly, we define the Gaussian flow value at a pixel as the weighted sum of the 2D Gaussians' contributions to its pixel shift, following alpha composition. With Eq. Band Eq. 4
the Gaussian flow value at pixel X from = 4 to = 4,2 is
K
Mowii
Σ
wi wi flow
(6)
im1
K
Bi B (Xe, Mixt) + Mi.t xx,)]
(7)
2=1
where K is the number of Gaussians along each camera ray sorted in depth order and each Gaussian has
weight wi = L E,T.o according to Eq. 1 but normalized to [0,1] along each pixel ray. The intuition behind the use the same weight as a-blending that, pixel color contributed by a weighted sum of a set of Gaussians, then its corresponding pixel shift i.c. pixel-wised optical flow should also be contributed by the same set of Gaussians with the same weights by nature, since optical flow is calculated based on the pixel-wised correspondences well.
In some cases Ling et al. (003): Keetha et al. (023 Yugay et al. 2023
Matsuki et al. 2022) cach Gaussian is assumed to be rsotro and its scaling matrix = oI, where a is the scaling factor. And its 3D covariance matrix RSST I'I. the scaling factor of each Gaussian does not change too much over time, B B Rs I. Therefore, to pair with this line of work, the formulation of our Gaussian flow as in Eq. can be simplified
flow Σ w p.,01).
In other words, for isotropic Gaussian fields, Gaussian flow between two different time steps be approx- imated as the weighted sum of individual translation of 2D Gaussian.
Following either Eq. or Eq. the Gaussian flow can be densely calculated at each pixel. The flow supervision at pixel Xt, from t f to = t2 can then be specified as
f = [[flow,, (x,,) - flow iyta
(9)
where optical flow flowf, can be calculated by off-the-shelf methods as pseudo ground-truth. Our met also allows for camera motions, please refer to our experiments on NeRF-DS dataset (Yan et al. 2023 for more dctails.
3.3 4D Content Generation
As shown in Fig. 8 4D content generation with Gaussian representation takes an uncalibrated monocular video either by real capturing or generating from text-to-vidco or imago-to-video models as input and output 4D Gaussian field. 3D Gaussians are initialized from the first video frame with photometric supervision between rendered image and input image and 3D-aware diffusion model (Liu et al. 2023b for multi-view

---

Published in Transactions on Machine Learning Research (06/2025)
Image-to- Video
Optical Flow
[ t2
bn-1
1
t2
n-
E
Flow MSE
Photometric MSE
Photometric MSE
Gaussian Flow C2
Flow MSE
SDS
SDS
Gaussian Flow n--14n
Zero-1-2-3
Zero-1-2-3
Gaussian Initialization
4D Gaussian Field Optimization
Figure Overview our 4D content generation pipeline. An uncalibrated monocular vidoo or video generated from an image is taken as the input. We optimize a 3D Gaussian field initialized by the first frame with both photometric and SDS supervision (Liu et. al. 2023b) (for 4D generation only). Then, we optimize the dynamics of the 3D Gaussians with the same two losses for each frame. Most importantly, Gaussian flows with our novel design on reference view for each consccutive two time steps and match it with a pre-computed optical flow of the input video. The gradients from the flow matching will propagate back through dynamics splatting and rendering process, resulting in a 4D Ganssian field with natural and smooth motions.
calculate
SDS supervision. In our method, 3D Gaussian initialization can be done by One-2-3-45 (Liu et al. 2024)
or DreanGaussian (Tang et al. 2023). After initialization, a 4D Gaussian field is optimized with per-frame photometric supervision, per-frame SDS supervision, and our flow supervision as in Eq. The loss function for 4D Gaussian field optimization can be wrilten as:
Lphotometric X1 Xion Azhadd + Aghothers
(10)
where X1, Ag and Ag are hyperparameters. Lother is optional and method-dependen Though not used in our method, we leave it for completeness.
3.4
4D novel view Synthesis
Unlike 4D content generation that has multi-view object-level prior from the 3D-aware diffusion model, 4D novel view synthesis takes only multi-viow or monocular input vidco frames for photometric supervision without any scene-level prior. Gaussians are usually initialized by sfn (Snavely et al. 2006 Schoobergerr & Frahm 2016) from input videos. After initialization, the 4D Gaussian field is optimized with per per-frant frame
photometric supervision and our flow supervision. We adopt the 4D Gaussian field from (Yang al. 2023c
The loss function for 4D Gaussian field optimization can be written as:
L = Lphotometric + X1 L + Aghothers
(11)
where Lother is optional and method-dopendes (please refer Yang al. (2023c
4 Experiments
In this section, we first provide implementation details the proposed method and then validate our method 4D Gaussian representations with (1) 4D novel view synthesis and (2) 4D generation. We tested

---

Published in Transactions on Machine Learning Research (06/2025)
the Plenoptic Video Datasets Li ct al. 2022 and the Consistent4D Dataset (Jiang et al. 2023 for both quantitative and qualitative evaluation. Our method achieves state-of-the-art results in both tasks. To obtain dense Gaussian flow, we efficiently splatting the Gaussian dynamics along with the original 3DGS et al, CUDA pipeline. Please refer to our supplemental materials for implementation details.
2023
4.1 Dataset
Plenoptic Video Dataset. A high-quality real-world dataset consists of scenes with 30FPS and 2028 x 2704 resolution. There are 15 to 20 camera views per scene for training and 1 camera view for testing. The cameras are distributed to face the frontal part of scenes from different angles.
NeRF-DS Dataset. This dataset Yan et al. 2023 consists of 8 scenes in everyday environments with various lypes of moving or deforming specular objects. Each scene contains two videos captured by two rigidly mounted feward-facing cameras.
Consistent4D Dataset. This dataset (Jiang al. 2023 includes 14 synthetic and 12 in-the-wild monoe- ular videos. All the videos have only one moving object with white background. 7 of the synthetic videos are provided with multi-view ground-truth for quantitative evaluation. Each imput monocular video with a static camera is set at an azimuth angle of 0 Ground-truth images include four distinct views at az- imuth angles of -75", 15,, 105", and 195", respectively, while keeping elevation, radius, and other camera parameters the same with input camera.
4.2 Results and Analysis
4D Novel View Synthesis. We visualize rendered images and depth maps of a very recent state-of-the-art 4D Gaussian method RT-4DGS (Yang et al. 2023c) with (yellow) and without (red) our flow supervision in Fig. q According to zoom-in comparisons, our method can consistently model realistic motions and correct structures. These regions are known to be challenging (Verbin et al. 2022 Liu et al., 2023d for most methods, even under adequate multi-view supervision. Our method can reduce ambiguities photometric supervision by involving motion cues and is shown to be consistently effective across frames. By using an off-the-shelf optical flow algorithm (Shi et al., 2023a),, we found that only small portion of image pixels from the Plenoptic Video Dataset have optical flow values larger than one pixel. Since our method benefits 4D Gaussian-based methods more on the regions with large motions, we report PSNR numbers on both full scene reconstruction and dynamic regions (optical flow value > 1) in Tab. [ With the proposed flow supervision, our method shows improved performance on all scenes and the gains are prominent on dynamic regions. Consequently, our 4D novel view synthesis results achieve state-of-the art quality. Both qualitative and quantitative comparisons on the NeRF-DS dataset in Fig. [0 and Tab. P show the effectiveness of the proposed method on scenes with complex camera motions. We also report PSNR, D-SSIM, LPIPS averaged over all scenes of the DyNeRF dataset (Li et al. 2022) in Tab. 5 More comparisons are shown in the Fig. 13 and the video in our supplemental material.
4D Generation. 4D Gaussian-baseel state-of-the-art generative model with open-sourced code, and dynamic N methods D-NeRF (Pumarola al. 2021), K-planes (Fridovich-Keil et al. 2023) and Consistent-ID (Jiang caleulated and averaged
We evaluate and compare our method with DreamGaussiannDD (Ren et al, 2023 a recent
et al. 2023) in Tab. on Consistent dataset. Scores on individual videos over four novel views mentioned above. Note that flow supervision is effective and helps with 4D generative Gaussian representation. Compared to DreamGaussian4D, our method shows better quality shown in Fig. Gafter the same number of training ilerations. For the two challenging cases shown in Fig. 6 our hod benefits from flow supervision and generates desirable motions, while DG4D shows prominent artifacts on the novel views. Furthermore, flow supervision helps our method avoid color drifting, compared with the dynamic NeRF-basod method Consistent (Jianng et al., 2023) (Fig. 5p. Our results are more consistent in terms of texture and geometry. We also show more generation results in Fig. of the supplemental material.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Quantitative evaluation between ours and other methods on the DyNeRF dataset Li et al. (0022) We report PSNR numbers on both full-scene novel view synthesis and dynamic regions where the ground- truth optical flow value larger than one pixel. "Ours" denotes RT-4DGS with the proposed flow supervision. We also achieve the best results D-SSIM and LPIPS (sce the Tab. and 4
in the supplemental material).
Method
Coffee Martini
Spinach
Cut Beef
Flame Salmon
Flame Steak
Sear Steak
Mean
HexPlane Cao & Johnson
2023
32.04
32.55
29.47
32.08
32.39
31.70
K-Planos
Prdovich- Keil
2203
29.99
32.60
31.82
30.44
32.38
32.52
31.63
MixVoxels
Wang et
2023b
29.36
31.30
29.92
31.21
31.43
30.80
NeRFPlayer Song et
2122
31.53
30.56
29.35
31.93
29.12
30.69
HyperReel Attal
2023
28.37
32.30
32.92
28.26
32.20
32.37
31.10
4DGS
27.34
32.46
32.51
32.49
31.15
RT-4DGS
Yang et al.
2023c
28.33
32.93
33.85
29.38
34.03
33.51
32.01
Ours
28.42
33.68
34.19
29.37
34.22
34.06
32.32
Dynamic Region Only
RT-4DGS
Yang al.
2023c
27.36
27.47
34.48
23.16
26.04
29.52
28.00
Ours
28.02
28.71
35.18
23.36
27.53
31.14
28.99
Figure Qualitative comparisons on DyNeRF dataset (Li et al. 2022 The left column shows the novel view rendered images and depth maps of RT-4DGS (Yang al. (022c). which suffers from artifacts in the dynamic regions. The right column shows the results of RT-4DGG optimized with our flow supervision during training. We refer to supplementary material (Fig. comparisons.
including the video) for more visual
5 Ablation Study
We validate our flow supervision through qualitative comparisons shown in Fig. 6 Compared with Ours (no flow) and Ours, the proposed flow supervision shows its effectiveness on moving parts. For the skull, 3D Gaussians in the teeth region initialized at t move freely long as they look photometrically correct from view
very close to each other and it is hard split apart
completely when t i2 Gaussians 0, while SDS supervision applicd from novel views works on latent domains and cannot provide pixel-wised supervision. This problem becomes more severe when involving Local Rigidity Loss (comparing Ours-r and Ours) because the motions of 3D Gaussians initialized at t t= fj are constrained by their neighbors and the Gaussians are harder split apart at 41- Similarly, for bird, regions consisting of thin structures such the bird's beak cannot be perfectly maintained across frames without our flow supervision. While originally utilized in 4D Gaussian fields [Luiten al.] 2023 to maintain structure consistency during motion, Local

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Quantitative comparisons on NeRF-DS dataset Yan et al. 2023
"Ours" denotes Deformable-
3DGS with the proposed flow supervision. Please refer section D. our supplementary material for more details and qualitative results.
PSNR 1
SSIM
LPIPS|
3DGS (Kerbl et al. 2023
20.79
0.78
0.29
TiNeuVo Fang et al. 2022
21.60
0.83
0.28
HyperNeRF Park et al.
2021b
23.45
0.85
0.19
NeRF-DS Yan et al.
2023
23.60
0.85
0.18
Deformable-3DGr
Yang et al.
2024
23.89
0.84
0.18
Ours
24.23
0.85
0.17
Table 3: Quantitative comparisons between ours and others on Consistent4D dataset.
Pistol
Gopple
Crocadile
Monster
Skall
Trump
Aurorus
Moan
Method
CLIP1
LPIPS,
CLIP1
LPIPS]
LPIPSL
CLIP1
LPIPS)
LPIPS]
LPIPS)
CLIP1
LPIPS]
D. NeftF
0.32
0.66
0.32
0.76
0.54
0.61
0.52
0.79
0.53
0.72
0.35
0.60
0.58
0.66
0.51
0.68
K-glanes
0.74
0.29
0.75
0.73
0.51
0.08
0.67
0.36
Consissent
0.10
0.90
0.12
0.90
0.12
0.82
0.18
0.90
0.17
0.88
0.23
0.85
0.17
0.83
0.16
0.87
DC4D
0.12
0.92
0.12
0.91
0.12
0.58
0.19
0.50
0.18
0.30
0.22
0.81
0.17
0.85
0.16
0.87
Table 4: SSIM, MSSIM, D-SSIM and LPIPS of our method on the DyNeRF dataset breakdown by scenes.
Coffee Martini
Spinach
Cut Beef
Flame Salmon
Flame Steak
Sear Steak
Mean
SSIM t MSSIM
0.9185
0.9578
0.9598
0.9248
0.9643
0.9645
0.9483
0.9544
0.9786
0.9808
0.9597
0.9816
0.9808
0.9726
D-SSIM +
0.0228
0.0107
0.0096
0.0202
0.0092
0.0096
0.0137
LPIPS &
0.0708
0.0389
0.0378
0.0639
0.0337
0.0354
0.0468
Table 5: Overall quantitative comparisions between ours and other methods on the DyNeRF dataset
et al.
2022 We report PSNR, D-SSIM, LPIPS averaged over all scenes. "Ours" denotes RT-4DGS with
the proposed flow supervision.
Mean PSNR t
Mean D-SSIM 1
Mean LPIPS J.
HexPlane
Cao & Johnson
22023
31.70
0.014
0.075
K-Planes Fridovich- Keil et al.
(0233
31.63
0.018
Mix Voxels Wang et al.
20236
30.80
0.02
0.126
NeRFPlayer Song
2023
30.69
0.034
0.111
Hyper Reel
Attal al.
2023
31.10
0.036
0.096
4DGS Wu et al. (2023
31.15
0.016
0.150
RT-4DGS Yang et al.
2003cc
32.01
0.014
0.055
Ours
32.32
0.014
0.047
Rigidity Loss motion constraint can incorrectly group Gaussians and is less effective than our flow supervision.
We also visualize optical flow flow, and Gaussian flow flow. with and without our flow supervision in Fig. 7 In both cases, the optical flow flows, between rendered images on the input view are very similar to each other (shown #1 and # 4 column) and align with ground-truth motion because of direct photometric supervision on input view. However, comparing optical flows on novel view as shown in #3 and #6, without photometric supervision on novel views, inconsistent Gaussian motions are witnessed without flow supervision. Gaussian flow flow in #2 column also reveals the inconsistent Gaussian motions.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Inputs
Con4D
Ours
Inputs
Con4D
Ours
t=t
t=tz
t=13
Figure Comparisons between Consistent (Jiang et al. 2023) (a dynamic NeRF-based method) and ours.
The flow supervision help us avoid the "bubble like" texture and non-consistent geometry on novel views.
Input
DG4D
Ours flow)
Ours-r
Ours
view
view
2
view
view
view
view 2
view 2
t=t
t t=
t=
t=
Figure 6: Qualitative comparisons among DresmGaussian4D (Ren et al. 2023 method without flow loss, our method without flow loss but with Local Rigidity Loss (Ours-r) and ours.
Incorrect Gaussian motion can still hallucinate correct image frames on input view. However, this motion- appearance ambiguity can lead to unrealistic motions from novel views (the non-smooth flow color on moving parts in #3). While #5 shows consistent Gaussian flow, indicating consistent Gaussian motions with flow supervision.
6 More Discussions
By aligning with the optical flow, our Gaussian flow effectively optimizes the motion of Gaussian splats. However, if the optical flow cannot be reliably estimated, our method cannot provide beneficial signal for optimization. For similar reason, this supervision is less helpful for modeling dynamic objects with constantly changing textures, which remains challenge for current 4D generation methods. Unlike commonly used motion priors as optical flow, recent. advances in dynamic scene reconstruction propose more compact and robust scene representations and regularizations, such as graph-based representations in MoSca (Lei et al. 2024), depth priors for 3D-aware tracking in SpatialTrackkr (Xiao et al. 2024), low-rank motion bases Shape of Motion WWang al. (2024a), and motion-aware point map representations MonST3R (Zhang et al. 2024), leveraging Dust3R (Wang et al. 2024bb. All of these techniques tightly couple the learnable
11

---

Published in Transactions on Machine Learning Research (06/2025)
Input Video
Ours (no flow)
Ours (flow)
t=
t=t2
flowi
flowl
flow
flow 1122
flow
flowe
#1
#2
#3
#4
#5
#6
input view
input view
input view
novel view
input view
input view
novel view
Figure 7: Visualization of optical and Gaussian flows the input view and novel view. "Ours (no flow)) denotes our model without flow supervision while "Ours" is our full model. The optical flow values of the background should be ignored because dense optical flow algorithms calculate correspondences among background pixels. We calculate optical flow flow?, on rendered sequences by autoflow (Sun et al. 2021 From column #1 and #4, we can see that both rendered sequences from input view have high-quality optical flow, indicating correct motions and appearance. Comparing Gaussian flow flow column #2 and #5 we that the underlining Gaussians move inconsistently without flow supervision. Tt is due to the ambiguity of appearance and motions while only being supervised by photometric loss single input view. Aligning Gaussian flow to optical flow can drastically improve irregular motions(column #3) and create high-quality dynamic motions (column #6) on novel views.
representations with the objective in an end-to-end fashion, making them more effective than integrating off-the-shelf optical flow into dynamic scene reconstruction, done in our work. However, our proposed GaussianFlow be seamlessly applied to any pixel-wise 2D-3D supervision, general relationship between motions observed in 2D frames and the dynamics of 3D Gaussian primitives.
formulation establishes
7 Conclusion and Future Work
We present GaussianFlow, analytical solution for supervising 3D Gaussian dynamics including scaling, rotation, and translation with 2D optical flow. Extensive qualitative and quantitative comparisons demon- strate that our method is general and beneficial to Gaussian-based representations for both 4D generation and 4D novel view synthesis with motions. this paper, we only consider the short-term flow supervision between every two neighbor frames all our experiments. Long-term flow supervision across multiple frames is expected to be better and smoother, which we leave as future work. Another promising future direction is explore view-conditioned flow SDS to supervise Gaussian flow novel view in the 4D generation task.
References
Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael ZZollhoefer, Johannes Kopf, Matthew O'Toole, and Changil Kim. Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling. In Proceedings of the IEEE/CVP Conference Computer Vision and Pallern Recognilion, pp. 16610-16620) 2023.
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation
12

---

Published in Transactions on Machine Learning Research (06/2025)
using hybrid score distillation sampling. arXiv preprint arXiv:2811.1798, 2023.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation.
arXiv preprint arXiv:2401.1280 12945, 2024.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.
Video generation models as world simulators. 2024. URL httpp://openaii https//ppppaaaiccrerrerrrlll com/research/
rideo-generationnnddeels ass----a--ooo-o-d www-wwowldddswsaeee
Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes, In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023.
Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient. geomelry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123 16133, 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. European Conference on Computer Vision, pp. 333 350. Springer, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 13142 13153, 2023.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent. Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference Robotics and Automation (ICRA), pp. 2553 2560. IEEE, 2022.
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Popers, pp. 1 9, 2022.
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæek Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479-12488, 2023.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthosis from dynamic monocu- lar video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5712 5712-5721, 2021.
Quankai Gao, Qiangeng Xu, Hao Su, Ulrich Neumann, and Zexiang Xu. Strivec: Sparse tri-vector radiance fields. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 17569-17579, 2023.
Zhiyang Guo, Wengang Zhou, Li Li, Min Wang, and Houqiang Li. Motion-aware 3d gaussian splatting for efficient dynamic scene reconstruction. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Fong Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXio preprint 2xXiv2311.044000 2023.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4220 4230, 2024.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision, pp. 624-642. Springer, 2022.
Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent-dd: Consistent 360 {'deg) dynamic object generation from monocular video. arXiv preprint arXiv:2311.096: 028844, 2023.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint 20N::2305502463, 2023.
Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprinl ar Xrv: 021226, 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leinkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering, ACM Transactions on Graphics, 42(4), 2023.
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Danillidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. Xiv preprint orXiv:2405. 17421, 2024.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 5521 5521-5531, 2022.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognilion, pp. 6498 6508, 2021.
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273 4284, 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolttinn text-to-3d content creation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 300-309, 2023.
Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian- flow: 4d reconstruction with dynamic 3d gaus- sian particle. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21136 21145, 2024.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-Ad with dynamic 3d gaussians and composed diffusion models, arXiv preprint arX.::2912.13763 2023.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint 10X7022311.07885, 2023a.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. Onc-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9298 9309, 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synodreamer: Generating multiview--onsiiteen images from single-view image. arXiv preprint arXiv:2309.0344 2023c.
Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. arXiv preprinl ar 2iv:22305. 17398, 2023d.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song- Hai Zhang, Mare Habermann, Christian Thcobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. Xáv preprinl ar Xix::22300. 15008, 2023.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.099 2023.
Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison. Gaussian splatting slam. arXiv preprinl arXiv:2312.001 06741, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium mized and augmented realily, pp. 127 136. leee, 2011.
Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamiclusion: Reconstruction and tracking of non-rigid scenes in tial-time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 343 352, 2015.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint 3rXiv:2212.0875/ 2022.
Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven Seitz, and Ricardo Marain-Brualla. Nerfics: Deformable neural radiance fields. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5865-5874 5874, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimenssonal reprosentation for topo- logically varying neural radiance fields. Xi preprint arXic:2106. 13228, 2021b.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreanfusion: Text-to-3d using 2d dilfusion. arXiv preprint TrXio:2209.14988 2022.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10318-1188 10188-1027,, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir A Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d gener- ation. arXiv preprint arXiv:2303.1880 2023.
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint NNw:2312.17143 2023.
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Soung Wook Kim, et al. L4gm: Large 4d gaussian reconstruction model. arXiv preprint 2r0:22400. 1 10324, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684 10695, 2022.
Johannes L Schonberger and Jan-Michael Frahm. Structurefromm-motion revisited. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4104-4113, 2016.
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 16632 16642, 2023.
Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. arXiv preprint WXXi:2230.0833/2 2023a.
Yichun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arxie preprint or orxi:2230 16512, 2023b.
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filipposs Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXie preprint wrXiv:2301.11280 2023.
Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers, pp. 833-846. 2006.
Liangehen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphies, 29(5):2732-2742, 2023.
Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. Autoflow: Learning better training set for optical flow. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 10093 10093-10102, 2021.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hicrarchical 3d generation with bootstrapped diffusion prior. arXie preprint arXiw:2310.16818 2023.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint 1xXi::2309.1655. 2023.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 12959-12970, 2021.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref- nerf: Structured riewddepen appearance for neural radiance fields. in 2022 iece. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5481-5400 2022.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. Xiv preprint or orXi:21005.05094 2021.
Chaoyang Wang, Lachlan Ewen MacDonald, Laszlo A Jeni, and Simon Lucey. Flow supervision for de- formable nerf. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 21128-2122 21137, 2023a.
Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In Proceedings of the IEEE/CVF International Conference Compuler Vision, pp. 19706-19000 19716, 2023b.
Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lrm: Poso-free large reconstruction model for joint pose and shape prediction. arXiv preprinl ar Xiv: 12024, 2023c.
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint wrXiv22407137666 2024a.
Shizun Wang, Xingyi Yang, Qiuhong Shen, Zhenxiang Jiang, and Xinchao Wang. Gflow: Recovering 4d world from monocular video. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7862-7870, 2025.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made casy. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 20697 20709, 2024b.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolifiddreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024c.
Guanjun WU, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xing- gang Wang. 4d gaussian splatting for real-time dynamic scene rendering, arXiv preprint Xrx:v:2300.00 20.22800, 2023.
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou Spa- tialtracker: Tracking any 2d pixels in 3d space. Tn Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pp. 1440620417,, 2024.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusson using 3d large reconstruction model. arXiv preprint arXiv:2311.009 0922177 2023.
Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 8285-8295, 2023.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 15980- 15989, 2021a.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva. Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems, 34:19326 19338, 2021b.
Gengshan Yang, Chaoyang Wang, N Dinesh Reddy, and Deva Ramanan. Reconstructing animatable cate- gories from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nilion, pp. 16995-1999 17005, 2023a.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Gengshan Yang, Shno Yang, John Z Zhang, Zachary Manchester, and Deva Ramanan. Ppr: Physically plausible reconstruction from monocular videos. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 3914 3924, 2023b.
Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.1000 2023c.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. ar preprint arXiv:2309.181 2023d.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 20331-2001 20311, 2024.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chemming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 9150 9161, 2023.
Vladimir Yugay, Yue Li, Theo Gevers, and Martin R Oswald. Gaussian-slam: Photo-realistic dense slam with gaussian splatting, arXiv preprint arXiv:2312.1007) 2023.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. Xiv preprint Xr 03825, 2024.
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate124: Animating one image 4d dynamic scene. arXiv preprint arX4v:2311.14603, 2023.
Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, et al. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (ToG), 33(4):1-12, 2014.
18

---

Published in Transactions on Machine Learning Research (06/2025)
A Appendix
A.1 Implementation Details
We take l2 as the next. time step of I and calculate the optical flow between every two neighbor frames in all experiments. In our CUDA implementation of Gaussian dynamics splatting, although the number of Gaussians K along cach pixel ray is usually different, we use K 20 to balance speed and effectiveness. larger K meams more number of Gaussians and their gradient will be counted through backpropagation. For video frames with size H x W x 3, we track the motions of Gaussians between every two neighbor timesteps t and l2 by maintaining two x W K tensors to record the indices of the top-K Gaussians sorted in depth order, top-K Gaussians' rendered weights wy for each pixel and another tensor with size H W K x 2 denotes the distances between the pixel coordinate and 2D Gaussian means Xx₁ Ma respectively. In addition, the 2D mean Mi,t and 2D covariance matrices Σ and Li of each Gaussian at different two timesteps are accessible via camera projection (Kerbl et al. 2023
Algorithm 1: Detailed pseudo code for GaussianFlow
Input:
flowf Pseudo ground-truth optical flow from off-the-shelf optical flow algorithm; : ground-truth images where k = 1. renderer: A Gaussian renderer;
Gaussians Gaussianst, 1 : Gaussians with learnable paramcters at t, and fR413 camy, and cam Camera parameters at t, and tx41;
# Loss init
L 0
for timestep k T do
// renderer outputs at is renderer,, = (endderrr(aaussionns. Ire Irender renderer, ["mmage"]:
cam)y ); # H x W 3
idæ;, renderer b2
['ndex""]:
H W K, Gaussian indices that cover each pixels
Wt, renderert. [ (wwiights"
# H W x K
Wta wx,/ ssmm Webt dim = -1); # H x W x K, weight normalization
I h_t renderer 1x ["T_m""]] "] #
H x W x K x 2, denotes Itz Pte
Hr, rendereriy
12D mean"): "I; # n x 2
Σ2 = renderert.
"2 " # n x 2 x
By = E
# renderer outputs t&+1
renderer Sanderer(Gaussianss 0am44,1) 1)
ph--1 renderer 2* mean")m)) "]; n x
Eta rendererta 22 cov"l: # x
B Ex
# Eq.8 while ignoring resize operations for simplicity
flow in = Wt, (Biztn [idzi,) * inw [idr.,, x Me + [idare. - Mea [idxt,] 1 - x ple))
# Eq.10
Eilow = -nom(fflow, f sum(flow 14411 dim = -0)
# (1) Loss for 4D novel view synthosis
C -+ Cphotemetric (oinnntttie(rreaaae [Irender, IIa.) + X + XyLother
# (2) Loss for 4D generation
C = L + Cyhotometric (Irennd
[Irender, ,1 + & + XzCads + NaCother
end
19

---

Published in Transactions on Machine Learning Research (06/2025)
A detailed pscudo code for our flow supervision be found in Algorithm 1. We extract the projected Gaussian dynamics and obtain the final Gaussian flow by rendering these dynamics. Variables including the weights and top-K indices of Gaussians per pixel (as mentioned in implementation details of our main paper) are calculated in CUDA by modifying the original CUDA kernel codes of 3D Ganssian Splatting (Kerbl et al. 2023). And Gaussian flow flow G is calculated by Eq.8 with PyTorch.
In our 4D generation experiment, we run 500 iterations of static optimization to initialize 3D Gaussian fields with batch size of 16. The Tmax in SDS is linearly decayed from 0.98 to 0.02. For dynamic representation, run 600 iterations with batch size of for both DG4D RRen et al. 2023 and ours. The flow loss weight A1 in Eq. 11 of our main paper is 1.0.
Our method slightly decreases speed and increases memory only in training stage but not in the inference stage because our flow supervision is only for training a better/robust deformation field or other 4DGS designs and then will not be needed in inference stage. The training speed for DG4D around 1.4it/s while it then becomes around 2.2it/8 with our flow supervision. And the difference between training speeds with (around 2.5s/it) and without (around 2.2s/it) our flow supervision for RT-4DGS is marginal. Even with more memory footprint by tracking per-pixel gradients for Gaussians, single 30GB GPU is adequate for reproducing all our results. In our 4D novel view synthesis experiment, we follow RT-4DGS((aan) et al. 2023c) except that we add our proposed flow supervision for all cameras. The flow loss weight. d in Eq. 11 of our main paper is 0.5.
A.2 More Visualization and Comparison in 4D Generation.
More comparisons between Gaussian flow flow and optical flow flow rendered images are shown in Fig. 9
The first row of each example is the rgb frames rendered from optimized 4D Gaussian field. We rolate our cameras for each time step that the object can move as optimized and the camera is moving at the same time to show the scene from different angles. The second row of each example shows the visualized Gaussian flows. These Gaussian flows are calculated by the rondered images of consecutive time steps at cach camera view, therefore, containing no camera motion in the flow values. The third row shows the estimated optical flows between the rendered images of consecutive time steps in each camera view. We use off-the-shelf AutoFlow (Sun al.) 2021) for the estimation. We can see that, enhanced by the flow supervision from the single-input view, our 4D generation pipeline can model fast motion such the explosive motion of the gun hammer (see the last example in Fig. g.
Figure 8: Qualitative results on Consistent4D datasct.
20

---

Published in Transactions on Machine Learning Research (06/2025)
rendered
flow"
flowo
rendered
flow
flowo
rendered
flow
flow
rendered
flow
flow
Figure 9: Visualization of Gaussian flow flowG and optical flow flow rendered sequences from different views.
A.3 More Qualitative Results on the DyNeRF Dataset.
More qualitative results on DyNeRF dataset Li et al. 2022 can be found in Fig. 11 our vidco.
Fig.
12
Fig.
13
and
A.4 More Quantitative Ablations the DyNeRF Dataset.
More quantitative ablations on the DyNeRF dataset Li et al. (2022) can be found in Tab. 6 If X is too small (e.g., 109), the performance closely matches that of the baseline without flow supervision. Conversely, if A₁. is too large (e.g., 10*), leads to a performance drop in quantitative metrics such as PSNR. However, our flow supervision remains robust and effective when A₁ set within moderate rango. Another tunable parameter is the number of Gaussian tracked per-pixe per-pixel, we observed that increasing its value does not lead to significant, performance gains, but it does slow down training due to the need to track larger number
21

---

Published in Transactions on Machine Learning Research (06/2025)
of Gaussians. Therefore, mentioned in our Implementation Details, use K 20 to balance speed and effectiveness.
Table 6: Quantitative ablations on DyNeRF dataset Li et al. 2022.. We report PSNR of our method with different number of Gaussians K tracked per-pixel and the flow loss weight Å1.
PSNR
Ours (K=0, =
28.00
Ours (K=10, 00.5)
28.54
Ours (K=40, -0.5)
29.12
Ours (K=20, =1
28.23
Ours (K=20, ===1e²)
27.35
Ours (default:K-20) A 00.5)
28.99
A.5 Flow Visualization in Dynamic Gaussian Fields
Note that dynamic 3D Gaussian (Luiten et al. 2023 provided way to visualize 3D scene motions between consecutive frames, however, by tracking one "most influential" 3D Gaussian per pixel. This is neither efficient nor effective to be used in flow supervision, because the "mast influential' 3D Gaussian for each pixel is determined by searching the nearest 3D Gaussian's center from tens of thousands of 3D Gaussian candidates with a virtual 3D point along pixel ray lifted with corresponding rendered depth. Also, the "most influential' Gaussian of pixel might not even cover the same pixel but still be considered just because this Gaussian's center is the nearest one the virtual point among all 3D Gaussians. We have also applied flow supervision in this way, but find it has no observable benefit for rendering quality while resulting in slower training speed due to the per-pixel nearest search. On the other hand, RT-4DGS showed "render flow" in their paper only for visualization purpose and the detail were not clarified and the function was not enabled, please refer to their code
issue I and issue
A.6 Quantitative Results the NeRF-DS Dataset.
When considering the cases with both camera motions and object motions, we have the relationship warp(flow") = flow where warp is the view warping of optical flow cased by camera motion and flow is still the foreground object Gaussian dynamics. Therefore, we supervise the Gaussian flow with the warped optical flow by following camera and scene motion warping as (Li et al., 2021). We present qualitative comparisons Deformable-3DGS (Yang et al. 2023d) with and without our flow supervision Fig. 10
22

---

Published in Transactions on Machine Learning Research (06/2025)
Deformable 3DGS
Ours
Ground Truth
Basin (view 1)
Basin (view2 2)
Press (view 1)
Press (view 2)
Sieve (view 1)
Sieve (view 2)
Press (view 1)
Press (view 2)
Figure 10: Qualitative comparisons NeRF-DS dataset Yan al. 2023
23

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 11: Qualitative comparisons on DyNeRE dataset Li et al. 2022). The left column shows the novel view rendered images and depth maps of 4D Gaussian method Yang al. (2023c), which suffers from artifacts in the dynamic regions. The right column shows the results of the same method while optimized with our flow supervision during training.
24

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 12: Qualitative comparisons DyNeRF dataset Li et al. (2022). The left column shows the novel
view rendered images and depth maps of 4D Gaussian method Yang et al. 2023c). While The right column shows the results of the same method while optimized with our flow supervision during training.
25

---

Published in Transactions on Machine Learning Research (06/2025)
Figure 13: Qualitative comparisons on DyNeRF dataset (Li al. 2022 Since the details of depth maps on Flame Salmon are hard to be recognized, only compare the rendered images. The left column shows the novel view rendered images of a 4D Gaussian method YYngg et al. 2023c). While The right column shows the results of the same method while optimized with our flow supervision during training.
26


Reviews:
[{"review_id": "BOPkfRPdxe", "reviewer": "Reviewer_tqNG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Such a plug-in flow supervision module is effective and can be easily integrated into different works.\n2. The authors demonstrate the effectiveness of their proposed method on 4D generation and 4D NVS tasks.\n3. Extensive experiments show their superiority.\n\n**Questions**\n1. Can the proposed method handle the scenes with specular surface objects?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced a novel method considering the flow of the Gaussian ellipsoids. The whole flow algorithm can mainly be divided into two parts: the flows of a single Gaussian ellipsoid (through normalization and unnormalization) and multiple Gaussian ellipsoids (through flow composition by alpha-blending). The idea is simple but effective via the shown experiments. Also, they test their proposed method on two tasks: 4D generation and 4D NVS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I understand the key contribution of this paper is not simply flow supervision, but I highly suggest that the author should compare it with some more flow supervision methods to further demonstrate the effectiveness of their method. For example, 'GFlow: Recovering 4D Worlds from Monocular Video' and 'Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction'.", "broader_impact_concerns": "Nothing.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JUcZlBdTUN", "reviewer": "Reviewer_bpt7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The introduction of GaussianFlow, bridging Gaussian splatting dynamics directly with optical flow supervision, represents a meaningful technical advancement. The approach addresses the challenge of directly supervising 3D Gaussian motions from 2D pixel data.\n+ Experimental results show improvements, especially in highly dynamic scenarios where previous methods typically suffer from motion ambiguities and visual artifacts. This indicates robustness and practical applicability.\n+ The method leverages efficient CUDA-based implementations, ensuring minimal computational overhead despite its sophisticated supervision mechanism. This is critical for practical deployment and scalability in real-time applications.\n\nWeaknesses:\n- The method seems to heavily rely on the accuracy of optical flow estimation. Inaccuracies or failures in optical flow estimation could directly degrade GaussianFlow’s performance, especially in scenes with repetitive textures or fast motion.\n- The current approach primarily supervises short-term optical flows between consecutive frames. This may limit effectiveness in long-duration sequences, potentially causing cumulative errors or drift over extended temporal windows.\n- While the evaluation includes high-quality synthetic datasets, there is limited demonstration of the method's robustness in uncontrolled, real-world scenarios, which may exhibit more severe challenges such as lighting changes, occlusions, and noise.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces GaussianFlow, an innovative approach to connect the motion dynamics of 3D Gaussian representations to 2D pixel velocities, enabling effective supervision from optical flow for Gaussian-based dynamic scene reconstruction. The approach involves splatting 3D Gaussian motions onto the image plane, thus establishing a differentiable relationship between Gaussian dynamics and pixel optical flow. Experiments on various datasets demonstrate that GaussianFlow significantly enhances the quality of 4D novel view synthesis and 4D content generation, particularly in dynamic and challenging motion scenarios. The method also addresses the color drifting artifacts common in existing techniques, providing state-of-the-art results compared to contemporary 4D reconstruction methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could the authors clarify how GaussianFlow’s performance degrades as the accuracy of the optical flow deteriorates? Are there mechanisms proposed or envisioned to mitigate such dependencies?\n- Are there plans or potential directions for extending GaussianFlow to handle long-term temporal consistency across multiple frames rather than just consecutive frames?\n- How significantly does the choice of initialization methods (e.g., DreamGaussian or One-2-3-45) influence the GaussianFlow optimization process and the final visual results? Could this approach integrate more adaptive or data-driven initialization techniques?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b3C6uPJCCa", "reviewer": "Reviewer_gzjY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper proposed a generalizable and differentiable flow calculation method for 4D Gaussian splatting, which is valuable to the community.\n- The authors presented a good evaluation on three datasets and two tasks, showcasing that their method can be applied to different 4D tasks, and could also deal with camera motion and object motion. The evaluation results also look great.\n- Good analysis of overhead caused by the algorithm, which shows that the proposed method has minimal cost but provides relatively high improvement.\n\nWeaknesses:\n- The presentation of the paper might be further improved. The left and right part of Figure 2 is not described properly in the main text or caption. Lack of math symbols on the left of Figure 2.\n- Is this method also works for monocular 4D reconstructions? What might be the challenge of doing that?\n- I am curious about how large motion this method can handle since optical flow might be inaccurate for long-term dynamics.\n- Lack of quantitive results on ablation study.\n- Reference of most recent papers: GFlow, 4D-Rotor Gaussian splatting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to calculate the flow of 4D Gaussian representations. Specifically, the flow of each pixel between two frames is calculated by weighting all the Gaussian movements on that pixel through alpha blending. The flow calculation acts as a plug-in module of flow supervision for 4D generation and reconstruction methods using Gaussian splatting. The authors demonstrate the effectiveness of their method by evaluating on different tasks, including 4D content generation and novel view synthesis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the mentioned weaknesses in last section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)