GAUSSIANFLOW: SPLATTING GAUSSIAN DYNAMICS FOR 4D CONTENT CREATION

## Abstract

Abstract Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to be handled by existing methods. The common color drifting issue that happens in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality on extensive experiments demonstrates our method’s effectiveness. Quantitative and qualitative evaluations show that our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis. Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/

###### Keywords:

## 1 Introduction

4D dynamic content creation from monocular or multi-view videos has garnered significant attention from academia and industry due to its wide applicability in virtual reality/augmented reality, digital games, and movie industry. Studies  [, , , ] model 4D scenes by 4D dynamic Neural Radiance Fields (NeRFs) and optimize them based on input multi-view or monocular videos. Once optimized, the 4D field can be viewed from novel camera poses at preferred time steps through volumetric rendering. A more challenging task is generating 360 degree 4D content based on uncalibrated monocular videos or synthetic videos generated by text-to-video or image-to-video models. Since the monocular input cannot provide enough multi-view cues and unobserved regions are not supervised due to occlusions, studies [, , ] optimizes 4D dynamic NeRFs by leveraging generative models to create plausible and temporally consistent 3D structures and appearance. The optimization of 4D NeRFs requires volumetric rendering which makes the process time-consuming. And real-time rendering of optimized 4D NeRFs is also hardly achieved without special designs. A more efficient alternative is to model 4D Radiance Fields by 4D Gaussian Splatting (GS) [, ], which extends 3D Gaussian Splatting [] with a temporal dimension. Leveraging the efficient rendering of 3D GS, the lengthy training time of a 4D Radiance Field can be drastically reduced [, ] and rendering can achieve real-time speed during inference.

The optimization of 4D Gaussian fields takes photometric loss as major supervision. As a result, the scene dynamics are usually under-constraint. Similarly to 4D NeRFs [, , ], the radiance properties and the time-varying spatial properties (location, scales, and orientations) of Gaussians are both optimized to reduce the photometric Mean Squared Error (MSE) between the rendered frames and the input video frames. The ambiguities of appearance, geometry, and dynamics have been introduced in the process and become prominent with sparse-view or monocular video input. Per-frame Score Distillation Sampling (SDS) [] reduces the appearance-geometry ambiguity to some extent by involving multi-view supervision in latent domain. However, both monocular photometric supervision and SDS supervision do not directly supervise scene dynamics.

To avoid temporal inconsistency brought by fast motions, Consistent4D [] leverages a video interpolation block, which imposes a photometric consistency between the interpolated frame and generated frame, at a cost of involving more frames as pseudo ground truth for fitting. Similarly, AYG [] uses text-to-video diffusion model to balance motion magnitude and temporal consistency with a pre-set frame rate.
4D NeRF model [] has proven that optical flows on reference videos are strong motion cues and can significantly benefit scene dynamics. However, for 4D GS, connecting 4D Gaussian motions with optical flows has following two challenges. First, a Gaussian’s motion is in 3D space, but it is its 2D splat that contributes to rendered pixels. Second, multiple 3D Gaussians might contribute to the same pixel in rendering, and each pixel’s flow does not equal to any one Gaussian’s motion.

To deal with these challenges, we introduce a novel concept, Gaussian flow, bridging the dynamics of 3D Gaussians and pixel velocities between consecutive frames. Specifically, we assume the optical flow of each pixel in image space is influenced by the Gaussians that cover it. The Gaussian flow of each pixel is considered to be the weighted sum of these Gaussian motions in 2D. To obtain the Gaussian flow value on each pixel without losing the speed advantage of Gaussian Splatting, we splat 3D Gaussian dynamics, including scaling, rotation, and translation in 3D space, onto the image plane along with its radiance properties. As the whole process is end-to-end differentiable, the 3D Gaussian dynamics can be directly supervised by matching Gaussian flow with optical flow on input video frames. We apply such flow supervision to both 4D content generation and 4D novel view synthesis to showcase the benefit of our proposed method, especially for contents with rich motions that are hard to be handled by existing methods. The flow-guided Guassian dynamics also resolve the color drifting artifacts that are commonly observed in 4D Generation. We summarize our contributions as follows:

- •
We introduce a novel concept, Gaussian flow, that first time bridges the 3D Gaussian dynamics to resulting pixel velocities. Matching Gaussian flows with optical flows, 3D Gaussian dynamics can be directly supervised.
- •
The Gaussian flow can be obtained by splatting Gaussian dynamics into the image space. Following the tile-based design by original 3D Gaussian Splatting, we implement the dynamics splatting in CUDA with minimal overhead. The operation to generate dense Gaussian flow from 3D Gaussian dynamics is highly efficient and end-to-end differentiable.
- •
With Gaussian flow to optical flow matching, our model drastically improves over existing methods, especially on scene sequences of fast motions. Color drifting is also resolved with our improved Gaussian dynamics.

## 2 Related Works

#### 2.0.1 3D Generation.

3D generation has drawn tremendous attention with the progress of various 2D or 3D-aware diffusion models [, , , ] and large vision models [, , ]. Thanks to the availability of large-scale multi-view image datasets [, , ], object-level multi-view cues can be encoded in generative models and are used for generation purpose. Pioneered by DreamFusion [] that firstly proposes Score Distillation Sampling (SDS) loss to lift realistic contents from 2D to 3D via NeRFs, 3D content creation from text or image input has flourished. This progress includes approaches based on online optimization [, , , ] and feedforward methods [, , , , ] with different representations such as NeRFs [], triplane [, , ] and 3D Gaussian Splatting []. 3D generation becomes more multi-view consistent by involving multi-view constraints [] and 3D-aware diffusion models [] as SDS supervision. Not limited to high quality rendering, some works [, ] also explore enhancing the quality of generated 3D geometry by incorporating normal cues.

#### 2.0.2 4D Novel View Synthesis and Reconstruction.

By adding timestamp as an additional variable, recent 4D methods with different dynamic representations such as dynamic NeRF [, , , , , , ], dynamic triplane [, , ] and 4D Gaussian Splatting [, ] are proposed to achieve high quality 4D motions and scene contents reconstruction from either calibrated multi-view or uncalibrated RGB monocular video inputs. There are also some works [, , ] reconstruct rigid and non-rigid scene contents with RGB-D sensors, which help to resolve 3D ambiguities by involving depth cues.
Different from static 3D reconstruction and novel view synthesis, 4D novel view synthesis consisting of both rigid and non-rigid deformations is notoriously challenging and ill-posed with only RGB monocular inputs. Some progress [, , , ] involve temporal priors and motion cues (e.g. optical flow) to better regularize temporal photometric consistency and 4D motions. One of recent works [] provides an analytical solution for flow supervision on deformable NeRF without inverting the backward deformation function from world coordinate to canonical coordinate. Several works [, , , ] explore object-level mesh recovery from monocular videos with optical flow.

#### 2.0.3 4D Generation.

Similar to 3D generation from text prompts or single images, 4D generation from text prompts or monocular videos also relies on frame-by-frame multi-view cues from pre-trained diffusion models. Besides, 4D generation methods yet always rely on
either video diffusion models or video interpolation block to ensure the temporal consistency. Animate124 [], 4D-fy [] and one of the earliest works [] use dynamic NeRFs as 4D representations and achieve temporal consistency with text-to-video diffusion models, which can generate videos with controlled frame rates. Instead of using dynamic NeRF, Align Your Gaussians [] and DreamGaussian4D [] generate vivid 4D contents with 3D Gaussian Splatting, but again, relying on text-to-video diffusion model for free frame rate control. Without the use of text-to-video diffusion models, Consistent4D [] achieves coherent 4D generation with an off-the-shelf video interpolation model []. Our method benefits 4D Gaussian representations by involving flow supervision and without the need of specialized temporal consistency networks.

## 3 Methodology

Figure: Figure 2: Between two consecutive frames, a pixel $x_{t_{1}}$ will be pushed towards $x_{t_{1}}\rightarrow x_{i,t_{2}}$ by the 2D Gaussian $i$’s motion $i^{t_{1}}\rightarrow i^{t_{2}}$. We can track $x_{t_{1}}$ in Gaussian $i$ by normalizing it to canonical Gaussian space as $\hat{x}_{i}$ and unnormalize it to image space to obtain $x_{i,t_{2}}$. Here, we denote this shift contribution from Gaussian $i$ as $flow^{G}_{i,t_{1},t_{2}}$. The Gaussian flow $flow^{G}_{t_{1},t_{2}}(x_{t_{1}})$ on pixel $x_{t_{1}}$ is defined as the weighted sum of the shift contributions from all Gaussians covering the pixel ($i$ and $j$ in our example). The weighting factor utilizes alpha composition weights. The Gaussian flow of the entire image can be obtained efficiently by splatting 3D Gaussian dynamics and rendering with alpha composition, which can be implemented similarly to the pipeline of the original 3D Gaussian Splatting [].
Refer to caption: x2.png

To better illustrate the relationship between Gaussian motions and corresponding pixel flow in 2D images, we first recap the rendering process of 3D Gaussian Splatting and then investigate its 4D case.

### 3.1 Preliminary

#### 3.1.1 3D Gaussian Splatting.

From a set of initialized 3D Gaussian primitives, 3D Gaussian Splatting aims to recover the 3D scene by minimizing photometric loss between input $m$ images $\{I\}_{m}$ and rendered images $\{I_{r}\}_{m}$. For each pixel, its rendered color $C$ is the weighted sum of multiple Gaussians’ colors $c_{i}$ in depth order along the ray by point-based $\alpha$-blending as in Eq. 1,

$$ $C=\sum^{N}_{i=1}T_{i}\alpha_{i}c_{i},$ (1) $$

with weights specifying as

$$ $\alpha_{i}=o_{i}e^{-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_{i})^{T}\mathbf{ \Sigma}_{i}^{-1}(\mathbf{x}-\boldsymbol{\mu}_{i})}\quad\text{and}\quad T_{i}= \sum^{i-1}_{j=1}(1-\alpha_{i}).$ (2) $$

where $o_{i}\in[0,1]$, $\boldsymbol{\mu}_{i}\in\mathbb{R}^{2\times 1}$, and $\mathbf{\Sigma}_{i}\in\mathbb{R}^{2\times 2}$ are the opacity, 2D mean, and 2D covariance matrix of $i$-th Gaussian, respectively. And $\mathbf{x}$ is the intersection between a pixel ray and $i$-th Gaussian. As shown in Eq. 1, the relationship between a rendered pixel and 3D Gaussians is not bijective.

#### 3.1.2 3D Gaussian Splatting in 4D.

Modeling 4D motions with 3D Gaussian Splatting can be done frame-by-frame via either directly multi-view fitting [] or moving 3D Gaussians with a time-variant deformation field [, ] or parameterize 3D Gaussians with time []. While with monocular inputs, Gaussian motions are under-constrained because different Gaussian motions can lead to the same rendered color, and thus long-term persistent tracks are lost []. Though Local Rigidity Loss [, ] is proposed to reduce global freedom of Gaussian motions, it sometimes brings severe problems due to poor or challenging initialization and lack of multi-view supervision. As shown in Fig. 6, 3D Gaussians initialized with the skull mouth closed are hard to be split when the mouth open with Local Rigidity Loss.

### 3.2 GaussianFlow

We consider the full freedom of each Gaussian motion in a 4D field, including 1) scaling, 2) rotation, and 3) translation at each time step. As the time changes, Gaussians covering the queried pixel at $t=t_{1}$ will move to other places at $t=t_{2}$, as shown in Fig. 2. To specify new pixel location $\mathbf{x}_{t_{2}}$ at $t=t_{2}$, we first project all the 3D Gaussians into 2D image plane as 2D Gaussians and calculate their motion’s influence on pixel shifts.

#### 3.2.1 Flow from Single Gaussian.

To track pixel shifts (flow) contributed by Gaussian motions, we let the relative position of a pixel in a deforming 2D Gaussian stay the same.
This setting makes the probabilities at queried pixel location in Gaussian coordinate system unchanged at two consecutive time steps. According to Eq. 2, the unchanged probability will grant the pixel with the same radiance and opacity contribution from the 2D Gaussian, albeit the 2D Gaussian is deformed.

The pixel shift (flow) is the image space distance of the same pixel at two time steps.
We first calculate the pixel shift influenced by a single 2D Gaussian that covers the pixel. We can find a pixel $\mathbf{x}$’s location at $t_{2}$ by normalizing its image location at $t_{1}$ to canonical Gaussian space and unnormalizing it to image space at $t_{2}$:

1) $normalize$. A pixel $\mathbf{x}_{t_{1}}$ following $i$-th 2D Gaussian distribution can be written as $\mathbf{x}_{t_{1}}\sim N(\boldsymbol{\mu}_{i,t_{1}}\mathbf{\Sigma}_{i,t_{1}})$. And in $i$-th Gaussian coordinate system with 2D mean $\boldsymbol{\mu}_{i,t_{1}}\in\mathbb{R}^{2\times 1}$ and 2D covariance matrix $\mathbf{\Sigma}_{i,t_{1}}\in\mathbb{R}^{2\times 2}$. After normalizing the $i$-th Gaussian into the standard normal distribution, we denote the pixel location in canonical Gaussian space as

$$ $\hat{\mathbf{x}}_{t_{1}}=\mathbf{B}^{-1}_{i,t_{1}}(\mathbf{x}_{t_{1}}- \boldsymbol{\mu}_{i,t_{1}}),$ (3) $$

which follows $\mathbf{\Sigma}_{i,t_{1}}=\mathbf{B}_{i,t_{1}}\mathbf{B}_{i,t_{1}}^{T}$, $\hat{\mathbf{x}}_{t_{1}}\sim N(\mathbf{0},\mathbf{I})$ and $\mathbf{I}\in\mathbb{R}^{2\times 2}$ is identity matrix.

2) $unnormalize$. When $t=t_{2}$, the new location along with the Gaussian motion denotes $\mathbf{x}_{i,t_{2}}$ on the image plane.

$$ $\displaystyle\mathbf{x}_{i,t_{2}}$ $\displaystyle=\mathbf{B}_{i,t_{2}}\hat{\mathbf{x}}_{t_{1}}+\boldsymbol{\mu}_{i ,t_{2}},$ (4) $$

and $\mathbf{\Sigma}_{i,t_{2}}=\mathbf{B}_{i,t_{2}}\mathbf{B}_{i,t_{2}}^{T}$, $\mathbf{x}_{t_{2}}\sim N(\boldsymbol{\mu}_{i,t_{2}},\mathbf{\Sigma}_{i,t_{2}})$. Eq. 3 and Eq. 4 preserve Mahalanobis distance between the tracked pixel and the 2D Gaussian leading to consistent probability density across consecutive time steps.
The pixel shift (flow) contribution from each Gaussian therefore can be calculated as:

$$ $\displaystyle flow^{G}_{i,t_{1}t_{2}}=\mathbf{x}_{i,t_{2}}-\mathbf{x}_{t_{1}}$ (5) $$

#### 3.2.2 Flow Composition.

In original 3D Gaussian Splatting, a pixel’s color is the weighted sum of the 2D Gaussians’ radiance contribution. Similarly, we define the Gaussian flow value at a pixel as the weighted sum of the 2D Gaussians’ contributions to its pixel shift, following alpha composition. With Eq. 3 and Eq. 4, the Gaussian flow value at pixel $\mathbf{x}_{t_{1}}$ from $t=t_{t_{1}}$ to $t=t_{t_{2}}$ is

$$ $\displaystyle flow^{G}_{t_{1}t_{2}}$ $\displaystyle=\sum^{K}_{i=1}w_{i}flow^{G}_{i,t_{1}t_{2}}$ (6) $\displaystyle=\sum^{K}_{i=1}w_{i}(\mathbf{x}_{i,t_{2}}-\mathbf{x}_{t_{1}})$ (7) $\displaystyle=\sum^{K}_{i=1}w_{i}\left[\mathbf{B}_{i,t_{2}}\mathbf{B}^{-1}_{i, t_{1}}(\mathbf{x}_{t_{1}}-\boldsymbol{\mu}_{i,t_{1}})+\boldsymbol{\mu}_{i,t_{2 }}-\mathbf{x}_{t_{1}})\right],$ (8) $$

where $K$ is the number of Gaussians along each camera ray sorted in depth order and each Gaussian has weight $w_{i}=\frac{T_{i}\alpha_{i}}{\Sigma_{i}T_{i}\alpha_{i}}$ according to Eq. 1, but normalized to [0,1] along each pixel ray.

In some cases [, , , ], each Gaussian is assumed to be isotropic, and its scaling matrix $\mathbf{S}=\sigma\mathbf{I}$, where $\sigma$ is the scaling factor. And its 3D covariance matrix $\mathbf{RS}\mathbf{S}^{T}\mathbf{R}^{T}=\sigma^{2}\mathbf{I}$. If the scaling factor of each Gaussian doesn’t change too much across time, $\mathbf{B}_{i,t_{2}}\mathbf{B}^{-1}_{i,t_{1}}\approx\mathbf{I}$. Therefore, to pair with this line of work, the formulation of our Gaussian flow as in Eq. 8 can be simplified as

$$ $\displaystyle flow^{G}_{t_{1}t_{2}}$ $\displaystyle=\sum^{K}_{i=1}w_{i}(\boldsymbol{\mu}_{i,t_{2}}-\boldsymbol{\mu}_ {i,t_{1}}).$ (9) $$

In other words, for isotropic Gaussian fields, Gaussian flow between two different time steps can be approximated as the weighted sum of individual translation of 2D Gaussian.

Following either Eq. 8 or Eq. 9, the Gaussian flow can be densely calculated at each pixel. The flow supervision at pixel $\mathbf{x}_{t_{1}}$ from $t=t_{1}$ to $t=t_{2}$ can then be specified as

$$ $\displaystyle\mathcal{L}_{flow}=||flow^{o}_{t_{1}t_{2}}(\mathbf{x}_{t_{1}})- flow^{G}_{t_{1}t_{2}}||,$ (10) $$

where optical flow $flow^{o}_{t_{1}t_{2}}$ can be calculated by off-the-shelf methods as pseudo ground-truth.

Figure: Figure 3: Overview of our 4D content generation pipeline. Our model can take an uncalibrated monocular video or video generated from an image as the input. We optimize a 3D Gaussian field by matching the first frame photometrically on reference view and using a 3D-aware SDS loss [] to supervise the field on novel views. Then, we optimize the dynamics of the 3D Gaussians with the same two losses for each frame. Most importantly, we calculate Gaussian flows on reference view for each consecutive two time step and match it with pre-computed optical flow of the input video. The gradients from the flow matching will propagate back through dynamics splatting and rendering process, resulting in a 4D Gaussian field with natural and smooth motions.
Refer to caption: x3.png

### 3.3 4D Content Generation

As shown in Fig. 3, 4D content generation with Gaussian representation takes an uncalibrated monocular video either by real capturing or generating from text-to-video or image-to-video models as input and output a 4D Gaussian field.
3D Gaussians are initialized from the first video frame with photometric supervision between rendered image and input image and a 3D-aware diffusion model [] for multi-view SDS supervision. In our method, 3D Gaussian initialization can be done by One-2-3-45 [] or DreamGaussian []. After initialization, 4D Gaussian field is optimized with per-frame photometric supervision, per-frame SDS supervision, and our flow supervision as in Eq. 10. The loss function for 4D Gaussian field optimization can be written as:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{photometric}+\lambda_{1}\mathcal{L}_{ flow}+\lambda_{2}\mathcal{L}_{sds}+\lambda_{3}\mathcal{L}_{other},$ (11) $$

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyperparameters. $\mathcal{L}_{other}$ is optional and method-dependent. Though not used in our method, we leave it for completeness.

### 3.4 4D novel view Synthesis

Unlike 4D content generation that has multi-view object-level prior from 3D-aware diffusion model, 4D novel view synthesis takes only multi-view or monocular input video frames for photometric supervision without any scene-level prior. 3D Gaussians are usually initialized by sfm [, ] from input videos. After initialization, 4D Gaussian field is then optimized with per-frame photometric supervision and our flow supervision. We adopt the 4D Gaussian Fields from []. The loss function for 4D Gaussian field optimization can be written as:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{photometric}+\lambda_{1}\mathcal{L}_{ flow}+\lambda_{3}\mathcal{L}_{other},$ (12) $$

## 4 Experiments

In this section, we first provide implementation details of the proposed method and then valid our method on 4D Gaussian representations with (1) 4D generation and (2) 4D novel view synthesis. We test on the Consistent4D Dataset [] and the Plenoptic Video Datasets [] for both quantitative and qualitative evaluation. Our method achieves state-of-the-art results on both tasks.

### 4.1 Implementation Details

We take $t_{2}$ as the next timestep of $t_{1}$ and calculate optical flow between every two neighbor frames in all experiments. In our CUDA implementation of Gaussian dynamics splatting, though the number of Gaussians $K$ along each pixel ray is usually different, we use $K=20$ to balance speed and effectiveness. A larger $K$ means more number of Gaussians and their gradient will be counted through backpropagation. For video frames with size $H\times W\times 3$, we track the motions of Gaussians between every two neighbor timesteps $t_{1}$ and $t_{2}$ by maintaining two $H\times W\times K$ tensors to record the indices of top-$K$ Gaussians sorted in depth order, top-$K$ Gaussians’ rendered weights $w_{i}$ for each pixel and an another tensor with size $H\times W\times K\times 2$ denotes the distances between pixel coordinate and 2D Gaussian means $\mathbf{x}_{t_{1}}-\boldsymbol{\mu}_{i,t_{1}}$, respectively. Besides, 2D mean $\boldsymbol{\mu}_{i,t_{1}}$ and 2D covariance matrices $\mathbf{\Sigma}_{i,t_{1}}$ and $\mathbf{\Sigma}_{i,t_{2}}$ of each Gaussian at different two timesteps are accessible via camera projection [].

**Table 1: Quantitative comparisons between ours and others on Consistent4D dataset.**
| Method | Pistol | Guppie | Crocodile | Monster | Skull | Trump | Aurorus | Mean |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ |  |
| D-NeRF [] | 0.52 | 0.66 | 0.32 | 0.76 | 0.54 | 0.61 | 0.52 | 0.79 | 0.53 | 0.72 | 0.55 | 0.60 | 0.56 | 0.66 | 0.51 | 0.68 |
| K-planes [] | 0.40 | 0.74 | 0.29 | 0.75 | 0.19 | 0.75 | 0.47 | 0.73 | 0.41 | 0.72 | 0.51 | 0.66 | 0.37 | 0.67 | 0.38 | 0.72 |
| Consistent4D [] | 0.10 | 0.90 | 0.12 | 0.90 | 0.12 | 0.82 | 0.18 | 0.90 | 0.17 | 0.88 | 0.23 | 0.85 | 0.17 | 0.85 | 0.16 | 0.87 |
| DG4D [] | 0.12 | 0.92 | 0.12 | 0.91 | 0.12 | 0.88 | 0.19 | 0.90 | 0.18 | 0.90 | 0.22 | 0.83 | 0.17 | 0.86 | 0.16 | 0.87 |
| Ours | 0.10 | 0.94 | 0.10 | 0.93 | 0.10 | 0.90 | 0.17 | 0.92 | 0.17 | 0.92 | 0.20 | 0.85 | 0.15 | 0.89 | 0.14 | 0.91 |

Figure: Figure 4: Qualitative results on Consistent4D dataset.
Refer to caption: x4.png

Figure: Figure 5: Qualitative comparisons between Consistent4D [] (Con4D) and ours. As a dynamic NeRF-based method, Consistent4D shows “bubble like” texture and non-consistent geometry on novel views.
Refer to caption: x5.png

Figure: Figure 6: Qualitative comparisons among DreamGaussian4D [], our method without flow loss, our method without flow loss but with Local Rigidity Loss (Ours-r) and ours.
Refer to caption: x6.png

### 4.2 Dataset

#### 4.2.1 Consistent4D Dataset.

This dataset includes 14 synthetic and 12 in-the-wild monocular videos. All the videos have only one moving object with a white background. 7 of the synthetic videos are provided with multi-view ground-truth for quantitative evaluation. Each input monocular video with a static camera is set at an azimuth angle of 0^∘. Ground-truth images include four distinct views at azimuth angles of -75^∘, 15^∘, 105^∘, and 195^∘, respectively, while keeping elevation, radius, and other camera parameters the same with input camera.

#### 4.2.2 Plenoptic Video Dataset.

A high-quality real-world dataset consists of 6 scenes with 30FPS and 2028 × 2704 resolution. There are 15 to 20 camera views per scene for training and 1 camera view for testing. Though the dataset has multi-view synchronized cameras, all the viewpoints are mostly limited to the frontal part of scenes.

### 4.3 Results and Analysis

#### 4.3.1 4D Generation.

We evaluate and compare DreamGaussian4D [], which is a recent 4D Gaussian-based state-of-the-art generative model with open-sourced code, and dynamic NeRF-based methods in Tab. 1 on Consistent4D dataset with ours. Scores on individual videos are calculated and averaged over four novel views mentioned above. Note that flow supervision is effective and helps with 4D generative Gaussian representation. We showcase our superior qualitative results in Fig. 4. Compared to DreamGaussian4D, our method shows better quality as shown in Fig. 6 after the same number of training iterations.
For the two hard dynamic scenes shown in Fig. 6, our method benefit from flow supervision and generate desirable motions, while DG4D shows prominent artifacts on the novel views. Besides, our method also shows less color drifting compared with dynamic NeRF-based method Consistent4D in Fig. 5, and our results are more consistent in terms of texture and geometry.

Figure: (a) $Flame$ $Steak$
Refer to caption: x7.png

#### 4.3.2 4D Novel View Synthesis.

We visualize rendered images and depth maps of a very recent state-of-the-art 4D Gaussian method RT-4DGS  [] with (yellow) and without (red) our flow supervision in Fig. 7(a) and Fig. 7(b). According to zoom-in comparisons, our method can consistently model realistic motions and correct structures, even on glossy objects with specular highlights. These regions are known to be challenging [, ] for most methods, even under adequate multi-view supervision. Our method can reduce ambiguities in photometric supervision by involving motion cues and is shown to be consistently effective across frames. By using an off-the-shelf optical flow algorithm [], we found that only 1$\%$ to 2$\%$ of image pixels from Plenoptic Video Dataset have optical flow values larger than one pixel. Since our method benefits 4D Gaussian-based methods more on the regions with large motions, we report PSNR numbers on both full scene reconstruction and dynamic regions (optical flow value $>1$) in Tab. 2. With the proposed flow supervision, our method shows better performance on all scenes and the gains are more prominent on dynamic regions. Consequently, our method also achieves state-of-the art results on 4D novel view synthesis.

**Table 2: Quantitative evaluation between ours and other methods on the DyNeRF dataset []. We report PSNR numbers on both full-scene novel view synthesis and dynamic regions where the ground-truth optical flow value is larger than one pixel. “Ours” denotes RT-4DGS with the proposed flow supervision.**
| Method | Coffee Martini | Spinach | Cut Beef | Flame Salmon | Flame Steak | Sear Steak | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HexPlane [] | - | 32.04 | 32.55 | 29.47 | 32.08 | 32.39 | 31.70 |
| K-Planes [] | 29.99 | 32.60 | 31.82 | 30.44 | 32.38 | 32.52 | 31.63 |
| MixVoxels [] | 29.36 | 31.61 | 31.30 | 29.92 | 31.21 | 31.43 | 30.80 |
| NeRFPlayer [] | 31.53 | 30.56 | 29.35 | 31.65 | 31.93 | 29.12 | 30.69 |
| HyperReel [] | 28.37 | 32.30 | 32.92 | 28.26 | 32.20 | 32.57 | 31.10 |
| 4DGS [] | 27.34 | 32.46 | 32.90 | 29.20 | 32.51 | 32.49 | 31.15 |
| RT-4DGS [] | 28.33 | 32.93 | 33.85 | 29.38 | 34.03 | 33.51 | 32.01 |
| Ours | 28.42 | 33.68 | 34.12 | 29.36 | 34.22 | 34.00 | 32.30 |
| Dynamic Region Only |  |  |  |  |  |  |  |
| RT-4DGS [] | 27.36 | 27.47 | 34.48 | 23.16 | 26.04 | 29.52 | 28.00 |
| Ours | 28.02 | 28.71 | 35.16 | 23.36 | 27.53 | 31.15 | 28.99 |

Figure: Figure 8: Visualization of optical and Gaussian flows on the input view and a novel view. “Ours (no flow)” denotes our model without flow supervision while “Ours” is our full model. Note that optical flow values of the background should be ignored because dense optical flow algorithms calculate correspondences among background pixels. We calculate optical flow $flow^{o}_{t_{1}t_{2}}$ on rendered sequences by autoflow []. From the $\#$1 and the $\#$4 column, we can see that both rendered sequences on input view have high-quality optical flow, indicating correct motions and appearance. Comparing Gaussian flows at the $\#$2 and the $\#$5 column, we can see that the underlining Gaussians will move inconsistently without flow supervision. It is due to the ambiguity of appearance and motions while only being optimized by photometric loss on a single input view. Aligning Gaussian flow to optical flow can drastically improve irregular motions ( $\#$3 column) and create high-quality dynamic motions ($\#$6 column) on novel views.
Refer to caption: x9.png

## 5 Ablation Study

We validate our flow supervision through qualitative comparisons shown in Fig. 6. Compared with Ours (no flow) and Ours, the proposed flow supervision shows its effectiveness on moving parts. For the skull, 3D Gaussians on the teeth region initialized at $t=t_{1}$ are very close to each other and are hard to split apart completely when $t=t_{2}$. Because the gradient of incorrectly grouped Gaussians is small due to the small photometric MSE on view 0. Moreover, SDS supervision works on latent domains and cannot provide pixel-wised supervision. And the problem becomes more severe when involving Local Rigidity Loss (comparing Ours-r and Ours) because the motions of 3D Gaussians initialized at $t=t_{1}$ are constrained by their neighbors and the Gaussians are harder to split apart at $t=t_{1}$. Similarly, for bird, regions consisting of thin structures such as the bird’s beak cannot be perfectly maintained across frames without our flow supervision. While originally utilized in 4D Gaussian fields [] to maintain the structure consistency during motion, Local Rigidity Loss as a motion constraint can incorrectly group Gaussians and is less effective than our flow supervision.

We also visualize optical flow $flow^{o}_{t_{1}t_{2}}$ and Gaussian flow $flow^{G}_{t_{1}t_{2}}$ with and without our flow supervision in Fig. 8. In both cases, the optical flow $flow^{o}_{t_{1}t_{2}}$ between rendered images on the input view are very similar to each other (shown in $\#$1 and $\#$ 4 column) and align with ground-truth motion because of direct photometric supervision on input view. However, comparing optical flows on novel view as shown in $\#$3 and $\#$6, without photometric supervision on novel views, inconsistent Gaussian motions are witnessed without our flow supervision. Visualization of Gaussian flow $flow^{G}_{t_{1}t_{2}}$ as in $\#$2 column also reveals the inconsistent Gaussian motions. Incorrect Gaussian motion can still hallucinate correct image frames on input view. However, this motion-appearance ambiguity can lead to unrealistic motions from novel views (the non-smooth flow color on moving parts in $\#$3). While $\#$5 shows consistent Gaussian flow, indicating the consistent Gaussian motions with flow supervision.

## 6 Conclusion and Future Work

We present GaussianFlow, an analytical solution to supervise 3D Gaussian dynamics including scaling, rotation, and translation with 2D optical flow. Extensive qualitative and quantitative comparisons demonstrate that our method is general and beneficial to Gaussian-based representations for both 4D generation and 4D novel view synthesis with motions.
In this paper, we only consider the short-term flow supervision between every two neighbor frames in our all experiments. Long-term flow supervision across multiple frames is expected to be better and smoother, which we leave as future work. Another promising future direction is to explore view-conditioned flow SDS to supervise Gaussian flow on novel view in the 4D generation task.

## 7 Acknowledgments

We thank Zhengqi Li and Jianchun Chen for thoughtful and valuable discussions.

## Appendix 0.A Additional Implementation Details

A detailed pseudo code for our flow supervision can be found at Algorithm $\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{1}$. We extract the projected Gaussian dynamics and obtain the final Gaussian flow by rendering these dynamics. Variables including the weights and top-$K$ indices of Gaussians per pixel (as mentioned in implementation details of our main paper) are calculated in CUDA by modifying the original CUDA kernel codes of 3D Gaussian Splatting []. And Gaussian flow $flow^{G}$ is calculated by Eq.8 with PyTorch.

In our 4D generation experiment, we run 500 iterations static optimization to initialize 3D Gaussian fields with
a batch size of 16. The Tmax in SDS is linearly decayed
from 0.98 to 0.02. For dynamic representation, we run
600 iterations with batch size of 4 for both DG4D [] and ours. The flow loss weight $\lambda_{1}$ in Eq. 11 of our main paper is $1.0$.

In our 4D novel view synthesis experiment, we follow RT-4DGS[] except that we add our proposed flow supervision for all cameras. The flow loss weight $\lambda_{1}$ in Eq. 11 of our main paper is $0.5$.

Figure: Algorithm 1 Detailed pseudo code for GaussianFlow

## Appendix 0.B More Results

### 0.B.1 More Gaussian Flow in 4D Generation.

More comparisons between Gaussian flow $flow^{G}$ and optical flow $flow^{o}$ on rendered images are shown in Fig. 9. The first row of each example is the rgb frames rendered from a optimized 4D Gaussian field. We rotate our cameras for each time steps so that the object can move as optimized and the camera is moving at the same time to show the scene from different angles. The second row of each example shows the visualized Gaussian flows. These Gaussian flows are calculated by the rendered images of consecutive time steps at each camera view, therefore, containing no camera motion in the flow values. The third row is the estimated optical flows between the rendered images of consecutive time steps at each camera view. We use off-the-shelf AutoFlow [] for the estimation. We can see that enhanced by the flow supervision from the single input view, our 4D generation pipeline can model fast motion such as
the explosive motion of the gun hammer (see the last example in Fig. 9).

### 0.B.2 More Results on the DyNeRF Dataset.

More qualitative results on DyNeRF dataset [] can be found in Fig. 10 and our video.

Figure: Figure 9: Visualization of Gaussian flow $flow^{G}$ and optical flow $flow^{o}$ on rendered sequences from different views.
Refer to caption: x10.png

Figure: (a) $Sear$ $Steak$
Refer to caption: x12.png

Figure: Figure 11: $Flame$ $Salmon$
Refer to caption: x14.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. There are many other methods that use tracking models as their strong 2D priors. How do the authors incorporate tracking priors for 4D reconstruction and generation? What is the difference between this approach and using optical flow? Which method might be better?

2. What about applying flow supervision to other dynamic 3D Gaussian representations, such as 4D-GS (Wu et al., CVPR 2024), space-time Gaussians (Li et al., CVPR 2024), or 4D-Rotor-GS (Duan et al., SGA 2024)?

Overall, my main concerns are the lack of experiments, discussions and the importance of contributions.
soundness: 3
strengths: 1. The authors propose a novel theory that demonstrates the efficiency of flow supervision.
2. The authors propose both 4D reconstruction and generation experiments to demonstrate the efficiency of their method.
3. Experiments show that this paper outperforms some important baselines in certain benchmarks.
summary: This paper proposes a novel method that enhances 4D Gaussians using flow supervision. The authors introduce dense optical flow as a strong prior to supervise 4D Gaussians, ensuring the accuracy of motion. They implement the method in CUDA and demonstrate its efficiency. Experiments show that Gaussian-flow has the potential to model fast and complex dynamic scenes.
weaknesses: 1. I know there are some concurrent works focusing on flow-based Gaussian splatting, such as: "GFlow: Recovering 4D Worlds from Monocular Video" and "Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction." Please consider citing, comparing, or discussing these concurrent works.

2. The authors mention "CUDA with minimal overhead." Please provide more ablation studies and evaluation metrics(PSNR, memory, GPU, training times) on the reconstruction dataset and some generation benchmarks to support this claim.

3. I just think using optical flow as supervision can only enhance the quality of 4D Gaussians. It didn't solve the failure cases of 4D Gaussians (and the improvement is not strong, just ~0.5 PSNR). Therefore, I just think the novelty and the application of this method are limited.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: The paper's flow-rendering algorithm for 3D Gaussian Splatting shows promising potential beyond its demonstrated applications. I suggest expanding the discussion section to explore additional use cases:

1. Camera pose estimation and bundle adjustment, particularly for scenarios where traditional methods struggle without a flow prior
2. Creation of pseudo-ground-truth flow data using static scenes with known camera poses (accurate enough), which could serve as a novel benchmark for evaluating optical flow estimators

While extensive experiments in these directions might be beyond the current scope, including these potential applications in the future work section would highlight the broader impact of the proposed method and inspire further research in the community.
soundness: 3
strengths: There are two compelling reasons to accept the paper:

1. The paper introduces a plug-and-play flow rendering algorithm for Gaussian Splatting that leverages existing flow estimators and video trackers. The proposed flow supervision is particularly valuable as it extends beyond dynamic reconstruction to static scenarios where camera poses are unknown. When COLMAP fails or camera poses are unavailable, robust flow estimators can provide the necessary prior for bundle optimization of 3D Gaussians and camera poses. This simple yet effective solution for rendering Gaussian Flow represents a significant contribution to the field.

2. The paper presents a framework for 4D generation that utilizes flow supervision. While generating a front-view video is straightforward, the key innovation lies in how this framework effectively supervises dynamic Gaussians through flow rendering and supervision, taking the best use of limited front-view ground truths. This approach, though conceptually simple, solves a non-trivial technical challenge in 4D generation. Such an exploration of using flow loss in 4D generation is inspiring.
summary: This paper introduces a novel flow rendering algorithm compatible with Gaussian Splatting. The key technical contribution lies in its approach to computing optical flow: it first normalizes pixel coordinates relative to the previous frame's Gaussian positions, then maps these pixels to their corresponding locations in the current frame's moved Gaussians, and finally calculates flow values through alpha blending. The algorithmic simplicity is kept while maintaining effectiveness. It is also a natural integration with existing Gaussian Splatting frameworks. I believe that it has a broad applicability across multiple domains.
weaknesses: While the paper is generally sound, I have a few minor suggestions for improvement:

1. The paper should acknowledge and cite the flow rendering implementation found in "SC-GS: Sparse-controlled Gaussian splatting for editable dynamic scenes." Although their approach simply renders Gaussian mean shifts for flow mapping, a proper citation would help contextualize the current work's contribution.

2. To strengthen the paper's evaluation, I recommend including additional comparisons:
   - A novel view synthesis (NVS) comparison with SC-GS, the current state-of-the-art in this domain
   - A 4D generation comparison with "SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer" (ECCV '24), which is now publicly available
These additional comparisons would provide more comprehensive validation of the proposed method's effectiveness.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please, see the weaknesses section.

Overall, the approach to modeling gaussian flow and supervising it with a pretrained flow model in this paper is interesting. However, the results are not entirely convincing, as the paper lacks comparisons with state-of-the-art methods.
soundness: 3
strengths: 1. The introduction of Gaussian flow represents a significant innovation. The paper successfully demonstrates how this concept enables dynamic supervision using optical flow, which is novel in the realm of Gaussian Splatting for 4D tasks.
2. The authors conducted extensive evaluations, showing state-of-the-art results in both 4D content generation and 4D novel view synthesis on multiple challenging datasets (Plenoptic Video, NeRF-DS, and Consistent4D). Improvements in PSNR and SSIM values across dynamic regions underscore the effectiveness of GaussianFlow in handling complex motion.
3. The paper includes valuable ablation studies, demonstrating that Gaussian flow supervision directly contributes to improved visual consistency and reduced motion-appearance ambiguity. This reinforces the effectiveness of GaussianFlow’s design choices.
summary: The paper presents a novel approach, GaussianFlow, that enhances 4D content creation and novel view synthesis through Gaussian Splatting dynamics supervised by optical flow. The authors propose a “Gaussian flow” model to bridge 3D Gaussian dynamics and pixel velocities between consecutive frames, thus providing a more effective way to manage scene dynamics in 4D content. This approach addresses limitations in previous methods, especially in handling rich motion content and mitigating color drift artifacts in 4D generation.
weaknesses: 1. The paper lacks comparisons with state-of-the-art methods in 4D lifting tasks, such as STAG4D (ECCV 2024), DreamMesh4D (NeurIPS 2024), DreamScene4D (NeurIPS 2024), Animate3D (NeurIPS 2024), and 4Diffusion (NeurIPS 2024).
2. For the 4D reconstruction task, results on single-camera scenes (e.g., Nerfies and Dycheck) would strengthen the evaluation, as these scenarios are more suitable and reasonable for flow supervision.
3. The paper focuses on modeling the flow of 3DGS but does not compare it with other methods that also utilize flow in 3DGS, such as CompactDy3DGS (ECCV 2024).

[1] Zeng, Yifei, et al. Stag4d: Spatial-temporal anchored generative 4d gaussians. ECCV 2024.

[2] Li, Zhiqi, et al. DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation. NeurIPS, 2024.

[3] Chu, Wen-Hsuan, et al. Dreamscene4d: Dynamic multi-object scene generation from monocular videos. NeurIPS. 2024.

[4] Jiang, Yanqin, et al. Animate3d: Animating any 3d model with multi-view video diffusion. NeurIPS. 2024.

[5] Zhang, Haiyu, et al. 4Diffusion: Multi-view Video Diffusion Model for 4D Generation. NeurIPS. 2024.

[6] Katsumata, et al. A compact dynamic 3d gaussian representation for real-time dynamic view synthesis. ECCV. 2024.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: For input with both object and camera motion, will the formula becomes different? Will the flow-cam be different in terms of formula given there is only view point change without point movement? Could the method be further used to decompose the object motion (by Gaussian movements) and camera motion given a video input?
soundness: 4
strengths: **Differentiable flow from Gaussian**

The proposed technique is quite useful for bridging the GS dynamics with pixel movement in the image space. The author also integrate the technique into the cuda, thus could enable various application in the community beneficial for generation and reconstruction. 


**Good illustration and figures**

Figure 1 and figure 7 shows the correctness of the method and provide a good illustration of how the algorithm works. 


**Good experimental results**

From the attached video and all experiments in the paper, I could see the better quality compared against RT-4DGS and other methods. It demonstrate the effectiveness of using optical flow supervision.


**Overhead**

By using the proposed strategy, it did add extra cost to the current pipeline but it shows the cost is within reasonable range with better performance.
summary: The paper introduces GaussianFlow to differentiable render image-space flows induced by 3D Gaussian dynamics. The induced flow could help both 4D generation and 4D novel view synthesize by utilizing the optical flow supervision from nearby frames in the video. Experimental results show superior results over existing methods on multiple datasets for both tasks.
weaknesses: **flow assumption on long-term or large movement**

The work only apply flow supervision on nearby frames, in the case if the fps is small or scene with large movement, I wonder will the technique and assumption still holds. The author mention their method is not suitable for long-term flow, I wonder is the restriction coming from inaccurate flow estimation or the basic assumption. 


**Ambiguity of optical flow**

In practice, given video with both camera and object motions, the 2D optical flow will be ambiguous. Under such scenario, how could the method still benefit from the optical flow supervision to achieve good quality results. I wonder what is the corresponding contribution of with and without optical flow supervision signal in this cases.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Although the experiments demonstrate the effectiveness of the proposed method, several concerns arise:

1. How do the authors handle Gaussian densification and pruning? If fixed Gaussians are used, how can the optimal number of Gaussians be determined?

2. Since pseudo labels from optical flow are utilized, what happens if the prior model’s predictions are incorrect or inaccurate? Could this lead to failure or blurring in the proposed method?

3. Is the flow loss applied only to the input video? For novel views, do we solely rely on the SDS loss?

4. The statement on line 340 mentions, "Since our method benefits 4D Gaussian-based methods more in regions with large motions." Could the authors clarify this point? 1) What is the relationship between the degree of motion and the performance of the proposed method? 2) It would be beneficial for the authors to include more visualizations to enhance understanding.

5. Scenes typically consist of dynamic foregrounds and static backgrounds. How does the proposed method affect these two types of regions? Can this method improve results for static scenes with optical flow supervision?

6. In the ablation studies, could the authors provide some quantitative analysis? Relying on demos is not sufficient to convincingly demonstrate the effectiveness of the proposed method.

7. Could the authors provide video visualizations and comparisons to illustrate the effectiveness of the proposed method more clearly?
soundness: 3
strengths: The idea of combining Gaussian dynamics with pixel velocities is interesing. This approach is innovative and has the potential to inspire research in other topics.
summary: In this paper, the authors present a novel framework for generating 4D scenes. First, they propose using a Gaussian flow loss supervised by optical flow to model scene dynamics. Building on this representation, they extend it with the SDS loss from a generative model to facilitate 4D content generation.
weaknesses: See below.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: In related work part, L152, L4GM seems to be misclassified, it doesn't utilize text-to-video models.
soundness: 2
strengths: 1) It is natural to use 2D optical flow as supervison for 4D generation and reconstruction.
2) The paper is well-written and easy to follow
3) The authors validate their optical flow implementation on both generation and reconstruction tasks.
summary: Summary: This paper works on the task of video-to-4d, and the main contribution is adding optical supervision to the 4D generation and reconstruction task. To compute the 2D optical flow of dynamic gaussians, the authors carefully calculate the relationship between gaussian motions and 2D optical flows. They modify the original CUDA implementation of 3D Gaussian Splatting to achieve efficient optical flow rendering and loss back propogation. Experiments on 4D generation and reconstruction tasks validate the effectiveness of their designs.
weaknesses: 1) More comparison methods should be included in term of both 4D generation and 4D reconstruction. For example, 4DGen/STAG4D/Diffusion^2 for 4D generation and MotionGS for 4D reconstruction. Especially, MotionGS also adopts optical flow supervision for dynamic Gaussian, and I think it is necessary to point out how this work differ from that one.
2) The authors didn't provide the training time of the proposed method. For example, the comparison between training time w/ and w/o the proposed optical flow loss. This is important consindering efficiency of the proposed method.
3) Quantitative ablation on 4d generation task is suggested