BulletGen: Improving 4D Reconstruction with Bullet-Time Generation

## Abstract

Abstract Transforming casually captured, monocular videos into fully immersive dynamic experiences is a highly ill-posed task, and comes with significant challenges, e.g . , reconstructing unseen regions, and dealing with the ambiguity in monocular depth estimation. In this work we introduce BulletGen, an approach that takes advantage of generative models to correct errors and complete missing information in a Gaussian-based dynamic scene representation. This is done by aligning the output of a diffusion-based video generation model with the 4D reconstruction at a single frozen “bullet-time” step. The generated frames are then used to supervise the optimization of the 4D Gaussian model. Our method seamlessly blends generative content with both static and dynamic scene components, achieving state-of-the-art results on both novel-view synthesis, and 2D/3D tracking tasks.

## 1 Introduction

The task of 3D reconstruction from color images is fundamental to computer vision, with a variety of solutions being proposed over the years [26, 51, 63, 64].
While impressive strides have been made towards improving both the completeness and accuracy of results, most methods and datasets focus exclusively on static scenes. However, the real world is dynamic, and the ability to reconstruct dynamic content opens up many new opportunities in immersive media generation and robotics.

The reconstruction of dynamic scenes presents a very challenging 4D problem, as it requires reasoning about both geometry and motion of the scene. This is exacerbated by the fact that traditional 4D reconstruction methods require capturing multi-view images of dynamic scenes with expensive camera rigs [53, 23], resulting in limited amount of useful datasets for algorithm development and evaluation. Therefore, building on developments in novel 3D scene representations [26, 51], the attention of the research community in recent years has focused on 4D reconstruction from single-view monocular videos [49, 54, 76]. But, as a monocular video only observes the scene from a single viewpoint at any given timestep, the problem of 4D reconstruction in this context is heavily under-constrained. As a consequence, existing methods can only find locally optimal solutions, and fail when synthesizing novel views substantially deviating from the training set views (Fig. [1](https://arxiv.org/html/2506.18601v1#S1.F1)).

Motivated by breakthroughs in learning priors from large-scale data, many recent methods have sought to mitigate the local minima problem by leveraging generative models – particularly diffusion-based methods [61, 67, 69] – to constrain the output to lie in the distribution of natural images. However, in the majority of cases, these methods only predict 2D projections of a dynamic scene, and therefore fail to take advantage of the efficient rendering and global consistency advantages provided by 3D scene representations that are optimized per-scene, such as Neural Radiance Fields (NeRFs) [51] and 3D Gaussians [26]. In this paper, we propose a method to incorporate these 2D projections into a consistent and plausible 4D reconstruction.

Our method, termed BulletGen, uses a video diffusion model to generate novel views for selected frozen timesteps – so-called bullet times [40, 72].
The diffusion model is conditioned on a rendered frame along with natural language image captions from the input video.
The generated views are then used to iteratively supervise a global 3D representation. This latter step is achieved by accurately tracking and aligning the generated views to the input coordinate frame, and by using a novel reconstruction loss based on photometric, perceptual, semantic, and depth errors.
By repeating this process for different time stamps, the initially inconsistent 2D projections are robustly incorporated into a consistent dynamic 3D reconstruction. We use 3D Gaussians as the underlying scene representation, but our method is general and could work with any differentiable 3D representation.

In summary, we make the following contributions:

- •
We propose a method for dynamic 3D scene reconstruction from monocular RGB videos by augmenting the training views using a generative video diffusion model at selected bullet times.
- •
We propose a robust reconstruction loss that allows us to align the generated views in a global coordinate frame, and use them to supervise a dynamic Gaussian reconstruction.
- •
We compare our method against prior work, and show that BulletGen achieves state-of-the-art results on several benchmark datasets for novel view synthesis quality, and 2D/3D tracking accuracy.
- •
The reconstructed dynamic scene also contains new synthesized parts of the scene that seamlessly and plausibly blend in the original scene for both the static
(*e.g*. new pillows in the cat scene and generated walls in the skating scene in Fig. [1](https://arxiv.org/html/2506.18601v1#S1.F1)) and the dynamic parts (*e.g*. backside of the cat, the full head of the dog).

Figure: Figure 1: Extreme novel view synthesis of a 4D scene with generative model guidance in frozen-time instances (bullet times). The input is only a monocular video on the left. We compare to the Shape-of-Motion (SoM) [73] method on the cat and dog sequences from the iPhone dataset [14] and the skating sequence from Nvidia dataset [87].
Refer to caption: extracted/6560608/imgs/teaser/skating/input.jpg

## 2 Related work

The problem of 3D reconstruction and novel-view synthesis has a long history in the fields of computer vision and graphics.
Early approaches used dense multi-view images [8, 28, 75] to capture a spatio-angular light fields [18, 34], which could be re-parameterized to generate novel views via quadrilinear interpolation. However, the number of views required by sampling theory to enable such interpolation has been shown to be exorbitantly high [3]. As a result, novel view synthesis remained a niche task requiring expensive camera rigs, specialized sensors, or time-consuming capture setups [8, 16, 52, 75]. Mildenhall *et al*. [50] and Zhou *et al*. [95] were among the first to seek to overcome the sampling limits using learned data priors encoded as the weights of a convolutional neural network. Subsequent work built on this work by exploiting the ever-increasing capabilities of deep neural networks to not only lower the sampling requirements for view synthesis, even down to a single image  [4, 20, 27, 45, 47, 48, 68, 90], but also proposed new representations that could be optimized per scene to achieve exceptionally high quality and improved spatio-temporal consistency by exploiting scene-specific correlations across views  [49, 36, 30, 42]. These two research directions naturally converge on the task of 4D reconstruction from monocular videos. However, in this context, past work that has used per-scene optimization to track and correlate points across frames has often failed to take advantage of the strong data priors of pre-trained models specifically for view synthesis.

Per-scene monocular reconstruction.
Following the seminal work of Mildenhall *et al*. [51] on static reconstruction by optimizing a scene-specific 3D representation by Neural Radiance Fields (NeRFs), many works [1, 13, 37, 54, 55, 59, 65, 66, 84] have adopted a similar strategy to exploit more global context for the problem of 4D reconstruction from a monocular video. The underlying representation often models the scene as an implicit field [79] parameterized by time, which is optimized using the input frames as supervision. The use of a single globally consistent representation, which also implicitly encodes a local smoothness prior, allows per-scene optimization to achieve impressive reconstruction results without leveraging any learned data priors [91]. However, as the optimization process can be poorly regularized and time-consuming, several methods have proposed using a monocular depth prior to guide the optimization process [9, 44, 62].
Furthermore, the introduction of 3D Gaussian splats [26] in recent years has led to methods that reconstruct either scene geometry [41, 76, 84, 44, 7], or both geometry and motion [36, 43, 49, 85, 31, 12], explicitly. Thus, a recent class of methods has additionally used 2D motion trajectories from pre-trained models to initialize and supervise the optimization of a scene-specific global representation [42, 33, 73]. However, these methods use learned priors primarily to regularize the optimization and to reason about the visible parts of the scene. As such, while they generate more stable reconstructions in these visible regions, they perform no better than previous methods on view extrapolation tasks.

Generative 4D reconstruction.
Given the extremely high number of images required to meet the Nyquist sampling rate for view interpolation from multi-view images [3, 50], and the fundamentally under-constrained nature of the view extrapolation problem, many works have explored the use of learned data priors to circumvent these theoretical limitations [4, 50, 86, 90, 95]. In the extreme case, these methods aim for view synthesis from a single RGB image. The most effective prior for this task, in terms of output quality, turns out to be a learned conditional probability distribution over the space of natural images. This is represented as a deep neural network that generates samples from the underlying distribution [2, 17, 80]. Among this class of “generative” methods, Denoising Diffusion Probabilistic Models (DDPM) [21] have become especially popular due to their stability, mode coverage, quality, and flexibility [6]. Consequently, a number of recent works have used diffusion to set the bar of quality on 3D reconstruction and view synthesis from single or sparse images [88, 15, 46, 78, 70]. Subsequent work has extended the basic approach to 4D reconstruction from single images [67, 81, 94], sparse image sets [5, 94], and from monocular videos [61, 69, 89]. These methods rely on sampling all output frames from the generative model. Consequently, they do not provide accurate control over the camera movement and lack explicit spatio-temporal consistency constraints. In addition, the high memory requirements of video diffusion models limit these methods to short video sequences.

Concurrent work.
CAT4D [77] and Vivid4D [22] are concurrent works with ours, and show the use of generative models to improve per-scene optimized monocular reconstructions. Both works proceed by first generating multi-view video from a monocular sequence, and then use it to supervise the optimization of a dynamic scene representation based on Gaussian splats. As such, the generation and optimization process is strongly decoupled. Our method uses an iterative approach where the generation steps alternate with the training of a Gaussian-based global 4D representation.

## 3 Method

Given $N$ frames of a monocular video sequence $\{I_{t}\}_{t=1}^{N}$ as input, our method acquires an initial 4D reconstruction of the scene as dynamic Gaussian splats. It then uses diffusion-based generative augmentation to obtain new observations for under-constrained regions, as well as synthesizing unseen parts of the scene. Using the original frames along with the generated views as supervision, the initial reconstruction is robustly optimized to obtain a photorealistic and globally consistent dynamic scene representation that allows for rendering of novel views at arbitrary time stamps and camera positions. Fig. [2](https://arxiv.org/html/2506.18601v1#S3.F2) provides an overview of our pipeline.

Figure: Figure 2: BulletGen architecture. Starting from a monocular RGB video, we reconstruct the dynamic scene with Shape-of-Motion [73] given data-driven priors (motion masks, depths, long-term 2D tracks). Then, we generate novel views at selected frozen timesteps (bullet times) using a conditioned generative model. These generated views are localized and mapped to the current scene using an optimization based on photometric, perceptual, semantic, and depth errors. Final 4D reconstruction augments the scene and allows for higher quality extreme novel view synthesis.
Refer to caption: x1.png

### 3.1 Initial dynamic Gaussian Splatting reconstruction

To bootstrap the initial reconstruction, we build on top of several state-of-the-art methods.
We choose dynamic Gaussian Splatting as a scene representation, which is an extension to the original 3D Gaussian Splatting (3DGS) [26] method.
In particular, we use Shape-of-Motion [73], which relies on a range of scene priors extracted using methods like Track-Anything [30, 82] to extract masks for the moving objects, Depth Anything [83] to estimate relative monocular depth maps, UniDepth [58, 57] for metric depth alignment, and TAPIR [11, 10] for obtaining long-range 2D tracks of moving objects.

Each 3D Gaussian is represented by RGB color $c\in[0,1]^{3}$, its center position $\mu(t)\in\mathbb{R}^{3}$, rotation $r(t)\in\mathbb{R}^{4}$ (quaternion), scale $s\in\mathbb{R}^{3}$ and opacity $o\in[0,1]$.
For static Gaussians, $\mu(t)$ and $r(t)$ are constant for all time stamps.
For dynamic Gaussians, they are defined as a time-dependent weighted combination of a global set of motion bases [73] that are learned during optimization (usually a low number, *e.g*., 20).
Standard 3DGS [26] can be used to render the dynamic Gaussians for a given time stamp $t$.
Given the required camera pose, all Gaussians are sorted by distance from the camera center and then rendered by alpha-composing the splatted 2D projections, which are affine approximations of the exact 3D projection.
Given the complexity and ill-posedness of the task due to limited observations, we do not model view-dependent illumination changes with spherical harmonics.
In the initial bootstrapping phase, we run Shape-of-Motion for 1000 optimization epochs to obtain a dynamic scene reconstruction without any generative component (Fig. [2](https://arxiv.org/html/2506.18601v1#S3.F2)).

### 3.2 Generative augmentation

After initial scene reconstruction, we generate a set $\{G_{k}^{t}\}_{k=1}^{K}$ of $K$ novel views of the scene at selected times $t$, and select a conditioning view and target motion (explained below).
We generate novel views of the dynamic scene at bullet time $t$, assuming that the scene is frozen in time.
We use a controllable image-to-video diffusion model that is trained to create novel views of the scene given a single image and a text prompt, which is generated by Llama3 [19].
The conditioning prompt is a highly descriptive image captioning from the original monocular video, which serves as an anchor to be consistent with the original input.
The used generative model also controls the motion by training different models for target motions, such as left and up directions.
Then, the generated images $\{G_{k}^{t}\}_{k=1}^{K}$ are localized in the scene and used to update the scene with new observations.

Camera tracking.
We estimate initial relative camera poses between frames of the same bullet time using the recently introduced VGGT [71] transformer.
Since VGGT-predicted depth is inaccurate, we estimate state-of-the-art monocular depth from MoGe [74] and align it to VGGT depth.
As MoGe/VGGT depths are not metric, we estimate a single scaling factor to align them to the current 4D reconstruction by minimizing the depth and RGB reprojection error in the covisible regions, producing final depth estimates $\{D_{k}\}_{k=1}^{K}$.
At this stage, the camera poses of generated views are roughly aligned to the current scene reconstruction, but the alignment is not yet pixel-perfect, which is crucial for the following scene augmentation.
To achieve pixel-perfect alignment, we leverage the Gaussian Splatting SLAM method SplaTAM [25] for accurate camera tracking.
Camera intrinsics for the generated cameras are assumed to be equal to the estimated intrinsics of the original monocular video.
Extrinsics $\mathrm{\textbf{E}}_{k}\in\mathbb{R}^{4\times 4}$ for generated views $G_{k}^{t}$ are initialized to the scale-aligned VGGT estimates.
Then, the scene is rendered using the rendering function $\mathcal{R}(\mathrm{\textbf{E}}_{k})$ producing RGB images $\mathcal{R}_{I}(\mathrm{\textbf{E}}_{k})$ and depth images $\mathcal{R}_{D}(\mathrm{\textbf{E}}_{k})$.
Additionally, we render silhouettes $\mathcal{R}_{S}(\mathrm{\textbf{E}}_{k})$ by accumulating Gaussian densities along the ray, which are used to define visibility masks as $V_{k}=\mathcal{R}_{S}(\mathrm{\textbf{E}}_{k})>0.99$.

Robust loss and optimization.
The minimized loss function consists of four terms representing the photometric, perceptual, semantic, and depth errors.
The photometric loss is the L1 loss in the RGB space, the perceptual loss is the LPIPS loss [92] based on AlexNet [32] features, and the semantic loss is the cosine similarity between CLIP scores [60] of the renderings $\mathcal{R}_{I}(\mathrm{\textbf{E}}_{k})$ and the generated images $G_{k}$.
Finally, the depth loss is the L1 distance between depth renderings and the aligned depth maps.
Since the generated images are usually not perfectly 3D consistent in the pixel space, we put the highest weight on the semantic and perceptual losses and keep the photometric loss only as a low-level learning signal.
The final loss function is the weighted sum:

$$ $\begin{split}\mathcal{L}(\{\mathrm{\textbf{E}}_{k}\}|V_{k})=\sum_{k=1}^{K}& \alpha_{1}\text{L1}\Bigl{(}G_{k},\mathcal{R}_{I}(\mathrm{\textbf{E}}_{k})\Big{ |}V_{k}\Bigr{)}+\alpha_{2}\text{LPIPS}\Bigl{(}G_{k},\mathcal{R}_{I}(\mathrm{ \textbf{E}}_{k})\Big{|}V_{k}\Bigr{)}\\ +&\alpha_{3}\text{CLIP}\Bigl{(}G_{k},\mathcal{R}_{I}(\mathrm{\textbf{E}}_{k}) \Big{|}V_{k}\Bigr{)}+\alpha_{4}\text{L1}\Bigl{(}D_{k},\mathcal{R}_{D}(\mathrm{ \textbf{E}}_{k})\Big{|}V_{k}\Bigr{)}\,,\end{split}$ (1) $$

where we compute the loss only over the visible area $V_{k}$.
We optimize camera poses $\{\mathrm{\textbf{E}}_{k}\}$ by minimizing the loss for 100 epochs, starting from the initialization described above.
All other parameters, including the scene representation, are fixed.
Once the camera poses are optimized, producing $\{\mathrm{\textbf{E}}_{k}^{*}\}$, it mostly leads to sufficiently accurate, pixel-wise alignment. To remove bad estimates, we keep only the first $K^{\prime}$ generated views up until the first generative view whose loss $\mathcal{L}$ ([1](https://arxiv.org/html/2506.18601v1#S3.E1)) is above a threshold $\gamma$, which is set in a conservative way so only well-aligned views are retained.
After this step, we have $K^{\prime}\leq K$
generated views $\{G_{k}\}$, their depths $\{D_{k}\}$, and optimized camera poses $\{\mathrm{\textbf{E}}_{k}^{*}\}$.

Figure: Figure 3: Extreme novel view synthesis across space and time. The generation for training was performed $n_{G}=7$ times for $n_{S}=5$ bullet-time stamps, *i.e*. 1, 45, 90, 135, 180 for a sequence with length of 180 frames. The renderings shown here are at time stamps and viewpoints that were not generated using a generative model, which shows that using only several bullet-time reconstructions is enough to reconstruct a dynamic scene. Temporal slices for all 180 time stamps are shown in Fig. [5](https://arxiv.org/html/2506.18601v1#S4.F5).
Refer to caption: extracted/6560608/imgs/example/depth0.png

Densification.
After obtaining well-aligned generative views, we now proceed to updating the scene.
First, we perform Gaussian densification based on the densification mask proposed in SplaTAM [25]:

$$ $M_{k}=\Bigl{(}\mathcal{R}_{S}(E_{k}^{*})<0.5\Bigr{)}+\Bigl{(}D_{k}<\mathcal{R} _{D}(E_{k}^{*})\Bigr{)}\Bigl{(}L_{1}(D_{k},\mathcal{R}_{D}(E_{k}^{*}))<\lambda \text{MDE}\Bigr{)}\,,$ (2) $$

where we densify areas with insufficient density or where new geometry is in front of the current geometry, unless it is due to noise as measured by $\lambda=50$ times the median depth error [25].
This way, we initialize a new Gaussian at every pixel in the densification mask based on its color and depth.
For the newly densified Gaussians, we decide if they are static or dynamic based on the nearest neighboring Gaussian’s static/dynamic label, where the dynamic Gaussians’ locations are moved to the currently generated bullet time stamp.
The weights for the motion bases of new dynamic Gaussians are initialized to the nearest neighbor.

Scene update.
To update the reconstructed scene, we optimize a weighted joint loss, which is a sum of the proposed tracking loss $\mathcal{L}$ ([1](https://arxiv.org/html/2506.18601v1#S3.E1)) on the generated views, including all previous generated views over all bullet time stamps, and the SoM loss function to always keep the scene consistent with the original video.
Since the scene is already densified, we do not need the visibility masks $V_{k}$ in this stage and, thus, the loss is computed over the entire generated image, as compared to the loss used for camera tracking.
We optimize this joint loss function for 100 epochs by optimizing parameters of all Gaussians and the motion bases.
After this, we repeat it all with new generated views
(Fig. [2](https://arxiv.org/html/2506.18601v1#S3.F2)).

Time selection.
One of the hyperparameters of our method is the number $n_{S}$ of sampled bullet time stamps.
We sample them uniformly between the first and last time stamps, *i.e*. $t=1$ and $t=N$.
In order to guarantee smooth motion, it is important to keep some distance between sampled time stamps since each bullet time generation may contain slight inconsistencies.
If the distance between two sampled time stamps is low, this can generate effects similar to jittering.
However, if the distance is sufficiently large,
it gives enough time for smooth transitions between two bullet-time reconstructions with their own slight inconsistencies.
For robust estimation, we start with the middle frame bullet-time stamp, *e.g*. $N/2$.
Then, we proceed with the furthest time stamps.
For instance, when $N=100$ and $n_{S}=5$, the sampled time stamps are $\{1,25,50,75,100\}$, but in the following order $\{50,1,100,25,75\}$. Also note that the bullet-time stamp selection can be progressive and proceed auto-regressively, and in our example, can continue with additional time stamps $\{12,37,62,87\}$ for $n_{S}=9$.

View selection.
The used generative model currently supports only static scenes and comes with two operating modes for generating left- and upward trajectories.
By flipping the image horizontally, applying the leftward model, and flipping back, we can also generate rightward trajectories.
Note that we do not apply the same principle to the upward direction, because the model only works well with upright imagery.
Thus, we select between three modes for the view selection, *i.e*. left, right, up.
Once bullet time is selected, we proceed with $n_{G}$ generations for the selected time.
We tested three modes.
First, for $n_{G}=3$, we select {up, left, right} in this order.
For $n_{G}=5$, we select {up, left, right, left, right}, and for $n_{G}=7$, the used sequence is {up, left, up, right, up, left, right}.
The view selection for generative model conditioning for each chosen direction is as follows.
For leftward motion, we choose the left-most image (either the original one or the generated one), where the angle is measured from the middle of the original camera poses and viewing at the middle of the dynamic scene.
Similarly, for rightward motion, we choose the right-most image.
In all cases, only images at the currently selected bullet-time stamp are considered, either generated or the original one (there is only one original image for every time stamp).
For upward motion, we choose the last generated image at this bullet time (or the original one if this is the first generation).

Figure: Figure 4: Qualitative evaluation on several benchmark datasets. The input is a monocular video as shown in the training view column. Both benchmark datasets (Nvidia [87] and DyCheck iPhone [14]) have several additional testing cameras. We compare novel view synthesis to Shape-of-Motion (SoM) [73] and zoom-in to highlight the differences. Our method is able to provide more accurate and sharper reconstructions, both in static and dynamic parts of the scene.
Refer to caption: extracted/6560608/imgs/eval/spin/input.png

Implementation details.
All optimizations are performed using the ADAM optimizer [29].
The batch size for the generative views and the original video frames is set to 8 (thus, we have two concatenated batches of 8 in every iteration).
The hyperparameters in Eq. ([1](https://arxiv.org/html/2506.18601v1#S3.E1)) have been empirically determined and fixed to $\alpha_{1}=0.02$, $\alpha_{2}=0.1$, $\alpha_{3}=0.1$, $\alpha_{4}=0.5$ for all experiments.
The threshold for generative views is set to $\gamma=0.4$.
For the SoM loss function on the original video, we use their default weights.
The total number of generated views per generation is fixed at $K=50$.
The number of generations used is $n_{G}=7$, while the number of selected bullet times is $n_{S}=9$, unless specified otherwise.
The average optimization time on a sequence from the iPhone dataset [14] with full resolution takes around 3 hours (including initial SoM optimization of 1.5 hours) on an Nvidia A100 80GB GPU.
All main experiments required around 1 week of A100 GPU time, and preliminary non-included experiments required an additional month of GPU time.
For evaluation on custom data, we use MegaSaM [38] to estimate camera poses.
For evaluation on benchmark datasets, we used provided poses computed from COLMAP [63, 64] for fair comparison to other methods.

## 4 Experiments

We evaluate on the DyCheck iPhone dataset [14] and Nvidia dynamic dataset [87], which are commonly used datasets for novel view synthesis evaluation in dynamic scenes.

View synthesis metrics.
We evaluate the standard PSNR, SSIM, and LPIPS scores between the ground truth and the rendered novel views.
On the iPhone dataset, we measure those metrics on the covisible regions for fair comparison to other methods.
As mentioned by [42], these scores are not robust enough on the current datasets due to camera misalignment, color differences between the training and test cameras, as well as the nature of the task.
Thus, as proposed in [42], we also report CLIP-I [60] scores, which we see as more representative in this ill-posed task.

Tracking metrics.
The iPhone dataset contains long-range 3D point tracking annotations.
Thus, we also measure 3D end-point-error (EPE) at every timestep and the percentage of points that are within a certain radius of the ground truth 3D point (denoted by $\delta_{3D}^{\tau}$, where $\tau$ is either 5cm or 10cm).
By projecting those 3D tracking annotations, we obtain 2D tracking annotations and report Average Jaccard (AJ) metric, average position accuracy ($<\delta_{avg}$) and Occlusion Accuracy (OA) metric.

Baselines.
We compare to a range of methods that are specifically designed for dynamic scene reconstruction.
Methods based on Neural Radiance Fields (NeRF) [51] and its variants are T-NeRF [35], HyperNeRF [56], and DynIBaR [39].
More recently, methods based on Gaussian Splatting were proposed, such as Deformable-3DGS [84], 4D-GS [76], Shape-of-Motion [73], and HiMoR [42].
We also compare to methods based on generative diffusion models, such as CoCoCo [96], StereoCrafter [93], ViewCrafter [88], CAT4D [77], and Vivid4D [22].
Many of these methods are concurrent with our work, are only pre-prints (not published), and do not provide code, *e.g*. [77, 22, 42].
Nevertheless, we strive to provide insightful comparisons by evaluating on the reported datasets chosen by these methods.
For 2D and 3D tracking scores, we additionally compare to CoTracker [24] and TAPIR [11], which are lifted to 3D by DepthAnything [83].

**Table 1: Evaluation on the iPhone dataset, 3D and 2D tracking. Our method outperforms all state-of-the-art methods in terms of 2D and 3D tracking accuracy as measured by various metrics.**
|  | 3D Tracking |  | 2D Tracking |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | EPE$\downarrow$ | $\delta_{3D}^{.05}\uparrow$ | $\delta_{3D}^{.10}\uparrow$ |  | AJ$\uparrow$ | $<\delta_{avg}$$\uparrow$ | OA$\uparrow$ |
| HyperNeRF [56] | 0.182 | 28.4 | 45.8 |  | 10.1 | 19.3 | 52.0 |
| DynIBaR [39] | 0.252 | 11.4 | 24.6 |  | 5.4 | 8.7 | 37.7 |
| Deformable-3D-GS [84] | 0.151 | 33.4 | 55.3 |  | 14.0 | 20.9 | 63.9 |
| CoTracker [24] + DA [83] | 0.202 | 34.3 | 57.9 |  | 24.1 | 33.9 | 73.0 |
| TAPIR [11] + DA [83] | 0.114 | 38.1 | 63.2 |  | 27.8 | 41.5 | 67.4 |
| Shape-of-Motion [73] | 0.082 | 43.0 | 73.3 |  | 34.4 | 47.0 | 86.6 |
| Ours | 0.071 | 51.6 | 77.6 |  | 36.6 | 49.5 | 87.4 |

**Table 2: Evaluation on the iPhone dataset, novel view synthesis. The proposed method achieves state-of-the-art performance in terms of novel view synthesis.**
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CLIP-I$\uparrow$ |
| --- | --- | --- | --- | --- |
| T-NeRF [35] | 15.60 | 0.55 | 0.55 | 0.86 |
| HyperNeRF [56] | 15.99 | 0.59 | 0.51 | 0.87 |
| Deform.-3DGS [84] | 11.92 | 0.49 | 0.66 | 0.79 |
| SoM [73] | 16.72 | 0.63 | 0.45 | 0.86 |
| HiMoR [42] (no code) | - | - | 0.46 | 0.89 |
| CAT4D [77] (no code) | 17.39 | 0.61 | 0.34 | - |
| Ours | 16.78 | 0.64 | 0.39 | 0.90 |

The iPhone dataset.
The iPhone dataset [14] contains 12 sequences with a moving training camera captured with hand-held iPhone device, two static test cameras (for 5 sequences), and 3D tracking annotations.
As shown in Table [1](https://arxiv.org/html/2506.18601v1#S4.T1), our method outperforms all other state-of-the-art methods on all metrics in terms of 2D and 3D tracking scores.
This is achieved by providing more constraints for scene geometry, which allows for better 3D triangulation during scene reconstruction.
We also achieve state-of-the-art scores in terms of novel view synthesis (Table [3](https://arxiv.org/html/2506.18601v1#S4.T3)).
For the sake of completeness, we add results reported in CAT4D [77] (unpublished work), which achieves slightly better PSNR and LPIPS scores, whereas our method obtains better SSIM results. However, we want to stress that the table remains incomplete in terms of CLIP-I score due to a lack of publicly accessible code and missing information about the training procedure and its overall training scale.
To compare to the recently released Vivid4D [22] method (also without publicly available code), we choose their subset of the iPhone dataset that was used in their evaluation.
Our proposed method again outperforms other methods (Table [3](https://arxiv.org/html/2506.18601v1#S4.T3)).

**Table 4: Evaluation on the Nvidia dataset. Our method improves the Shape-of-Motion (SoM) reconstruction on all metrics.**
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CLIP-I$\uparrow$ |
| --- | --- | --- | --- | --- |
| SoM | 15.26 | 0.454 | 0.388 | 0.87 |
| Ours | 17.02 | 0.462 | 0.386 | 0.87 |

The Nvidia dataset.
The Nvidia dataset contains 9 dynamic scenes captured by 12 synchronized static cameras.
We use camera 1 as input, and cameras 2 and 3 as output for evaluation.
This dataset contains testing views close to the training views, therefore the CLIP-I and LPIPS metrics do not show significant improvement, but in terms of PSNR, our improvement w.r.t. Shape-of-Motion is significant (see Table [4](https://arxiv.org/html/2506.18601v1#S4.T4)).

**Table 5: Ablation study (iPhone dataset). The number of generations $n_{G}$ per time stamp slightly improves most metrics, and the number of selected bullet-time stamps $n_{S}$ improves all metrics.**
|  |  | Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | CLIP-I$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | SoM | 16.72 | 0.63 | 0.45 | 0.86 |
| number of | generations | $n_{s}=9$, $n_{g}=3$ | 16.81 | 0.63 | 0.40 | 0.89 |
| $n_{s}=9$, $n_{g}=5$ | 16.80 | 0.63 | 0.40 | 0.89 |  |  |
| $n_{s}=9$, $n_{g}=7$ | 16.78 | 0.64 | 0.39 | 0.90 |  |  |
| number of | bullet-times | $n_{s}=3$, $n_{g}=7$ | 15.95 | 0.62 | 0.45 | 0.88 |
| $n_{s}=5$, $n_{g}=7$ | 16.36 | 0.62 | 0.44 | 0.89 |  |  |
| $n_{s}=9$, $n_{g}=7$ | 16.78 | 0.64 | 0.39 | 0.90 |  |  |

Ablation study.
The main hyperparameters of our method are the number of generations $n_{G}$ per selected bullet-time stamp, and the total number of those time stamps $n_{S}$.
Table [5](https://arxiv.org/html/2506.18601v1#S4.T5) shows that increasing $n_{G}$ per time stamp slightly improves most metrics, but is not significant given that the test cameras are not far from the training ones.
A high value for $n_{G}$ is important for extreme novel view point synthesis.
However, the number of selected bullet time stamps $n_{S}$ is vital even in the standard setting, and a higher number of time stamps improves all metrics.

Qualitative results.
Qualitative results for the iPhone dataset are provided in Fig. [1](https://arxiv.org/html/2506.18601v1#S1.F1) (cat, dog) and Fig. [4](https://arxiv.org/html/2506.18601v1#S3.F4) (spin, paper-windmill), and for the Nvidia dataset in Fig. [1](https://arxiv.org/html/2506.18601v1#S1.F1) (skating) and Fig. [4](https://arxiv.org/html/2506.18601v1#S3.F4) (balloon2).
As can be seen in Fig. [4](https://arxiv.org/html/2506.18601v1#S3.F4), the testing views in the benchmark datasets are close to the training views.
However, our method allows for much more extreme novel view synthesis, going beyond what is possible by other methods.
Unfortunately, there is no dataset with such extreme view point testing cameras, which can be seen as a limitation in this emerging field.
Fig. [3](https://arxiv.org/html/2506.18601v1#S3.F3) shows qualitative examples of extreme novel views.
BulletGen significantly improves rendering quality in such cases.
Temporal slices in the $x$ direction are shown in Fig. [5](https://arxiv.org/html/2506.18601v1#S4.F5), which highlights that our method produces smooth and consistent rendering results over time, especially when compared to SoM.

Figure: Figure 5: Temporal plane slices of extreme camera views. Visualizing the highlighted rows across time in $xt$ space shows that our method suffers from fewer temporal artifacts than Shape-of-Motion (SoM) caused by floating Gaussians and exploding geometry. Our rendered view is shown on the left.
Refer to caption: x2.png

Limitations.
BulletGen assumes that the initial Shape-of-Motion optimization performs reasonably well, at least from the viewpoints of the original video, which in turn depends on how well all monocular priors were estimated.

## 5 Conclusion

We introduced a method for dynamic 3D reconstruction from a monocular video, effectively addressing extreme novel view synthesis challenges with drastic improvements over the state of the art.
By integrating static bullet-time video generation with dynamic 3D Gaussian splatting our approach enhances both 2D and 3D tracking accuracy, and novel view synthesis. Our method seamlessly incorporates new scene elements, improving rendering quality. Our results on standard benchmark datasets confirm the efficacy of this approach for dynamic scene reconstruction in complex environments.

## References

- [1]
Ang Cao and Justin Johnson.
Hexplane: A fast representation for dynamic scenes.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 130–141, 2023.
- [2]
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng,
and Stan Z Li.
A survey on generative diffusion models.
IEEE Transactions on Knowledge and Data Engineering, 2024.
- [3]
Jin-Xiang Chai, Xin Tong, Shing-Chow Chan, and Heung-Yeung Shum.
Plenoptic sampling.
In Proceedings of the 27th annual conference on Computer
graphics and interactive techniques, pages 307–318, 2000.
- [4]
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas
Geiger, Tat-Jen Cham, and Jianfei Cai.
Mvsplat: Efficient 3d gaussian splatting from sparse multi-view
images.
In European Conference on Computer Vision, pages 370–386.
Springer, 2024.
- [5]
Gene Chou, Kai Zhang, Sai Bi, Hao Tan, Zexiang Xu, Fujun Luan, Bharath
Hariharan, and Noah Snavely.
Generating 3d-consistent videos from unposed internet photos.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), 2025.
- [6]
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.
Diffusion models in vision: A survey.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
45(9):10850–10869, 2023.
- [7]
Devikalyan Das, Christopher Wewer, Raza Yunus, Eddy Ilg, and Jan Eric Lenssen.
Neural parametric gaussians for monocular non-rigid object
reconstruction.
arXiv preprint arXiv:2312.01196, 2023.
- [8]
Abe Davis, Marc Levoy, and Fredo Durand.
Unstructured light fields.
In Computer Graphics Forum, volume 31, pages 305–314. Wiley
Online Library, 2012.
- [9]
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.
Depth-supervised NeRF: Fewer views and faster training for free.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2022.
- [10]
Carl Doersch, Pauline Luc, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta,
Joseph Heyward, Ignacio Rocco, Ross Goroshin, João Carreira, and Andrew
Zisserman.
BootsTAP: Bootstrapped training for tracking-any-point.
ACCV, 2024.
- [11]
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar,
Joao Carreira, and Andrew Zisserman.
TAPIR: Tracking any point with per-frame initialization and
temporal refinement.
In ICCV, pages 10061–10072, 2023.
- [12]
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan
Chen.
4d-rotor gaussian splatting: Towards efficient novel-view synthesis
for dynamic scenes.
In Proc. SIGGRAPH, July 2024.
- [13]
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin
Recht, and Angjoo Kanazawa.
K-planes: Explicit radiance fields in space, time, and appearance.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 12479–12488, 2023.
- [14]
Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa.
Monocular dynamic view synthesis: A reality check.
In NeurIPS, 2022.
- [15]
Ruiqi Gao*, Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo
Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole*.
Cat3d: Create anything in 3d with multi-view diffusion models.
NeurIPS, 2024.
- [16]
Todor G Georgiev, Ke Colin Zheng, Brian Curless, David Salesin, Shree K Nayar,
and Chintan Intwala.
Spatio-angular resolution tradeoffs in integral photography.
Rendering Techniques, 2006(263-272):21, 2006.
- [17]
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
Advances in neural information processing systems, 27, 2014.
- [18]
Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen.
The lumigraph.
In Seminal Graphics Papers: Pushing the Boundaries, Volume 2,
pages 453–464. 2023.
- [19]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek
Kadian, Ahmad Al-Dahle, et al.
The llama 3 herd of models, 2024.
- [20]
Yuxuan Han, Ruicheng Wang, and Jiaolong Yang.
Single-view view synthesis in the wild with learned adaptive
multiplane images.
In ACM SIGGRAPH, 2022.
- [21]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
Advances in neural information processing systems,
33:6840–6851, 2020.
- [22]
Jiaxin Huang, Sheng Miao, BangBang Yang, Yuewen Ma, and Yiyi Liao.
Vivid4d: Improving 4d reconstruction from monocular video by video
inpainting, 2025.
- [23]
Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo
Kanade, Shohei Nobuhara, and Yaser Sheikh.
Panoptic studio: A massively multiview system for social motion
capture.
In Proceedings of the IEEE international conference on computer
vision, pages 3334–3342, 2015.
- [24]
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea
Vedaldi, and Christian Rupprecht.
Cotracker: It is better to track together.
In ECCV, 2024.
- [25]
Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang,
Sebastian Scherer, Deva Ramanan, and Jonathon Luiten.
Splatam: Splat, track and map 3d gaussians for dense rgb-d slam.
In CVPR, 2024.
- [26]
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.
3d gaussian splatting for real-time radiance field rendering.
ACM TOG, 42(4), July 2023.
- [27]
Numair Khan, Eric Penner, Douglas Lanman, and Lei Xiao.
Tiled multiplane images for practical 3d photography.
International Conference on Computer Vision (ICCV), 2023.
- [28]
Changil Kim, Henning Zimmer, Yael Pritch, Alexander Sorkine-Hornung, and
Markus H Gross.
Scene reconstruction from high spatio-angular resolution light
fields.
ACM Trans. Graph., 32(4):73–1, 2013.
- [29]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization, 2017.
- [30]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura
Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr
Dollár, and Ross Girshick.
Segment anything.
arXiv:2304.02643, 2023.
- [31]
Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis.
Dynmf: Neural motion factorization for real-time dynamic view
synthesis with 3d gaussian splatting.
ECCV, 2024.
- [32]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.
In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors,
NeurIPS, volume 25. Curran Associates, Inc., 2012.
- [33]
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis.
Mosca: Dynamic gaussian fusion from casual videos via 4d motion
scaffolds.
arXiv preprint arXiv:2405.17421, 2024.
- [34]
Marc Levoy and Pat Hanrahan.
Light field rendering.
In Seminal Graphics Papers: Pushing the Boundaries, Volume 2,
pages 441–452. 2023.
- [35]
Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa.
Nerfacc: Efficient sampling accelerates nerfs.
arXiv preprint arXiv:2305.04966, 2023.
- [36]
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu.
Spacetime gaussian feature splatting for real-time dynamic view
synthesis.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 8508–8520, 2024.
- [37]
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.
Neural scene flow fields for space-time view synthesis of dynamic
scenes.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 6498–6508, 2021.
- [38]
Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie
Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely.
Megasam: Accurate, fast and robust structure and motion from casual
dynamic videos.
In arxiv, 2024.
- [39]
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely.
Dynibar: Neural dynamic image-based rendering.
In CVPR, 2023.
- [40]
Hanxue Liang, Jiawei Ren, Ashkan Mirzaei, Antonio Torralba, Ziwei Liu, Igor
Gilitschenski, Sanja Fidler, Cengiz Oztireli, Huan Ling, Zan Gojcic, and
Jiahui Huang.
Feed-forward bullet-time reconstruction of dynamic scenes from
monocular videos.
2024.
- [41]
Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen-Phuoc, Douglas Lanman, James
Tompkin, and Lei Xiao.
Gaufre: Gaussian deformation fields for real-time dynamic novel view
synthesis.
In 2025 IEEE/CVF Winter Conference on Applications of Computer
Vision (WACV), pages 2642–2652. IEEE, 2025.
- [42]
Yiming Liang, Tianhan Xu, and Yuta Kikuchi.
Himor: Monocular deformable gaussian reconstruction with hierarchical
motion representation, 2025.
- [43]
Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao.
Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle.
In CVPR, pages 21136–21145, 2024.
- [44]
Qingming LIU, Yuan Liu, Jiepeng Wang, Xianqiang Lyu, Peng Wang, Wenping Wang,
and Junhui Hou.
MoDGS: Dynamic gaussian splatting from casually-captured monocular
videos with depth priors.
In ICLR, 2025.
- [45]
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and
Carl Vondrick.
Zero-1-to-3: Zero-shot one image to 3d object, 2023.
- [46]
Xi Liu, Chaoyi Zhou, and Siyu Huang.
3dgs-enhancer: Enhancing unbounded 3d gaussian splatting with
view-consistent 2d diffusion priors.
In Advances in Neural Information Processing Systems (NeurIPS),
2024.
- [47]
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and
Wenping Wang.
Syncdreamer: Generating multiview-consistent images from a
single-view image.
arXiv preprint arXiv:2309.03453, 2023.
- [48]
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu,
Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al.
Wonder3d: Single image to 3d using cross-domain diffusion.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 9970–9980, 2024.
- [49]
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan.
Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.
In 3DV, 2024.
- [50]
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi
Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.
Local light field fusion: Practical view synthesis with prescriptive
sampling guidelines.
ACM Transactions on Graphics (ToG), 38(4):1–14, 2019.
- [51]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi
Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view
synthesis.
In ECCV, 2020.
- [52]
Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat
Hanrahan.
Light field photography with a hand-held plenoptic camera.
PhD thesis, Stanford university, 2005.
- [53]
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh
Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong
Dou, et al.
Holoportation: Virtual 3d teleportation in real-time.
In Proceedings of the 29th annual symposium on user interface
software and technology, pages 741–754, 2016.
- [54]
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman,
Steven M Seitz, and Ricardo Martin-Brualla.
Nerfies: Deformable neural radiance fields.
In Proceedings of the IEEE/CVF international conference on
computer vision, pages 5865–5874, 2021.
- [55]
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz,
Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.
Hypernerf: A higher-dimensional representation for topologically
varying neural radiance fields.
arXiv preprint arXiv:2106.13228, 2021.
- [56]
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz,
Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz.
Hypernerf: A higher-dimensional representation for topologically
varying neural radiance fields.
ACM TOG, 40(6), dec 2021.
- [57]
Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li,
Wim Abbeloos, and Luc Van Gool.
UniDepthV2: Universal monocular metric depth estimation made
simpler, 2025.
- [58]
Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li,
Luc Van Gool, and Fisher Yu.
UniDepth: Universal monocular metric depth estimation.
In CVPR, 2024.
- [59]
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.
D-nerf: Neural radiance fields for dynamic scenes.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 10318–10327, 2021.
- [60]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language
supervision.
In Marina Meila and Tong Zhang, editors, ICML, volume 139 of
Proceedings of Machine Learning Research, pages 8748–8763. PMLR,
18–24 Jul 2021.
- [61]
Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin
Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, and Jun Gao.
Gen3c: 3d-informed world-consistent video generation with precise
camera control.
In CVPR, 2025.
- [62]
Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and
Matthias Nießner.
Dense depth priors for neural radiance fields from sparse input
views.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2022.
- [63]
Johannes Lutz Schönberger and Jan-Michael Frahm.
Structure-from-motion revisited.
In CVPR, 2016.
- [64]
Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael
Frahm.
Pixelwise view selection for unstructured multi-view stereo.
In ECCV, 2016.
- [65]
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin
Liu.
Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic
reconstruction and rendering.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 16632–16642, 2023.
- [66]
Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari,
Gordon Wetzstein, and Leonidas Guibas.
Dynamic gaussian marbles for novel view synthesis of casual monocular
videos.
In SIGGRAPH Asia 2024 Conference Papers, pages 1–11, 2024.
- [67]
Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and
Yikai Wang.
Dimensionx: Create any 3d and 4d scenes from a single image with
controllable video diffusion.
arXiv preprint arXiv:2411.04928, 2024.
- [68]
Richard Tucker and Noah Snavely.
Single-view view synthesis with multiplane images.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2020.
- [69]
Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sargent, Ruoshi Liu, Pavel
Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick.
Generative camera dolly: Extreme monocular dynamic novel view
synthesis.
2024.
- [70]
Haiping Wang, Yuan Liu, Ziwei Liu, Zhen Dong, Wenping Wang, and Bisheng Yang.
Vistadream: Sampling multiview consistent images for single-view
scene reconstruction.
arXiv preprint arXiv:2410.16892, 2024.
- [71]
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian
Rupprecht, and David Novotny.
Vggt: Visual geometry grounded transformer.
In CVPR, 2025.
- [72]
Liao Wang, Ziyu Wang, Pei Lin, Yuheng Jiang, Xin Suo, Minye Wu, Lan Xu, and
Jingyi Yu.
ibutter: Neural interactive bullet time generator for human
free-viewpoint rendering.
In Proceedings of the 29th ACM International Conference on
Multimedia, MM ’21, page 4641–4650, New York, NY, USA, 2021. Association
for Computing Machinery.
- [73]
Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and
Angjoo Kanazawa.
Shape of motion: 4d reconstruction from a single video, 2024.
- [74]
Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and
Jiaolong Yang.
Moge: Unlocking accurate monocular geometry estimation for
open-domain images with optimal training supervision, 2024.
- [75]
Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez,
Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy.
High performance imaging using large camera arrays.
In ACM siggraph 2005 papers, pages 765–776. 2005.
- [76]
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu
Liu, Qi Tian, and Xinggang Wang.
4d gaussian splatting for real-time dynamic scene rendering.
In CVPR, pages 20310–20320, June 2024.
- [77]
Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T.
Barron, and Aleksander Holynski.
CAT4D: Create Anything in 4D with Multi-View Video Diffusion
Models.
arXiv:2411.18613, 2024.
- [78]
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel
Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and
Aleksander Holynski.
Reconfusion: 3d reconstruction with diffusion priors.
arXiv, 2023.
- [79]
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair
Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.
Neural fields in visual computing and beyond.
In Computer Graphics Forum, volume 41, pages 641–676. Wiley
Online Library, 2022.
- [80]
Jing Xiong, Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao,
Hui Shen, Zhongwei Wan, Jinfa Huang, et al.
Autoregressive models in vision: A survey.
arXiv preprint arXiv:2411.05902, 2024.
- [81]
Dejia Xul, Yifan Jiang Jiang, Chen Huang, Liangchen Song, Thorsten Gernoth,
Liangliang Cao, Zhangyang Wang, and Hao Tang.
Cavia: Camera-controllable multi-view video diffusion with
view-integrated attention.
arXiv preprint arXiv:2410, 2024.
- [82]
Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng.
Track anything: Segment anything meets videos, 2023.
- [83]
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang
Zhao.
Depth anything: Unleashing the power of large-scale unlabeled data.
In CVPR, 2024.
- [84]
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin.
Deformable 3d gaussians for high-fidelity monocular dynamic scene
reconstruction.
arXiv preprint arXiv:2309.13101, 2023.
- [85]
Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang.
Real-time photorealistic dynamic scene representation and rendering
with 4d gaussian splatting.
In ICLR, 2024.
- [86]
Xu Yinghao, Shi Zifan, Yifan Wang, Chen Hansheng, Yang Ceyuan, Peng Sida, Shen
Yujun, and Wetzstein Gordon.
Grm: Large gaussian reconstruction model for efficient 3d
reconstruction and generation, 2024.
- [87]
Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz.
Novel view synthesis of dynamic scenes with globally coherent depths
from a monocular camera.
In CVPR, June 2020.
- [88]
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun
Gao, Tien-Tsin Wong, Ying Shan, and Yonghong Tian.
Viewcrafter: Taming video diffusion models for high-fidelity novel
view synthesis.
arXiv preprint arXiv:2409.02048, 2024.
- [89]
David Junhao Zhang, Roni Paiss, Shiran Zada, Nikhil Karnad, David E Jacobs,
Yael Pritch, Inbar Mosseri, Mike Zheng Shou, Neal Wadhwa, and Nataniel Ruiz.
Recapture: Generative video camera controls for user-provided videos
using masked video fine-tuning.
arXiv preprint arXiv:2411.05003, 2024.
- [90]
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli,
and Zexiang Xu.
Gs-lrm: Large reconstruction model for 3d gaussian splatting.
In European Conference on Computer Vision, pages 1–19.
Springer, 2024.
- [91]
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.
Nerf++: Analyzing and improving neural radiance fields.
arXiv preprint arXiv:2010.07492, 2020.
- [92]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual
metric.
In CVPR, 2018.
- [93]
Sijie Zhao, Wenbo Hu, Xiaodong Cun, Yong Zhang, Xiaoyu Li, Zhe Kong, Xiangjun
Gao, Muyao Niu, and Ying Shan.
Stereocrafter: Diffusion-based generation of long and high-fidelity
stereoscopic 3d from monocular videos.
arXiv preprint arXiv:2409.07447, 2024.
- [94]
Yuyang Zhao, Chung-Ching Lin, Kevin Lin, Zhiwen Yan, Linjie Li, Zhengyuan Yang,
Jianfeng Wang, Gim Hee Lee, and Lijuan Wang.
Genxd: Generating any 3d and 4d scenes.
In ICLR, 2025.
- [95]
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.
Stereo magnification: Learning view synthesis using multiplane
images.
arXiv preprint arXiv:1805.09817, 2018.
- [96]
Bojia Zi, Shihao Zhao, Xianbiao Qi, Jianan Wang, Yukai Shi, Qianyu Chen, Bin
Liang, Kam-Fai Wong, and Lei Zhang.
Cococo: Improving text-guided video inpainting for better
consistency, controllability and compatibility.
ArXiv, abs/2403.12035, 2024.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: My main consider are the novelty of involving the video gen model to 4d reconstruction and detailed technique contribuitions.
soundness: 2
strengths: 1. the generation steps alternate with the training of a Gaussian-based global 4D representation.
2. The paper provides thorough quantitative evaluations on multiple datasets, demonstrating improvements in metrics like PSNR, SSIM, LPIPS, and tracking accuracy. The ablation study effectively analyzes the contributions of key hyperparameters, such as the number of bullet-time stamps and generations.
summary: This paper presents BulletGen, a method for 4D dynamic scene reconstruction from monocular videos. Its core contribution is the integration of static, diffusion-based bullet-time generation with dynamic 3D Gaussian Splatting to address under-constrained regions. The approach iteratively augments the scene representation at selected frozen timestamps. Evaluations on the DyCheck iPhone and Nvidia Dynamic datasets demonstrate state-of-the-art performance in novel view synthesis and 2D/3D tracking.
weaknesses: 1. A similar idea has already been employed in 3D reconstruction works like VistaDream, which also involves initializing a 3D Gaussian Splatting (3DGS) reconstruction and then iteratively inpainting it. This paper should further discuss its relationship with such literature.
2. The outputs of diffusion models often exhibit insufficient 3D coherence, which can complicate the alignment process and introduce visual artifacts, especially when handling complex dynamic scenarios.
3. The description of key technical components lacks clarity. The explanation of the generative augmentation pipeline and the loss function, while detailed, suffers from unnecessary complexity. It often fails to provide a clear intuition for design choices. For example, the iterative optimization loop and the mechanism for integrating generated views into the global 4D representation are not explained in a coherent, step-by-step manner, making the core contribution difficult to follow.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The justification for using Shape-of-Motion (SoM) as the baseline is unclear. Given that SoM itself has notable limitations, could the authors elaborate on why this particular model was chosen over other, stronger 4D reconstruction methods?
soundness: 4
strengths: 1. Strong Empirical Results: The method shows impressive quantitative and qualitative results. It achieves state-of-the-art performance on standard benchmarks for both novel-view synthesis and 2D/3D tracking, clearly outperforming its baseline (Shape-of-Motion) and other recent methods. The qualitative results for extreme novel views (e.g., Fig. 1 and Fig. 3) are particularly strong.
	2. Simple but effective strategy. A significant strength is the method's practicality. Instead of requiring a complex, computationally expensive 4D video diffusion model, the authors leverage a generator trained only on static scenes. This "bullet-time static diffusion strategy" is a clever way to augment a dynamic reconstruction while using more accessible and abundant static training data
summary: This paper presents BulletGen, a method for 4D dynamic scene reconstruction from a single monocular video. The core problem it addresses is the highly ill-posed nature of this task, particularly in reconstructing unseen regions and resolving depth ambiguities.
The authors demonstrate that this approach achieves state-of-the-art results on the DyCheck iPhone and Nvidia datasets, significantly improving novel-view synthesis quality and 2D/3D tracking accuracy.
weaknesses: 1. Limited Novelty of the Core Concept: The central idea of using diffusion models to generate novel views as supervision for a 3D/4D neural representation is not new. This concept has been substantially explored in the 3D reconstruction domain, particularly for sparse-view inputs (e.g., ReconFusion [1], Difix+[2], and others ). The paper's primary contribution is the application of this idea to the monocular 4D setting. While effective, this can be seen as an incremental, though logical, extension of existing work
	2. Unconvincing "Bullet-Time" Strategy and Temporal Consistency: My main concern lies with the "bullet-time" generation. The generative model is static and, more importantly, generates novel views for each time stamp independently. The paper's claim is that optimizing the global 4DGS representation (which uses shared motion bases ) is sufficient to enforce temporal consistency across these independently generated views.
	Other works in 4D object generation (e.g., EG4D [3], SV4D [4]) have proposed more principled solutions, such as attention-based mixing or latent-space temporal models, to explicitly enforce consistency during the generation phase. The paper lacks a discussion or comparison against such temporally-aware generative methods. The provided temporal slice (Fig. 5) only compares against the SoM baseline, which is insufficient to prove the temporal coherence of the generated content.
	
	[1] Wu et al. CVPR 2024.
	[2] Zhang et al. CVPR 2025.
	[3] Sun et al. ICLR 2025.
	[4] Xie et al. ICLR 2025.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - I'm curious about how the performance would change if this approach were applied to MoSca, a more recent method.
- Since the generation ultimately depends on the prompt, which frame was used to select the prompt? For example, I understand that the prompt can vary quite significantly across frames when objects are moving.
- Was the generative model not fine-tuned?
soundness: 3
strengths: - Generative augmentation for unobserved regions: The method employs a frozen diffusion-based image/video generator at "bullet-time" instants to hallucinate novel views (e.g., back sides, occluded areas), providing missing appearance and geometry cues. 
- Integration of 2D generative priors with global 4D scene representation: Generated 2D frames undergo pose-tracking and depth-alignment, then iteratively supervise a dynamic 3D Gaussian-splatting representation to maintain spatio-temporal consistency.
summary: - BulletGen addresses the ill-posed problem of 4D reconstruction from monocular videos by leveraging a diffusion-based video generation model to correct errors and complete missing information in Gaussian-based dynamic scene representations.
- The method aligns generated frames at specific "bullet-time" stamps with the initial 4D reconstruction to supervise and "iteratively" optimize the dynamic 4D Gaussian model using a robust loss incorporating photometric, perceptual, semantic, and depth err.
weaknesses: - Since the method ultimately relies on diffusion loss, performance improvements are inevitably limited when dealing with complex objects or large motions. This can be confirmed by the modest gains observed in SoM. (Particularly, while there is some gain shown in Table 3, the 0.06dB improvement in Table 2 is too minimal to be considered significant.)
- Furthermore, the use of CLIP loss tends to work better primarily on object-centric scenes. Consequently, as observed, the method performs better on object-centric and near-rigid scenes such as "spin" and "paper-windmill," which represents a limitation.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: - Could you generate all your novel views with the diffusion model from the input video? In this way, the prior provided to the diffusion model would be the strongest, as opposed to a degraded view produced by baseline reconstruction. It would be an interesting ablation.
- Regarding one of the weaknesses, can any diffusion be used in the pipeline?
- In higher numbers of n_g, what is the rationale of doing multiple generations the same way?
- It sounds like, given a timestep, an extreme view among input poses is selected as the prior for generation. Therefore, for some timesteps, the view will be close to the input, and for some, very far. For the far views, the prior (i.e. novel view) will highly likely contain some artefacts. This would, in turn, introduce a likely noisy supervision to the reconstruction model. Did you observe anything like that? Is there a way to mitigate that?
- Regarding time complexity, could you report a time for generating one sequence of novel views given the input image for the diffusion model?
soundness: 3
strengths: - The main strength of the method is the proposed iterative approach that allows for iterative refinement of the Gaussian scene representation. 
- The use of a novel view generation diffusion model, given a static scene, is a nice way of using a static model for dynamic reconstruction.
- The proposed alignment algorithm seems to be working well. Authors make a really good use of the state-of-the-art models for priors.
- The provided evaluation shows consistent improvement over the baseline Shape-of-Motion.
- The qualitative results align with the quantitative ones in terms of visible improvements with respect to Shape-of-Motion.
summary: The paper proposes BulletGen, a method for dynamic 3D scene reconstruction from monocular videos that leverages a generative diffusion model to enhance 4D reconstruction. The key innovation is using the static image-to-video diffusion models to generate novel views at a selected timestep, which are then aligned to the scene representation and used for supervision. The method benchmarks its performance on DyCheck and Nvidia datasets.
weaknesses: - A big part of the paper's contribution is dependent on the 'internal controllable image-to-video diffusion model'. This raises several concerns. Firstly, the reproducibility of the method will be largely limited unless the model is released. Any further comparison with this method will highly likely not be feasible. While it is fair to use an internal model to achieve a good performance, in my opinion, it should be accompanied by a detailed comparison of the same pipeline but with a publicly available diffusion model. This is particularly important given the fact that authors use ViewCrafter in their experiments. There is nothing in the paper suggesting that using ViewCrafter would not be feasible in this setup. In the current state, it is not clear how much of the performance improvement comes from the diffusion model.
- The novelty of the paper is more limited than the authors mention; several papers with a rather similar approach are not mentioned, and the evaluation is missing important comparisons.
	- Regarding novelty - I believe two important references are missing. Firstly, Difix3D+ [1] is an important work that proposed an iterative refinement process of the 3D scene with the use of a generalisable enhancement diffusion model. Not only does this work propose a significant method in part similar to this paper's contributions, but the diffusion model is available and could be used in an ablation study. Further, ViDAR [2] proposes a reconstruction method in which the novel views are generated and further enhanced with personalised diffusion to serve as the reconstruction supervision. This work uses a similar idea of generating an additional supervision signal in novel views and seems highly relevant as a related work. 
	- Regarding evaluation - the authors cite MoSca in line 133; the evaluation should include MoSca as the compared method. It looks like MoSca would outperform BulletGen in some metrics (PSNR, SSIM). Further, the aforementioned ViDAR could be included as well (performing better in PSNR, SSIM, LPIPS). Whilst the work recently got accepted to NeurIPS, and it seems not to have released the code yet, the arXiv release includes the numerical results on DyCheck in the same setup as in this paper, which makes it the same comparison as the Vivid4D used here.
- In terms of ablation, it would be good to see isolation of contributions, i.e., add contributions to Shape-of-Motion one by one to show the reader the importance of each. This would help with the previously raised point of not being able to isolate the impact of the internal model.  
- To strengthen the claim on the new synthesised plausible parts of the scene, one could show that both quantitatively and qualitatively. Namely, given that DyCheck provides covisibility masks, presenting results outside of such masks would effectively measure the performance of the approach in unseen parts of the scene.
- CAT4D is currently way past the date of being a concurrent work; it was published at CVPR 2025 and released on arXiv even earlier. 
- It would be good to see examples of the captions and whether they differ between times and views.
	
[1] Jay Zhangjie Wu, Yuxuan Zhang, Haithem Turki, Xuanchi Ren, Jun Gao, Mike Zheng Shou, Sanja Fidler, Zan Gojcic, Huan Ling, *Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models*, CVPR 2025

[2] Michal Nazarczuk, Sibi Catley-Chandar, Thomas Tanay, Zhensong Zhang, Gregory Slabaugh, Eduardo Pérez-Pellitero, *ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs*, NeurIPS 2025