StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams

## Abstract

Abstract Real-time reconstruction of dynamic 3D scenes from uncalibrated video streams is crucial for numerous real-world applications.
However, existing methods struggle to jointly address three key challenges: 1) processing uncalibrated inputs in real time, 2) accurately modeling dynamic scene evolution, and 3) maintaining long-term stability and computational efficiency.
To this end, we introduce StreamSplat , the first fully feed-forward framework that transforms uncalibrated video streams of arbitrary length into dynamic 3D Gaussian Splatting (3DGS) representations in an online manner, capable of recovering scene dynamics from temporally local observations.
We propose two key technical innovations: a probabilistic sampling mechanism in the static encoder for 3DGS position prediction, and a bidirectional deformation field in the dynamic decoder that enables robust and efficient dynamic modeling.
Extensive experiments on static and dynamic benchmarks demonstrate that StreamSplat consistently outperforms prior works in both reconstruction quality and dynamic scene modeling, while uniquely supporting online reconstruction of arbitrarily long video streams.
Code and models are available at https://github.com/nickwzk/StreamSplat .

## 1 Introduction

Real-time reconstruction of dynamic 3D scenes from video streams is crucial for numerous applications, *e.g*., robotics [1, 2], augmented/virtual reality (AR/VR) [3, 4], and autonomous driving [5].
AR/VR systems rely on accurate, continuously updated 3D models to deliver immersive user experiences, while robots and autonomous vehicles require real-time representations of dynamic environments for safe navigation and responsive interactions.
These applications demand robust online reconstruction systems capable of accurately capturing evolving scene geometry and appearance.

Despite growing research interest, real-time dynamic 3D reconstruction from uncalibrated video remains largely unsolved.
Although SLAM-based [6, 7, 8, 9] and scene-coordinate-based methods [10, 11, 12, 13] could effectively estimate camera poses and reconstruct static scenes, their extension to dynamic scenes is often hindered by the challenge of disentangling camera and object motion from uncalibrated inputs.
As a result, they often require computationally intensive post-optimization of camera poses or scene representations [14, 6], limiting their real-time applicability.
Meanwhile, feed-forward methods [15, 16, 17] are promising for static scene reconstruction from uncalibrated inputs.
Recent work [18] has begun to explore their potential for dynamic scene reconstruction.
However, such methods typically treat dynamic scenes as a sequence of static ones, ignoring temporal coherence and failing to explicitly model scene dynamics.
Moreover, many require access to entire video sequences during inference [18, 19], making them unsuitable for online streaming scenarios.

Building such an online 3D reconstruction system for dynamic scenes introduces three fundamental challenges:
(1) Real-Time Processing. Latency-sensitive tasks [1, 5] require immediate 3D reconstruction.
However, traditional methods often rely on offline calibration such as Structure-from-Motion (SfM) [20] or iterative optimization of scene representations [21, 22], making them unsuitable for real-time processing of uncalibrated video streams [23, 24, 25];
(2) Dynamic Motion Modeling.
In uncalibrated videos, camera and object motions are almost always entangled, and dynamic scenes exhibit complex, non-rigid motions that must be inferred from video frames [26, 27, 28];
(3) Long-Term Stability.
Online systems must process arbitrarily long video streams while preventing error accumulation and maintaining efficiency in both memory and computation [17].

To address these challenges, we propose StreamSplat, the first fully feed-forward framework for online dynamic 3D Gaussian Splatting (3DGS) [21, 29] reconstruction from uncalibrated video streams.
Inspired by the recent success of feed-forward Gaussian Splatting [15], we leverage pixel-aligned 3D Gaussians in an orthographic canonical space [26, 30] to support online, dynamic, and uncalibrated reconstruction.
We formulate online video reconstruction as a sequential, frame-by-frame process.
First, we encode each incoming frame into a set of pixel-aligned static 3D Gaussians in a probabilistic way.
This formulation is specifically designed for 3DGS position prediction (Section [3.1](https://arxiv.org/html/2506.08862v1#S3.SS1)), alleviating positional local minima caused by random initialization and better capturing uncertainty.

To model dynamic motion, we propose a novel bidirectional deformation field coupled with an adaptive Gaussian fusion mechanism.
These components enable smooth transitions and seamless fusion of Gaussians across frames by softly matching spatially and temporally coherent structures, while dynamically integrating newly observed content (Section [3.2](https://arxiv.org/html/2506.08862v1#S3.SS2)).
The bidirectional formulation ensures that dynamic Gaussians remain structured and pixel-aligned at key frames, improving robustness and mitigating the error accumulation common in long-term online reconstruction [17].
We evaluate StreamSplat on both static (CO3Dv2 [31], RealEstate10K [32]) and dynamic (DAVIS [33], YouTube-VOS [34]) benchmarks. Our method consistently outperforms prior works in both reconstruction quality and dynamic scene modeling, while uniquely supporting fully online reconstruction of arbitrarily long video streams (Section [4](https://arxiv.org/html/2506.08862v1#S4)).
In summary, our key contributions are:

- •
We propose StreamSplat, the first fully feed-forward framework for online dynamic 3DGS reconstruction from uncalibrated video streams.
- •
We introduce two key technical innovations: a probabilistic sampling mechanism in the static encoder for 3DGS position prediction, and a bidirectional deformation field in the dynamic decoder that enables robust and efficient dynamic modeling.
- •
StreamSplat achieves state-of-the-art reconstruction quality and dynamic scene modeling across multiple static and dynamic benchmarks, while uniquely supporting online reconstruction of arbitrarily long uncalibrated video streams.

## 2 Related Works

### 2.1 Neural Representations for Dynamic Scenes

Dynamic 3D reconstruction from monocular videos is critical for many real-world applications.
Early works leveraged implicit neural representations and modeled dynamic scenes using coordinate-based multilayer perceptrons (MLPs) optimized through per-image optimization [35, 36, 37, 38]. Later works extended by introducing learnable time-conditioned deformation fields with canonical representations [39, 40, 41, 42]. However, these methods typically fail to capture large and complex motions due to lack of explicit 3D structure and motions.
To address this, recent work shifts to explicit 3D primitives, notably dynamic Gaussian splatting [43, 44, 45, 46, 30, 47, 48], which leverages persistent and deformable Gaussians to efficiently represent dynamic scenes for real-time rendering. However, these methods still rely on calibrated camera poses and extensive per-scene optimization, making them unsuitable for fully uncalibrated and real-time scenarios.

### 2.2 Feed - Forward Dynamic 3D Reconstruction

By directly predicting 3D scene representations via neural networks, feed-forward methods have emerged as promising alternatives to optimization-based approaches. For example, based on DUSt3R [11], several approaches have shown the ability to estimate camera parameters and 3D point clouds from uncalibrated inputs [13, 49, 50, 12, 51, 52, 53].
MonST3R [14] extends this paradigm to dynamic scenes by producing temporally coherent point clouds and camera poses across frames.
Such methods typically require post-optimization on camera poses or scene geometry to maintain consistency in dynamic scenarios.
Moreover, scene-coordinate-based point clouds are inherently difficult to perform deformations, limiting their effectiveness in dynamic scene modeling [43, 45].

Another research line explores fully feed-forward 3D Gaussian Splatting (3DGS) methods for 3D reconstruction [54, 55, 56, 57, 58, 59, 60, 61, 62]. Prior works achieve fast static scene reconstruction from images pairs. Specifically, pixelSplat [15] employs multi-view transformers to regress 3D Gaussians from calibrated image pairs, while NoPoSplat [16] extends this to uncalibrated pairs, and StreamGS [17] achieves online static reconstruction from image streams. Recent works such as BTimer [18] further adapt these approaches to dynamic scene reconstruction by stacking multiple temporal-consistent static 3DGS from calibrated video frames. Despite their efficiency, these methods either primarily address static scenes or approximate dynamics by stacking static representations, which fail to explicitly model scene dynamics. Moreover, many require access to entire video sequences during inference [18, 19], making them unsuitable for online streaming scenarios.

## 3 Method

Figure: Figure 2: Overview of the StreamSplat framework. Given a pair of frames, we first encode them using the Static Encoder to produce canonical 3D Gaussians (Section [3.1](https://arxiv.org/html/2506.08862v1#S3.SS1)), and then pass the 3DGS Embeddings to the Dynamic Decoder to predict the deformation field (Section [3.2](https://arxiv.org/html/2506.08862v1#S3.SS2)). The resulting dynamic 3D Gaussians can be rendered at arbitrary time to produce RGB images and depth maps.
Refer to caption: x2.png

In this section, we present StreamSplat, a framework designed to instantly transforms uncalibrated online video streams into dynamic 3D Gaussian Splatting (3DGS) representations capable of capturing scene dynamics.
Figure [2](https://arxiv.org/html/2506.08862v1#S3.F2) provides an overview. We first encode the current frame into static 3D Gaussians in a canonical space (Section [3.1](https://arxiv.org/html/2506.08862v1#S3.SS1)), then predict a bidirectional deformation field between Gaussians encoded from current frame and propagated from the previous frame, and finally fuse them into a unified dynamic representation (Section [3.2](https://arxiv.org/html/2506.08862v1#S3.SS2)).
This representation supports rendering at arbitrary times and viewpoints, thereby effectively recovering scene dynamics (Section [3.3](https://arxiv.org/html/2506.08862v1#S3.SS3)).

### 3.1 Probabilistic 3D Gaussian Encoding

Canonical 3D Space.
Following [26, 30, 19], we define the canonical space using a shared orthographic camera coordinate system. Details are provided in Appendix [A](https://arxiv.org/html/2506.08862v1#A1).
We argue that this choice offers two key advantages. First, under orthographic projection, scene dynamics that typically entangle camera and object motion manifest as a shared global shift, which can be naturally absorbed by Gaussian motions and handled by our dynamic prediction module in an unified manner (Section [3.2](https://arxiv.org/html/2506.08862v1#S3.SS2)). This simplification allows us to bypass errors from inaccurate camera estimation.
Second, orthographic projection simplifies the camera model while still preserving the spatio-temporal structure of 3D motion [26].
Together, this orthographic canonical space provides a simplified, yet effective foundation for predicting dynamic 3D representations directly from uncalibrated videos.

Structured Static 3D Gaussian Encoding.
To overcome the inherent depth ambiguity under orthographic projection [63] and unstructured nature of 3DGS [64], we incorporate a pretrained depth estimator [65] to generate per-frame *pseudo-depth* maps as ground truth and predict 3D Gaussian positions in a pixel-aligned manner [15] to maintain spatial correspondence with input frames.

As shown in Figure [2](https://arxiv.org/html/2506.08862v1#S3.F2), given an RGB image $I\in\mathbb{R}^{H\times W\times 3}$, we first obtain the corresponding depth map and concatenate it with $I$ to form a 4-channel RGBD input. This is processed by a non-overlapping 2D convolution to extract $8\times 8$ patches, which are then fed into a Transformer encoder with self-attention and MLP blocks [66, 67], producing 3DGS Embeddings $\mathbf{h}\in\mathbb{R}^{N\times D}$, where $N=HW/64$. These embeddings are upsampled via a lightweight block composed of linear layers and window-based attention [68, 69], yielding static Gaussian tokens $\hat{\mathbf{E}}\in\mathbb{R}^{16N\times D/4}$. Each token $\hat{\mathbf{E}}_{i}$ is decoded by linear heads into 3DGS parameters: position offset $\boldsymbol{o}_{i}\in\mathbb{R}^{3}$, rotation $\mathbf{R}_{i}\in\mathbb{R}^{4}$, scale $\mathbf{S}_{i}\in\mathbb{R}^{3}$, opacity $\boldsymbol{\alpha}_{i}\in\mathbb{R}$, and color $\boldsymbol{c}_{i}\in\mathbb{R}^{3}$. The final 3D position $\boldsymbol{\mu}_{i}$ is computed via pixel-aligned prediction: $\boldsymbol{\mu}_{i}=\left(u+\boldsymbol{o}_{i,0},v+\boldsymbol{o}_{i,1},1/
\boldsymbol{o}_{i,2}\right)$, where $(u,v)$ is the pixel coordinate of the $i$-th token. The first two offset components $\boldsymbol{o}_{i,0}$, $\boldsymbol{o}_{i,1}$ specify the local offset within the $2\times 2$ image patch, while $\boldsymbol{o}_{i,2}$ denotes the inverse depth following [65] for a better depth estimation near the camera.

Probabilistic Position Sampling.
3D Gaussian Splatting is sensitive to position initialization [21] and prone to local minima [15], especially in feed-forward models [15, 58] that make predictions in a single forward pass without iterative refinement.
Inspired by [15], we predict a truncated normal distribution for each 3D offset $\boldsymbol{o}$ rather than regressing it directly: $\boldsymbol{o}\sim\mathcal{N}_{[-1,1]}(\boldsymbol{\mu}_{p},\boldsymbol{\Sigma
}_{p})$, where $\boldsymbol{\mu}_{p}$ and $\boldsymbol{\Sigma}_{p}$ are the predicted mean and covariance. As shown in Section [4.4](https://arxiv.org/html/2506.08862v1#S4.SS4), this strategy promotes spatial exploration during early training and stabilizes convergence toward optimal positions.

### 3.2 Dynamic Deformation Prediction

Bidirectional Deformation Field.
Online dynamic scene reconstruction involves complex motion, including large deformations and emerging objects/surfaces [17]. To handle this, we propose a bidirectional deformation field that jointly models forward and backward temporal motion between consecutive frames.
Specifically, our model estimates: (i) a forward deformation field that transforms the current-frame Gaussians $\mathcal{G}_{t}$ toward the next frame, and (ii) a backward deformation field that transforms next-frame Gaussians $\mathcal{G}_{t+1}$ back to the current frame.

Each deformation field consists of a 3D velocity vector $\mathbf{v}\in[-1,1]^{3}$ and an opacity coefficient $\boldsymbol{\gamma}$ that controls visibility over time. This bidirectional design naturally handles the appearance and disappearance of Gaussians without explicitly managing a varying number of Gaussians, thereby facilitating end-to-end training and online updates during inference.

Adaptive Gaussian Fusion via Soft Matching.
Directly combining new Gaussians often leads to spatial overlap and redundancy [17]. Traditional optimization-based methods resolve this by rigid one-to-one matching and iterative fusion [47], which are computationally expensive and hard to keep spatial structures, potentially resulting in long-term accumulation of errors.

Inspired by defining the life-cycle of Gaussians [70], we propose an adaptive Gaussian fusion mechanism based on opacity deformation. Each two-frame interval is normalized to $t\in[0,1]$, with $t=0$ and $t=1$ representing the previous and current frames, respectively. Each Gaussian persists across two consecutive frames, with a time-dependent opacity deformation:

$$ $\boldsymbol{\alpha}(t)=\boldsymbol{\alpha}\cdot\frac{\sigma\left(-\boldsymbol{ \gamma}_{0}\left(|t-t_{0}|-\boldsymbol{\gamma}_{1}\right)\right)}{\sigma\left( \boldsymbol{\gamma}_{0}\cdot\boldsymbol{\gamma}_{1}\right)},$ (1) $$

where $\sigma(\cdot)$ denotes sigmoid function, $\boldsymbol{\alpha}$ denotes initial opacity, $t_{0}\in\{0,1\}$ denotes the Gaussian’s creation frame, $\boldsymbol{\gamma}_{0}\in\mathbb{R}^{+}$ and $\boldsymbol{\gamma}_{1}\in[0,1]$ control the transition rate and fade-out timing.
This allows overlapped Gaussians from adjacent frames to be implicitly merged through opacity deformation, and the rendering loss can naturally drive the model to find a soft match between them.

Dynamic Deformation Decoding.
Given two consecutive frames from a randomly sampled time interval, $I_{t_{1}}$ and $I_{t_{2}}$, we first use the frozen static encoder to extract their 3DGS Embeddings $\mathbf{h}_{t_{1}}$ and $\mathbf{h}_{t_{2}}$. We also extract DINOv2 features [71] $\mathbf{f}_{t_{1}}$ and $\mathbf{f}_{t_{2}}$ from $I_{t_{1}}$ and $I_{t_{2}}$, respectively.
To distinguish the embeddings, we add learnable Type Embeddings $\mathbf{T}_{src},\mathbf{T}_{tgt}\in\mathbb{R}^{D}$ to $\mathbf{h}_{t_{1}}$ and $\mathbf{h}_{t_{2}}$, yielding $\hat{\mathbf{h}}_{t_{1}}$ and $\hat{\mathbf{h}}_{t_{2}}$. These are processed by the decoder as follows:

$$ $\displaystyle=\text{Self-Attn}([\hat{\mathbf{h}}_{t_{1}},\hat{\mathbf{h}}_{t_{ 2}}]),$ (2) $\displaystyle\hat{\mathbf{h}}_{t_{1}}$ $\displaystyle=\text{Cross-Attn}(\hat{\mathbf{h}}_{t_{1}},\mathbf{f}_{t_{2}}), \ \hat{\mathbf{h}}_{t_{2}}=\text{Cross-Attn}(\hat{\mathbf{h}}_{t_{2}},\mathbf{ f}_{t_{1}})$ $\displaystyle[\hat{\mathbf{h}}_{t_{1}},\hat{\mathbf{h}}_{t_{2}}]$ $\displaystyle=\text{FFN}([\hat{\mathbf{h}}_{t_{1}},\hat{\mathbf{h}}_{t_{2}}]),$ $$

where $[\cdot]$ denotes concatenation. After passing through a few decoder blocks, we obtain the Deformation Embeddings $[\mathbf{d}_{t_{1}},\mathbf{d}_{t_{2}}]\in\mathbb{R}^{2N\times D}$, which are then upsampled via the same upsampler to produce deformation tokens $\hat{\mathbf{d}}\in\mathbb{R}^{32N\times D/4}$.
Each deformation token $\hat{\mathbf{d}}_{j}$ is concatenated with the corresponding static Gaussian token from the same frame $\hat{\mathbf{E}}_{j}$ to form a joint token $\hat{\mathbf{E}}_{j}\oplus\hat{\mathbf{d}}_{j}$, which is passed through a 2-layer MLP head to predict the deformation field, including velocity $\mathbf{v}_{j}$ and opacity coefficient $\boldsymbol{\gamma}_{j}$.
These deformation fields allow Gaussians to move and fade over time, enabling computation of dynamic 3D Gaussians at arbitrary times for continuous scene reconstruction.

### 3.3 Training and Inference

Robust Stage-wise Training.
To address the difficulty of jointly optimizing static 3D Gaussian encoding and dynamic deformation prediction, we adopt a two-stage training protocol:

- •
Stage 1: Static 3DGS Encoder Training.
The static encoder aims to reconstruct static 3DGS from a single input frame $I_{t}$ and pseudo-depth $D_{t}$.
It produces 3DGS primitives that are used to render RGB $\hat{I}_{t}$ and depth $\hat{D}_{t}$.
The training loss combines photometric and depth supervision:
$\mathcal{L}_{\text{static}}=\mathcal{L}_{\text{recon}}(\hat{I}_{t},I_{t})+
\lambda_{\text{depth}}\mathcal{L}_{\text{depth}}(\hat{D}_{t},D_{t}),$
(3)
where $\mathcal{L}_{\text{recon}}$ is the sum of $L_{2}$ loss (RGB space) and LPIPS loss [72], and $\mathcal{L}_{\text{depth}}$ is a scale- and shift-invariant depth loss [73]:
$\mathcal{L}_{\text{depth}}=\mathbb{E}\|\tau(\hat{D}_{t})-\tau(D_{t})\|,\quad
\text{where}\quad\tau(\mathbf{x})=\frac{\mathbf{x}-\mathrm{median}(\mathbf{x})
}{\mathbb{E}\|\mathbf{x}-\mathrm{median}(\mathbf{x})\|}.$
(4)
To reduce the impact of noisy pseudo-depth, we introduce an adaptive decay factor into the depth loss weight. Specifically, we define the effective weight as $\hat{\lambda}_{\text{depth}}=\lambda_{\text{depth}}\cdot\sigma(-\|\tau(\hat{D}
_{t})-\tau(D_{t})\|/w)$, where $\lambda_{\text{depth}}$ is a fixed hyperparameter, and $w$ controls the sensitivity of the sigmoid-based decay. We use $\hat{\lambda}_{\text{depth}}$ instead of $\lambda_{\text{depth}}$ to reduce unreliable supervision and improve robustness.
- •
Stage 2: Dynamic Deformation Decoder Training.
With the encoder frozen, the dynamic decoder learns to predict the bidirectional deformation fields. Given two randomly sampled frames $I_{t_{1}}$ and $I_{t_{2}}$, we first encode them to obtain static 3DGS $\mathcal{G}_{t_{1}}$ and $\mathcal{G}_{t_{2}}$. Then we use the dynamic decoder to predict the deformation fields that transform $\mathcal{G}_{t_{1}}\to I_{t_{2}}$ and $\mathcal{G}_{t_{2}}\to I_{t_{1}}$, followed by adaptive fusion. The training objective is to reconstruct $I_{t}$ for all intermediate time $t\in[t_{1},t_{2}]$:
$\mathcal{L}_{\text{dynamic}}=\mathbb{E}_{t}\left[\mathcal{L}_{\text{recon}}(
\hat{I}_{t},I_{t})+\lambda_{\text{depth}}\mathcal{L}_{\text{depth}}(\hat{D}_{t
},D_{t})+\lambda_{\text{mask}}\mathcal{L}_{\text{mask}}(\hat{I}_{t}\odot M_{t}
,I_{t}\odot M_{t})\right],$
(5)
where $\mathcal{L}_{\text{mask}}$ is an auxiliary reconstruction loss that encourages the model to focus on moving foreground regions using a binary segmentation mask $M_{t}$ from datasets [33, 34].

Online Inference Pipeline.
As illustrated in Figure [2](https://arxiv.org/html/2506.08862v1#S3.F2), our StreamSplat can instantly process an incoming video stream $(I_{0},I_{1},\dots,I_{N})$ frame-by-frame. Let $\mathbf{h}_{t-1}$ and $\mathbf{f}_{t-1}$ denote the 3DGS Embedding and DINO feature from the previous frame (initialized as zero for $t=0$). When a new frame $I_{t}$ arrives, we estimate its pseudo-depth $D_{t}$ [65] and encode it into 3DGS Embedding $\mathbf{h}_{t}$. These 3DGS Embeddings and DINO features are passed to the dynamic decoder to predict a bidirectional deformation field. The resulting field is applied to both $\mathcal{G}_{t-1}$ and $\mathcal{G}_{t}$, which are fused and rendered from arbitrary viewpoints and interpolated times. Finally, $\mathbf{h}_{t}$ and $\mathbf{f}_{t}$ are cached for the next step.

## 4 Experiments

Figure: Figure 3: Qualitative results on RE10K. StreamSplat produces detailed and consistent reconstructions across diverse scenes. Videos are provided on the website.
Refer to caption: x3.png

### 4.1 Experimental Settings

Training Datasets.
We pre-train StreamSplat on a combination of real-world datasets, including static scenes from CO3Dv2 [31] and RealEstate10K (RE10K) [32], and dynamic video datasets DAVIS [33] and YouTube-VOS [34]. CO3Dv2 and RE10K are treated as pure video datasets without using any pre-calibrated camera information.
For dynamic scenes, we utilize object segmentation masks from DAVIS and YouTube-VOS to supervise motion-aware components, and no mask supervision is applied to CO3Dv2 and RE10K.
We follow the official train/validation splits and train only on the training sets. The same pre-trained model is evaluated across both static and dynamic benchmarks.

Implementation Details.
StreamSplat is trained on 8 NVIDIA A100 GPUs for approximately 3 days. We use FlashAttention-2 [74], gradient checkpointing [63], and mixed-precision training with BF16 for better efficiency. Input frames are resized to $512\times 288$ to preserve aspect ratio for pixel-aligned 3DGS prediction. We apply image-level augmentations following EDM [75] and optimize using AdamW [76] with gradient clipping set to 1.0.
For Stage 1, we use a batch size of 128, a peak learning rate of $5\times 10^{-4}$ with 20K linear warm-up iterations, and weight decay of 0.05.
For Stage 2, we use a batch size of 256 (with gradient accumulation), a peak learning rate of $1\times 10^{-4}$ with 100K linear warm-up iterations, and weight decay of 0.05.
Additional training details, including hyperparameters and model configurations, are provided in the Appendix [B](https://arxiv.org/html/2506.08862v1#A2).

Figure: Figure 4: Qualitative comparison on DAVIS. Blue box: given frames; Red box: interpolated frames. Our StreamSplat produces high-fidelity and temporal coherent interpolations across both (a) 5-frame and (b) 8-frame interval tasks. Videos are provided on the website.
Refer to caption: x4.png

Evaluation Settings.
We follow prior works [15, 28] and report peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) [77], and LPIPS [72], all evaluated at a resolution of $256\times 256$ for fair comparison [15, 28].
For static scene reconstruction, we follow [15], using two randomly sampled input views with at least $60\%$ overlap and five target views sampled between them.
For dynamic scene reconstruction, we first evaluate on full video sequences. To further assess dynamic modeling, we follow [28] and evaluate video interpolation on subsampled sequences with 5-frame and 8-frame intervals and use intermediate frames for metric computation.
Our website provide video results across all four datasets, including extended demonstrations on long videos.

**Table 1: Quantitative results on RE10K. We report results for 2 given views and 5 novel views.**
| Method | Rep.<br>Type | Given View | Novel View | Average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR^↑ | SSIM^↑ | LPIPS^↓ | PSNR^↑ | SSIM^↑ | LPIPS^↓ | PSNR^↑ | SSIM^↑ | LPIPS^↓ |  |  |
| pixelSplat [15] | Stat. | 30.70 | 0.952 | 0.055 | 28.31 | 0.905 | 0.097 | 28.99 | 0.918 | 0.085 |
| MVSplat [55] | 31.48 | 0.962 | 0.046 | 28.48 | 0.909 | 0.091 | 29.34 | 0.924 | 0.078 |  |
| NoPoSplat [16] | 29.50 | 0.939 | 0.069 | 28.65 | 0.913 | 0.096 | 28.90 | 0.920 | 0.089 |  |
| CoDeF [35] | Dyn. | 35.13 | 0.943 | 0.091 | 20.51 | 0.591 | 0.402 | 21.77 | 0.625 | 0.374 |
| DGMarbles [78] | 27.48 | 0.867 | 0.232 | 23.40 | 0.727 | 0.333 | 23.73 | 0.738 | 0.325 |  |
| StreamSplat (Ours) | 41.60 | 0.992 | 0.010 | 24.68 | 0.777 | 0.167 | 29.51 | 0.839 | 0.122 |  |

**Table 2: Quantitative results on DAVIS. ^† denotes results reported in the original papers.**
| Method | Scene<br>Rep. | Key Frames | Middle-4 Frames | Time |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR^↑ | SSIM^↑ | LPIPS^↓ | PSNR^↑ | SSIM^↑ | LPIPS^↓ |  |  |  |
| Omnimotion^† [26] | NeRF | 24.11 | 0.714 | 0.371 | – | – | – | > 8 hrs |
| RoDynRF^† [79] | NeRF | 24.79 | 0.723 | 0.394 | – | – | – | > 24 hrs |
| CoDeF [35] | NeRF | 31.49 | 0.939 | 0.088 | 19.40 | 0.498 | 0.400 | ~10 mins |
| MonST3R [14] | Points | 42.33 | 0.980 | 0.012 | – | – | – | ~30 s |
| 4DGS^† [45] | 3DGS | 18.12 | 0.573 | 0.513 | – | – | – | ~40 mins |
| Splater a Video^† [30] | 3DGS | 28.63 | 0.837 | 0.228 | – | – | – | ~30 mins |
| DGMarbles [78] | 3DGS | 28.38 | 0.879 | 0.172 | 21.33 | 0.619 | 0.313 | ~30 mins |
| StreamSplat (Ours) | 3DGS | 37.83 | 0.982 | 0.016 | 23.66 | 0.684 | 0.193 | 1.48 s |

### 4.2 Static Scene Reconstruction

We evaluate StreamSplat on the RE10K benchmark and compare it with recent static and dynamic reconstruction methods. Results on CO3Dv2 are in Appendix [C](https://arxiv.org/html/2506.08862v1#A3).
For dynamic methods without camera pose input, novel views are rendered using relative timestamps.
Quantitative and qualitative results are presented in Table [1](https://arxiv.org/html/2506.08862v1#S4.T1) and Figures [3](https://arxiv.org/html/2506.08862v1#S4.F3) and [7](https://arxiv.org/html/2506.08862v1#A1.F7). StreamSplat significantly outperforms all static baselines on the given-view reconstruction task. However, on novel-view reconstruction, static methods still lead, although StreamSplat achieves the best performance among all dynamic approaches.

According to our qualitative comparison in Figures [3](https://arxiv.org/html/2506.08862v1#S4.F3) and [7](https://arxiv.org/html/2506.08862v1#A1.F7), we attribute this to the absence of accurate camera poses input in our model. Our StreamSplat assumes smooth camera motion between two input views and relies solely on relative timestamps for novel view synthesis, which can cause slight misalignments. Furthermore, unlike static models that assume a fixed scene, StreamSplat is designed to model scene dynamics, which may introduce motion artifacts that hinder performance under static benchmarks.
Despite these challenges, our StreamSplat achieves competitive average performance across all test views and consistently surpasses all dynamic baselines in every evaluation setting.

### 4.3 Dynamic Scene Reconstruction

Figure: Figure 5: Visualization of reconstructed dynamic scene from canonical and novel views. Our method captures consistent 3D motion over time, enabling faithful reconstruction at arbitrary time and viewpoints. Videos are provided on the website.
Refer to caption: x5.png

**Table 3: Quantitative results on the DAVIS-7 [28] with 8-frame interval.**
| Method | Type | PSNR^↑ | SSIM^↑ | LPIPS^↓ |
| --- | --- | --- | --- | --- |
| AMT [80] | pixel | 21.09 | 0.544 | 0.254 |
| RIFE [81] | 20.48 | 0.511 | 0.258 |  |
| FILM [82] | 20.71 | 0.528 | 0.270 |  |
| LDMVFI [83] | 19.98 | 0.479 | 0.276 |  |
| VIDIM [28] | 19.62 | 0.470 | 0.257 |  |
| CoDeF [35] | 3D | 20.34 | 0.520 | 0.365 |
| DGMarbles [78] | 19.83 | 0.548 | 0.353 |  |
| Ours | 22.10 | 0.613 | 0.234 |  |

We evaluate StreamSplat on the DAVIS benchmark and compare it with state-of-the-art methods, and report the main results in Table [2](https://arxiv.org/html/2506.08862v1#S4.T2).
Notably, StreamSplat is the only method capable of near real-time dynamic 3D reconstruction per frame.

For key-frame reconstruction task, StreamSplat achieves performance competitive with the state-of-the-art scene-coordinate-based method MonST3R [14], which represents the dynamic scenes as sequences of static 3D point clouds. However, MonST3R requires extensive post-optimization and is limited to key-frame reconstruction. In contrast, StreamSplat operates in near real time and explicitly models the scene dynamics, enabling reconstruction of intermediate frames across substantial temporal gaps.

We further evaluate dynamic modeling performance under 5-frame and 8-frame interval settings, comparing against optimization-based 3D reconstruction methods and pixel-level video-interpolation methods. As shown in Table [3](https://arxiv.org/html/2506.08862v1#S4.T3) and Figures [4](https://arxiv.org/html/2506.08862v1#S4.F4) and [8](https://arxiv.org/html/2506.08862v1#A2.F8), StreamSplat consistently outperforms all baselines, including video-interpolation methods which lack explicit 3D modeling. This highlights the effectiveness of our approach in modeling temporally coherent dynamic scenes.
Moreover, as illustrated in Figures [4](https://arxiv.org/html/2506.08862v1#S4.F4) and [8](https://arxiv.org/html/2506.08862v1#A2.F8), StreamSplat maintains high visual fidelity even under challenging scenarios such as large camera motion and reflective surfaces, where other methods often fail.

In addition, we demonstrate the robustness of StreamSplat in long-range dynamic modeling. As shown in Figure [5](https://arxiv.org/html/2506.08862v1#S4.F5), our method maintains coherent 3D structure and appearance across large spatio-temporal distances, from both canonical and novel viewpoints.

Figure: Figure 6: Ablation. w/o sampling: deterministic position prediction; w/o depth: no depth supervision.
Refer to caption: x6.png

### 4.4 Ablation Studies

**Table 4: Component-wise ablations on key and intermediate frames.**
| Variants | Frame | PSNR^↑ | SSIM^↑ | LPIPS^↓ |
| --- | --- | --- | --- | --- |
| w/o Sampling | Key | 31.47 | 0.946 | 0.073 |
| w/o Depth | 36.68 | 0.975 | 0.039 |  |
| Full (Ours) | 37.83 | 0.982 | 0.016 |  |
| w/o Bi-Deform. | Mid. | 18.89 | 0.582 | 0.492 |
| Full (Ours) | 23.66 | 0.684 | 0.193 |  |

We conduct ablation studies to evaluate the contribution of each key component in StreamSplat. Quantitative and qualitative results are presented in Table [4](https://arxiv.org/html/2506.08862v1#S4.T4) and Figures [6](https://arxiv.org/html/2506.08862v1#S4.F6) and [9](https://arxiv.org/html/2506.08862v1#A2.F9).

Ablation on probabilistic position sampling.
We evaluate the impact of probabilistic position sampling for 3DGS by comparing it with a deterministic counterpart.
As shown in Table [4](https://arxiv.org/html/2506.08862v1#S4.T4), probabilistic sampling yields a substantial improvement (6.36dB) in PSNR for key-frame reconstruction.
Qualitatively, the deterministic variant is prone to local minima, particularly along the depth axis, resulting in blurry and inaccurate reconstructions. This aligns with findings from prior work [21, 15] and highlights the importance of probabilistic position prediction in feed-forward 3DGS models.

Ablation on pseudo depth supervision.
We evaluate the impact of removing pseudo depth supervision on key-frame reconstruction. Table [4](https://arxiv.org/html/2506.08862v1#S4.T4) shows that removing depth supervision leads to only a minor drop in reconstruction quality. However, as shown in Figure [6](https://arxiv.org/html/2506.08862v1#S4.F6), the model without depth supervision fails to capture accurate spatial structure. The learned depth becomes entangled with RGB values, resulting in distorted 3D reconstructions.

Ablation on bidirectional deformation field.
We evaluate the effectiveness of the bidirectional deformation field by comparing it with a conventional deformation field [44] on middle-frame reconstruction.
As shown in Table [4](https://arxiv.org/html/2506.08862v1#S4.T4) and Figure [9](https://arxiv.org/html/2506.08862v1#A2.F9), the bidirectional variant significantly improves reconstruction quality.
The baseline struggles to preserve pixel-aligned structures, leading to noticeable error accumulation over longer sequences.

## 5 Conclusion

In this paper, we introduced StreamSplat, the first feed-forward framework for instant, online dynamic 3D reconstruction from uncalibrated video streams.
By incorporating a probabilistic position sampling strategy and a bidirectional deformation field with adaptive Gaussian fusion,
our StreamSplat effectively addresses key challenges in online dynamic reconstruction, allowing to produce accurate dynamic 3D Gaussian Splatting (3DGS) representations from arbitrarily long video streams. These representations faithfully capture scene dynamics and support interpolation at arbitrary time.
Extensive experiments on both static and dynamic benchmarks validate the superior performance of StreamSplat in terms of reconstruction quality and dynamic scene modeling.
In future, we plan to explore its potential in video generation and autonomous driving.

## Acknowledgement

This work was funded, in part, by the NSERC DG Grant (No. RGPIN-2022-04636, No. RGPIN-2019-05448), the Vector Institute for AI, Canada CIFAR AI Chair, and a Google Gift Fund.
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through the Digital Research Alliance of Canada [alliance.can.ca](alliance.can.ca), and companies sponsoring the Vector Institute [www.vectorinstitute.ai/#partners](www.vectorinstitute.ai/#partners), and Advanced Research Computing at the University of British Columbia.
Additional hardware support was provided by John R. Evans Leaders Fund CFI grant.
ZW and QY are supported by UBC Four Year Doctoral Fellowships.
We thank Qiuhong Shen for constructive discussions and helpful comments.

## References

- Huang et al. [2023]
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei.
Voxposer: Composable 3d value maps for robotic manipulation with language models.
*arXiv preprint arXiv:2307.05973*, 2023.
- Huang et al. [2024]
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei.
Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation.
*arXiv preprint arXiv:2409.01652*, 2024.
- Carmigniani and Furht [2011]
Julie Carmigniani and Borko Furht.
Augmented reality: an overview.
*Handbook of augmented reality*, pages 3–46, 2011.
- Cruz-Neira et al. [1992]
Carolina Cruz-Neira, Daniel J Sandin, Thomas A DeFanti, Robert V Kenyon, and John C Hart.
The cave: Audio visual experience automatic virtual environment.
*Communications of the ACM*, 35(6):64–73, 1992.
- Sun et al. [2020]
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.
Scalability in perception for autonomous driving: Waymo open dataset.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 2446–2454, 2020.
- Matsuki et al. [2024]
Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison.
Gaussian splatting slam.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 18039–18048, 2024.
- Yan et al. [2024]
Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li.
Gs-slam: Dense visual slam with 3d gaussian splatting.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 19595–19604, 2024.
- Yugay et al. [2023]
Vladimir Yugay, Yue Li, Theo Gevers, and Martin R Oswald.
Gaussian-slam: Photo-realistic dense slam with gaussian splatting.
*arXiv preprint arXiv:2312.10070*, 2023.
- Zhu et al. [2024]
Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R Oswald, Andreas Geiger, and Marc Pollefeys.
Nicer-slam: Neural implicit scene encoding for rgb slam.
In *2024 International Conference on 3D Vision (3DV)*, pages 42–52. IEEE, 2024.
- Shotton et al. [2013]
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon.
Scene coordinate regression forests for camera relocalization in rgb-d images.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2930–2937, 2013.
- Wang et al. [2024a]
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud.
Dust3r: Geometric 3d vision made easy.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 20697–20709, 2024a.
- Xu et al. [2024a]
Kai Xu, Tze Ho Elden Tse, Jizong Peng, and Angela Yao.
Das3r: Dynamics-aware gaussian splatting for static scene reconstruction.
*arXiv preprint arXiv:2412.19584*, 2024a.
- Chen et al. [2025]
Xingyu Chen, Yue Chen, Yuliang Xiu, Andreas Geiger, and Anpei Chen.
Easi3r: Estimating disentangled motion from dust3r without training.
*arXiv preprint arXiv:2503.24391*, 2025.
- Zhang et al. [2024a]
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang.
Monst3r: A simple approach for estimating geometry in the presence of motion.
*arXiv preprint arXiv:2410.03825*, 2024a.
- Charatan et al. [2024]
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann.
pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 19457–19467, 2024.
- Ye et al. [2024]
Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang, and Songyou Peng.
No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images.
*arXiv preprint arXiv:2410.24207*, 2024.
- Li et al. [2025]
Yang Li, Jinglu Wang, Lei Chu, Xiao Li, Shiu-hong Kao, Ying-Cong Chen, and Yan Lu.
Streamgs: Online generalizable gaussian splatting reconstruction for unposed image streams.
*arXiv preprint arXiv:2503.06235*, 2025.
- Liang et al. [2024]
Hanxue Liang, Jiawei Ren, Ashkan Mirzaei, Antonio Torralba, Ziwei Liu, Igor Gilitschenski, Sanja Fidler, Cengiz Oztireli, Huan Ling, Zan Gojcic, et al.
Feed-forward bullet-time reconstruction of dynamic scenes from monocular videos.
*arXiv preprint arXiv:2412.03526*, 2024.
- Shen et al. [2025a]
Qiuhong Shen, Xuanyu Yi, Mingbao Lin, Hanwang Zhang, Shuicheng Yan, and Xinchao Wang.
Seeing world dynamics in a nutshell.
*arXiv preprint arXiv:2502.03465*, 2025a.
- Schonberger and Frahm [2016]
Johannes L Schonberger and Jan-Michael Frahm.
Structure-from-motion revisited.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 4104–4113, 2016.
- Kerbl et al. [2023]
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis.
3d gaussian splatting for real-time radiance field rendering.
*ACM Transactions on Graphics*, 42(4):1–14, August 2023.
ISSN 0730-0301, 1557-7368.
doi: 10.1145/3592433.
- Mildenhall et al. [2021]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.
Nerf: Representing scenes as neural radiance fields for view synthesis.
*Communications of the ACM*, 65(1):99–106, 2021.
- Bian et al. [2023]
Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu.
Nope-nerf: Optimising neural radiance field with no pose prior.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 4160–4169, 2023.
- Fu et al. [2024]
Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros, and Xiaolong Wang.
Colmap-free 3d gaussian splatting.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 20796–20805, 2024.
- Lin et al. [2021]
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey.
Barf: Bundle-adjusting neural radiance fields.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 5741–5751, 2021.
- Wang et al. [2023]
Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely.
Tracking everything everywhere all at once.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 19795–19806, 2023.
- Liu et al. [2024a]
Qingming Liu, Yuan Liu, Jiepeng Wang, Xianqiang Lyv, Peng Wang, Wenping Wang, and Junhui Hou.
Modgs: Dynamic gaussian splatting from casually-captured monocular videos.
*arXiv preprint arXiv:2406.00434*, 2024a.
- Jain et al. [2024]
Siddhant Jain, Daniel Watson, Eric Tabellion, Ben Poole, Janne Kontkanen, et al.
Video interpolation with diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 7341–7351, 2024.
- Zwicker et al. [2001]
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross.
Ewa volume splatting.
In *Proceedings Visualization, 2001. VIS’01.*, pages 29–538. IEEE, 2001.
- Sun et al. [2024]
Yang-Tian Sun, Yihua Huang, Lin Ma, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi.
Splatter a video: Video gaussian representation for versatile processing.
*Advances in Neural Information Processing Systems*, 37:50401–50425, 2024.
- Reizenstein et al. [2021]
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.
Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.
In *International Conference on Computer Vision*, 2021.
- Zhou et al. [2018]
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.
Stereo magnification: Learning view synthesis using multiplane images.
*ACM Trans. Graph. (Proc. SIGGRAPH)*, 37, 2018.
- Pont-Tuset et al. [2017]
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool.
The 2017 davis challenge on video object segmentation.
*arXiv:1704.00675*, 2017.
- Xu et al. [2018]
Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang.
Youtube-vos: Sequence-to-sequence video object segmentation.
In *Proceedings of the European conference on computer vision (ECCV)*, pages 585–601, 2018.
- Ouyang et al. [2024]
Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen.
Codef: Content deformation fields for temporally consistent video processing.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 8089–8099, 2024.
- Chen et al. [2021]
Yinbo Chen, Sifei Liu, and Xiaolong Wang.
Learning continuous image representation with local implicit image function.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 8628–8638, 2021.
- Sitzmann et al. [2020]
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.
Implicit neural representations with periodic activation functions.
*Advances in neural information processing systems*, 33:7462–7473, 2020.
- Tancik et al. [2020]
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.
Fourier features let networks learn high frequency functions in low dimensional domains.
*Advances in neural information processing systems*, 33:7537–7547, 2020.
- Pumarola et al. [2021]
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.
D-nerf: Neural radiance fields for dynamic scenes.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10318–10327, 2021.
- Park et al. [2021a]
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.
Nerfies: Deformable neural radiance fields.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 5865–5874, 2021a.
- Park et al. [2021b]
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.
Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields.
*ACM Transactions on Graphics (TOG)*, 40(6):1–12, 2021b.
- Li et al. [2022]
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al.
Neural 3d video synthesis from multi-view video.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 5521–5531, 2022.
- Luiten et al. [2024]
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan.
Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.
In *2024 International Conference on 3D Vision (3DV)*, pages 800–809. IEEE, 2024.
- Li et al. [2024]
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu.
Spacetime gaussian feature splatting for real-time dynamic view synthesis.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 8508–8520, 2024.
- Duan et al. [2024]
Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen.
4d gaussian splatting: Towards efficient novel view synthesis for dynamic scenes.
*arXiv e-prints*, pages arXiv–2402, 2024.
- Yuan et al. [2025]
Yuheng Yuan, Qiuhong Shen, Xingyi Yang, and Xinchao Wang.
1000+ fps 4d gaussian splatting for dynamic scene rendering.
*arXiv preprint arXiv:2503.16422*, 2025.
- Lee et al. [2024]
Junoh Lee, ChangYeon Won, Hyunjun Jung, Inhwan Bae, and Hae-Gon Jeon.
Fully explicit dynamic gaussian splatting.
*Advances in Neural Information Processing Systems*, 37:5384–5409, 2024.
- Shi et al. [2025]
Dongbo Shi, Shen Cao, Lubin Fan, Bojian Wu, Jinhui Guo, Renjie Chen, Ligang Liu, and Jieping Ye.
No parameters, no problem: 3d gaussian splatting without camera intrinsics and extrinsics.
*arXiv e-prints*, pages arXiv–2502, 2025.
- Jang et al. [2025]
Wonbong Jang, Philippe Weinzaepfel, Vincent Leroy, Lourdes Agapito, and Jerome Revaud.
Pow3r: Empowering unconstrained 3d reconstruction with camera and scene priors.
*arXiv preprint arXiv:2503.17316*, 2025.
- Leroy et al. [2024]
Vincent Leroy, Yohann Cabon, and Jérôme Revaud.
Grounding image matching in 3d with mast3r.
In *European Conference on Computer Vision*, pages 71–91. Springer, 2024.
- Smart et al. [2024]
Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu.
Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs.
*arXiv preprint arXiv:2408.13912*, 2024.
- Yang et al. [2025]
Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli.
Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass.
*arXiv preprint arXiv:2501.13928*, 2025.
- Wang et al. [2025]
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny.
Vggt: Visual geometry grounded transformer.
*arXiv preprint arXiv:2503.11651*, 2025.
- Wang et al. [2024b]
Yunsong Wang, Tianxin Huang, Hanlin Chen, and Gim Hee Lee.
Freesplat: Generalizable 3d gaussian splatting towards free view synthesis of indoor scenes.
*Advances in Neural Information Processing Systems*, 37:107326–107349, 2024b.
- Chen et al. [2024]
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai.
Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images.
In *European Conference on Computer Vision*, pages 370–386. Springer, 2024.
- Zhang et al. [2024b]
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu.
Gs-lrm: Large reconstruction model for 3d gaussian splatting.
In *European Conference on Computer Vision*, pages 1–19. Springer, 2024b.
- Shen et al. [2025b]
Qiuhong Shen, Zike Wu, Xuanyu Yi, Pan Zhou, Hanwang Zhang, Shuicheng Yan, and Xinchao Wang.
Gamba: Marry gaussian splatting with mamba for single-view 3d reconstruction.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2025b.
- Yi et al. [2024]
Xuanyu Yi, Zike Wu, Qiuhong Shen, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Shuicheng Yan, Xinchao Wang, and Hanwang Zhang.
Mvgamba: Unify 3d content generation as state space sequence modeling.
*arXiv preprint arXiv:2406.06367*, 2024.
- Xu et al. [2024b]
Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Marc Pollefeys.
Depthsplat: Connecting gaussian splatting and depth.
*arXiv preprint arXiv:2410.13862*, 2024b.
- Hong et al. [2024]
Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jisang Han, Jiaolong Yang, Chong Luo, and Seungryong Kim.
Pf3plat: Pose-free feed-forward 3d gaussian splatting.
*arXiv preprint arXiv:2410.22128*, 2024.
- Zhang et al. [2025]
Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yujun Shen, and Gordon Wetzstein.
Flare: Feed-forward geometry, appearance and camera estimation from uncalibrated sparse views.
*arXiv preprint arXiv:2502.12138*, 2025.
- Liu et al. [2024b]
Tianqi Liu, Guangcong Wang, Shoukang Hu, Liao Shen, Xinyi Ye, Yuhang Zang, Zhiguo Cao, Wei Li, and Ziwei Liu.
Mvsgaussian: Fast generalizable gaussian splatting reconstruction from multi-view stereo.
In *European Conference on Computer Vision*, pages 37–53. Springer, 2024b.
- Chen et al. [2016]
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.
Training deep nets with sublinear memory cost.
*arXiv preprint arXiv:1604.06174*, 2016.
- Chung et al. [2024]
Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee.
Depth-regularized optimization for 3d gaussian splatting in few-shot images.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 811–820, 2024.
- Yang et al. [2024]
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao.
Depth anything v2.
*Advances in Neural Information Processing Systems*, 37:21875–21911, 2024.
- Vaswani et al. [2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
*Advances in neural information processing systems*, 30, 2017.
- Dosovitskiy et al. [2020]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
*arXiv preprint arXiv:2010.11929*, 2020.
- Liu et al. [2021]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.
Swin transformer: Hierarchical vision transformer using shifted windows.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 10012–10022, 2021.
- Xu et al. [2024c]
Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein.
Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation.
In *European Conference on Computer Vision*, pages 1–20. Springer, 2024c.
- Zhao et al. [2024]
Boming Zhao, Yuan Li, Ziyu Sun, Lin Zeng, Yujun Shen, Rui Ma, Yinda Zhang, Hujun Bao, and Zhaopeng Cui.
Gaussianprediction: Dynamic 3d gaussian prediction for motion extrapolation and free view synthesis.
In *ACM SIGGRAPH 2024 Conference Papers*, pages 1–12, 2024.
- Oquab et al. [2023]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.
Dinov2: Learning robust visual features without supervision.
*arXiv preprint arXiv:2304.07193*, 2023.
- Zhang et al. [2018]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual metric.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 586–595, 2018.
- Ranftl et al. [2020]
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.
Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.
*IEEE transactions on pattern analysis and machine intelligence*, 44(3):1623–1637, 2020.
- Dao [2023]
Tri Dao.
Flashattention-2: Faster attention with better parallelism and work partitioning.
*arXiv preprint arXiv:2307.08691*, 2023.
- Karras et al. [2022]
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.
Elucidating the design space of diffusion-based generative models.
*Advances in neural information processing systems*, 35:26565–26577, 2022.
- Loshchilov and Hutter [2017]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
*arXiv preprint arXiv:1711.05101*, 2017.
- Wang et al. [2004]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.
Image quality assessment: from error visibility to structural similarity.
*IEEE transactions on image processing*, 13(4):600–612, 2004.
- Stearns et al. [2024]
Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wetzstein, and Leonidas Guibas.
Dynamic gaussian marbles for novel view synthesis of casual monocular videos.
In *SIGGRAPH Asia 2024 Conference Papers*, pages 1–11, 2024.
- Liu et al. [2023]
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang.
Robust dynamic radiance fields.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13–23, 2023.
- Li et al. [2023]
Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng.
Amt: All-pairs multi-field transforms for efficient frame interpolation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9801–9810, 2023.
- Huang et al. [2022]
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.
Real-time intermediate flow estimation for video frame interpolation.
In *European Conference on Computer Vision*, pages 624–642. Springer, 2022.
- Reda et al. [2022]
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless.
Film: Frame interpolation for large motion.
In *European Conference on Computer Vision*, pages 250–266. Springer, 2022.
- Danier et al. [2024]
Duolikun Danier, Fan Zhang, and David Bull.
Ldmvfi: Video frame interpolation with latent diffusion models.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 1472–1480, 2024.

## Appendix A Background

3D Gaussian Splatting.
3D Gaussian Splatting (3DGS) [21] represents a static scene using a collection of 3D Gaussians. Each Gaussian $\mathcal{G}$ is defined by its mean (position) $\boldsymbol{\mu}$, covariance matrix $\mathbf{\Sigma}$, opacity $\boldsymbol{\alpha}$, and color represented by spherical harmonics (SH) coefficients $\boldsymbol{c}$. The final opacity of a 3D Gaussian at a given point $\mathbf{x}$ is computed as:

$$ $\boldsymbol{\alpha}(\mathbf{x})=\boldsymbol{\alpha}\cdot\exp\left(-\frac{1}{2} (\mathbf{x}-\boldsymbol{\mu})^{T}\mathbf{\Sigma}^{-1}(\mathbf{x}-\boldsymbol{ \mu})\right).$ (6) $$

Normally, the covariance matrix $\mathbf{\Sigma}$ is decomposed into a diagonal scaling matrix $\mathbf{S}$ and a rotation quaternion $\mathbf{R}$ for differentiable optimization:

$$ $\mathbf{\Sigma}=\mathbf{R}\mathbf{S}\mathbf{S}^{T}\mathbf{R}^{T}.$ (7) $$

Given a set of $N$ 3DGS $\mathcal{G}=\{{G}_{i}\}_{i=1}^{N}$, the rendering process involves splatting them onto the 2D image plane and then blending their colors based on their opacity and depth.

The 3D Gaussians are projected using the approximate transformation [29]:

$$ $\mathbf{\Sigma}^{\prime}=\mathbf{J}\mathbf{W}\mathbf{\Sigma}\mathbf{W}^{T} \mathbf{J}^{T},$ (8) $$

where $\mathbf{J}$ is the Jacobian of the perspective projection function defined as:

$$ $\displaystyle(u,v)$ $\displaystyle=\Bigl{(}f_{x}\cdot x/z+c_{x},f_{y}\cdot y/z+c_{y}\Bigr{)},$ (9) $\displaystyle\mathbf{J}=\frac{\partial(u,v)}{\partial(x,y,z)}$ $\displaystyle=\begin{pmatrix}f_{x}/z&0&-f_{x}\cdot x/z^{2}\\ 0&f_{y}/z&-f_{y}\cdot y/z^{2}\end{pmatrix}.$ $$

In case of orthographic projection, the projection is simplified to:

$$ $\displaystyle(u,v)$ $\displaystyle=\Bigl{(}f_{x}\cdot x+c_{x},f_{y}\cdot y+c_{y}\Bigr{)},$ (10) $\displaystyle\mathbf{J}=\frac{\partial(u,v)}{\partial(x,y,z)}$ $\displaystyle=\begin{pmatrix}f_{x}&0&0\\ 0&f_{y}&0\end{pmatrix}.$ $$

The pixel color $\boldsymbol{C}$ is obtained by alpha-blending the projected 2D Gaussians sorted by depth:

$$ $\boldsymbol{C}=\sum_{i=1}^{N}\boldsymbol{c}_{i}\boldsymbol{\alpha}_{i}\prod_{j =1}^{i-1}(1-\boldsymbol{\alpha}_{j})$ (11) $$

where $\boldsymbol{\alpha}_{i}$ is the opacity of the $i$-th projected Gaussian obtained by Eq. ([6](https://arxiv.org/html/2506.08862v1#A1.E6)).

Dynamic 3D Gaussian Splatting.
Dynamic 3D Gaussian Splatting extends the original 3DGS with a deformation field to model the motion of the Gaussians over time [43]. Typically, the deformation fields $\mathcal{D}(t)=\{\boldsymbol{\mu}(t),\mathbf{R}(t),\boldsymbol{\alpha}(t)\}$ are used to update the static canonical Gaussian with various of approaches [44, 45, 30, 47, 48]. For example, given a static position $\mu$ and a deformation field $\mu(t)$, the deformed position can be obtained as: $\hat{\mu}_{t}=\mu+\mu(t)$.

Figure: Figure 7: Qualitative results on RE10K. Blue box: given frames; Red box: interpolated frames. Compared to other dynamic-scene reconstruction methods, StreamSplat produces more detailed and consistent 3D reconstructions across diverse scenes, whereas other methods often exhibit distortions in both color and geometry.
Refer to caption: x7.png

Figure: Algorithm 1 3DGS Parameters Predictions

## Appendix B Implementation Details

Model Configurations.
As shown in Table [5](https://arxiv.org/html/2506.08862v1#A5.T5), StreamSplat consists of an image tokenizer, a static encoder, a dynamic decoder, and an upsampler. The image tokenizer takes RGBD inputs at a resolution of 288$\times$512 and uses a patch size of 8 to produce 768-dimensional tokens. Both the static encoder and the dynamic decoder contain 10 transformer layers with an embedding dimension of 768 and 12 attention heads. The dynamic decoder uses 0.1 stochastic drop path to prevent overfitting on image features. An upsampler with 2 layers of window attention with 2304 token length increases the token length to $16\times$ and reduces the embedding dimension from 768 to 192. The model uses a linear head for static prediction and a two-layer MLP for deformation prediction. We apply loss balancing weights of 1.0 for MSE, 0.05 for both LPIPS and depth, and 3.0 for the mask loss.

3DGS Parameterization.
3DGS parameters are obtained as summarized in Algorithm [1](https://arxiv.org/html/2506.08862v1#algorithm1). A linear head projects each Gaussian token $\hat{\mathbf{E}}$ and, through parameter-specific activations, yields the static tuple $(\boldsymbol{\mu}_{0},\mathbf{S},\mathbf{R},\boldsymbol{\alpha}_{0},
\boldsymbol{c})$. A 2-layer MLP, conditioned on the concatenation $\hat{\mathbf{E}}\oplus\hat{\mathbf{d}}$, predicts the velocity $\mathbf{v}$ and opacity coefficients $\boldsymbol{\gamma}$, from which the final time-dependent values are computed by integrating the current time $t$. Note, we adopt our proposed Probabilistic Position Sampling for all position-related parameters, including position offset $\boldsymbol{o}$ and velocity $\mathbf{v}$, to improve robustness and avoid spatial local minima.

Figure: Figure 8: Qualitative results with 8-frame interval on different datasets. Blue box: given frames; Red box: interpolated frames.
Refer to caption: x8.png

Figure: Figure 9: Ablation. w/o bi.: without bidirectional deformation field. Blue box: given frames; Red box: interpolated frames.
Refer to caption: x9.png

## Appendix C Additional Experimental Results

Due to the limited space in the main manuscript, we provide additional experimental results in this section, including qualitative comparisons with other dynamic methods on static scene reconstruction (Figure [7](https://arxiv.org/html/2506.08862v1#A1.F7)), and qualitative comparisons on ablation studies of our bidirectional deformation field (Figure [9](https://arxiv.org/html/2506.08862v1#A2.F9)), and more qualitative results on video reconstruction and interpolation on CO3Dv2 [31], DAVIS [33], and Youtube-VOS [34] (Figure [8](https://arxiv.org/html/2506.08862v1#A2.F8)). Please refer to the website for video results.

## Appendix D Limitations and Impact

While StreamSplat achieves strong performance, certain design choices introduce some limitations. First, the framework relies on pseudo-depth maps predicted by an external monocular estimator, which may introduce noise–particularly around fine-scale geometry and depth discontinuities. To mitigate this, we incorporate an adaptive decay weighting scheme during training to downweight unreliable depth supervision. Nonetheless, scaling the training dataset to enable internal depth refinement remains a promising direction to reduce reliance on external priors. Second, the bidirectional deformation field is trained over a two-frame window for efficiency. As a result, information from earlier frames may be lost in dynamic scenes with fast motion or extended occlusions. Future work may explore efficient mechanisms for adaptively selecting and fusing Gaussians across extended frame histories, which could help retain more temporal context in challenging scenarios.

Societal impact. While our work enables instant dynamic 3D perception from monocular video, potentially improving accessibility and safety real-world applications like autonomous driving systems, it also warrants careful evaluation in edge cases where performance limitations could affect reliability in safety-critical scenarios.

## Appendix E Licenses

Datasets.

- •
CO3Dv2 [31]: CC BY-NC 4.0
- •
RealEstate10K [32]: CC BY 4.0
- •
DAVIS [33]: BSD 3-Clause License
- •
YouTube-VOS [34]: CC BY 4.0

Pre-trained models.

- •
DepthAnythingv2 [65]: Apache-2.0 License and CC BY-NC 4.0

**Table 5: Detailed model configuration of StreamSplat.**
| Parameter | Value |
| --- | --- |
| Image Tokenizer |  |
| input resolution | 288 $\times$ 512 |
| input channels | RGB (3) + Depth (1) |
| patch size | 8 |
| output channels | 768 |
| Static Encoder |  |
| layers | 10 |
| embed dim | 768 |
| attention heads | 12 |
| droppath rate | 0.0 |
| Dynamic Decoder |  |
| layers | 10 |
| embed dim | 768 |
| attention heads | 12 |
| droppath rate | 0.1 |
| Upsampler |  |
| layers | 2 |
| window size | 2304 |
| upsample ratio | 16$\times$ |
| embed dim | 768 $\to$ 192 |
| Prediction Heads |  |
| static head | Linear |
| deformation head | 2-layer MLP |
| Training |  |
| optimizer | AdamW |
| Adam $(\beta_{1},\beta_{2})$ | $(0.9,0.95)$ |
| lr scheduler | CosineAnnealingLR |
| epochs | 50 / 70 |
| batch size | 128 / 256 |
| peak learning rate | $5\times 10^{-4}$ / $1\times 10^{-4}$ |
| weight decay | 0.05 |
| gradient clipping | 1.0 |
| warm-up iterations | 20K / 100K |
| mixed precision | bf16 |
| Loss Weights |  |
| $\lambda_{\text{MSE}}$ | 1.0 |
| $\lambda_{\text{LPIPS}}$ | 0.05 |
| $\lambda_{\text{Depth}}$ | 0.05 |
| $\lambda_{\text{Mask}}$ | 3.0 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Please see the major weakness above. 

1. In the section on the Bidirectional Deformation Field, the authors describe how forward and backward deformations enhance temporal consistency. However, given that StreamSplat is evaluated on 5-frame and 8-frame interval tasks, it’s unclear why the model does not increase the number of active Gaussians to better represent accumulated motion over longer intervals.

2. Since the input frames are two, why not use the pretrain model from Croco (e.g. Dust3R, Cut3R)
soundness: 2
strengths: Strength:
1. The paper is well-written and easy to follow.
2. StreamSplat is feedforward and increase the speed of reconstruction.
3. In the figure 4, StreamSpalt shows persistent gaussians across frames, which shows the potential of long-term modeling.
summary: StreamSplat introduced a feed-forward framework for reconstructing dynamic 3D scenes from uncalibrated video streams. 

Streamsplat work simliarly as current Dust3R style model and predict the 3D gaussian parameters with a vision transformer.

StreamSplat also introduce three key technical innovation: 1. a probabilistic sampling mechanism 2. a bidirectional deformation field 3. an adaptive Gaussian fusion operation
weaknesses: Major Weakness:

1. Lack of Rigorous Evaluation Protocols: The evaluation does not follow established dynamic reconstruction benchmarks such as DyCheck or NVIDIA Dynamic Scene Dataset. The chosen datasets (DAVIS and YouTube-VOS) are more typical for video segmentation or interpolation, not 4D reconstruction.

2. Limited Training Dataset: The paper uses a mix of static (CO3Dv2, RealEstate10K) and limited dynamic (DAVIS, YouTube-VOS) datasets for training. However, DAVIS contains only a few short clips, this raises concerns about the model’s generalization to complex  motion and real-world application.

3. Dynamic reconstruction typically involves multi-view or multi-camera setups to test geometric consistency (e.g., DyCheck). StreamSplat, in contrast, evaluates on DAVIS using middle-frame interpolation and trivializes the problem.

Minor Weakness:

1.In Figure 1, “curent” should be corrected to “current.”

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: * The paper tackles an underexplored yet practically important problem: real-time dynamic 3D reconstruction from uncalibrated video streams, which existing 3DGS and NeRF-based methods generally overlook due to their offline and per-scene optimization nature.

* Unlike prior optimization-based dynamic 3DGS methods, StreamSplat introduces a fully feed-forward pipeline that supports online inference without requiring camera calibration or pre-computed poses, making it highly suitable for real-world deployment scenarios.

* The experiments cover static and dynamic benchmarks, interpolation tasks, and zero-shot evaluation, providing convincing evidence of the model’s robustness, scalability, and temporal coherence.
summary: The paper first points out the challenge of achieving real-time dynamic 3D reconstruction from uncalibrated video streams while maintaining the quality and functionality of offline methods. To address this, the paper proposes an online dynamic 3D reconstruction framework called StreamSplat, which integrates probabilistic 3D Gaussian encoding, a bidirectional deformation field, and adaptive Gaussian fusion for temporally coherent and efficient scene modeling. The method aims to enable scalable, feed-forward, and fully online reconstruction of arbitrarily long video streams with state-of-the-art quality and significant speed improvement over optimization-based approaches.
weaknesses: * It would be helpful if the authors could clarify whether their framework is capable of predicting or estimating camera poses, given that StreamSplat operates under uncalibrated input conditions. If not, discussing potential extensions in this direction would strengthen the paper's completeness.

* The paper would benefit from additional discussion or experiments on highly dynamic scenes with significant topological changes (e.g., frequent object entries and exits from the field of view). It remains unclear how robust the proposed method is under such conditions.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: **Q1.** Could you provide a visualization of *multi-point tracking*? Figures 3 and the supplementary videos only illustrate a single tracked point. From my understanding, a per-pixel feed-forward network may struggle with precise point correspondence. Showing multiple tracked points (e.g., ~30) in a local neighborhood would give a fairer sense of temporal coherence.  

**Q2.** Please include a **runtime breakdown**—how much time is spent in the encoder, deformation module, and rasterization, respectively? This would help assess real-time feasibility.
soundness: 3
strengths: - **Impressive empirical results:** The method achieves substantial gains over prior 3DGS and NeRF-based approaches, setting a new benchmark for *online dynamic reconstruction from uncalibrated video streams*.  
- **Reasonable and clear pipeline:** The proposed two-stage static/dynamic training scheme is well-motivated and reproducible. The combination of a strong image encoder and a dynamic decoder makes architectural sense.  
- **Excellent efficiency:** StreamSplat attains orders-of-magnitude speedup (seconds vs. hours) compared to optimization-based methods.  
- **Comprehensive evaluation:** The paper compares against strong baselines on multiple datasets, including both static (RE10K, CO3Dv2) and dynamic (DAVIS, YouTube-VOS) settings.
summary: This paper proposes **StreamSplat**, a **feed-forward framework for online dynamic 3D reconstruction from uncalibrated video streams**.  
It aims to replace conventional optimization-based dynamic 3DGS pipelines with a real-time, fully feed-forward alternative.

**Key contributions include:**
1. **First feed-forward dynamic 3D reconstruction framework** under an *uncalibrated* camera setting.  
2. **Effective technical components:**
   - *Probabilistic position sampling* for robust Gaussian position inference under geometric uncertainty.  
   - *Bidirectional deformation fields* that jointly model forward and backward motion to ensure temporal coherence.  
3. **High-quality results:** Both quantitative tables and qualitative visualizations show consistent improvements and strong temporal coherence across datasets.
weaknesses: Major Weaknesses

**W1.** The formulation in *line 157* appears problematic: $(u,v)$ represents pixel coordinates while the offset $o_i$ is in unit space. Their direct addition may be incorrect if the coordinate system is rectilinear. A clarification of this coordinate transformation is needed.  

**W2.** Algorithm 2’s *aggregation and fusion* step is ambiguous. The operation `UPDATE` in line 228 is not clearly defined—does it simply replace $\tilde{\mathcal{G}}$ with$ \mathcal{G}_{k-1}^+ $, or else?  
Additionally:
- The notation $Î_{t_{k−1}→t_k}$ is unclear—what is the meaning of the arrow?  
- The definition of $π$ (projection parameters) is missing.  
- The term **“cached”** in line 219 feels implementation-specific and should be formalized or rephrased.  
Finally, “rendered frames” in line 221 conflicts conceptually with “dynamic scenes” in line 292—please clarify the intended distinction.

Minor Weaknesses

**W3.** The frame indices in *Figure 2* appear inconsistent. Section 3.2 (line 209) describes deformation between $t=0$ and $t=1$; thus, the figure should use either $(t_n, t_{n-1})$ or $(t_0, t_1)$ for consistency.


Decision: Accept (Poster)