Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splatting
Authors: Inkyu Shin
Qihang Yu
Xiaohui Shen
In So Kweon
Kuk-Jin Yoon
Liang-Chieh Chen
Venue: Accepted by TMLR ()

Abstract: Recent advancements in zero-shot video diffusion models have shown promise for text-driven video editing, but challenges remain in achieving high temporal consistency. To address this, we introduce Video-3DGS, a 3D Gaussian Splatting (3DGS)-based video refiner designed to enhance temporal consistency in zero-shot video editors. Our approach utilizes a two-stage 3D Gaussian optimizing process tailored for editing dynamic monocular videos. In the first stage, Video-3DGS employs an improved version of COLMAP, referred to as MC-COLMAP, which processes original videos using a Masked and Clipped approach. For each video clip, MC-COLMAP generates the point clouds for dynamic foreground objects and complex backgrounds. These point clouds are utilized to initialize two sets of 3D Gaussians (Frg-3DGS and Bkg-3DGS) aiming to represent foreground and background views. Both foreground and background views are then merged with a 2D learnable parameter map to reconstruct full views. In the second stage, we leverage the reconstruction ability developed in the first stage to impose the temporal constraints on the video diffusion model. This approach ensures the temporal consistency in the edited videos while maintaining high fidelity to the editing text prompt. We further propose a recursive and ensembled refinement by revisiting the denoising step and guidance scale used in video diffusion process with Video-3DGS. To demonstrate the efficacy of Video-3DGS on both stages, we conduct extensive experiments across two related tasks: Video Reconstruction and Video Editing. Video-3DGS trained with 3k iterations significantly improves video reconstruction quality (+3 PSNR, +7 PSNR increase) and training efficiency (×1.9, ×4.5 times faster) over NeRF-based and 3DGS-based state-of-art methods on DAVIS dataset, respectively. Moreover, it enhances video editing by ensuring temporal consistency across 58 dynamic monocular videos.

Published in Transactions on Machine Learning Research (03/2025)
Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splatting
Inkyu Shin"
dlarbgg334ggaail.com com
Koreo Advanced Institute of Science and Technology
Qihang Yu ByteDance
Xiaohui Shen
shensiiohhai@hyttdancc.ccmm
ByteDance
In So Kweon Korea Advanced Institute of Science and Technology
oc.kr
Kuk-Jin Yoon
kjyoon&kaisst.ocm oc..i
Korea Advonced Institute of Science and Technology
Liang- Chieh Chen ByteDance
Gannghhich.chen@@ggtedee
Reviewed on OpenReview: https: opwrrevie.. net/ forun? id=sizfBJysbI
Abstract
Recent advancements in zero-shot video diffusion models have shown promise for text-driven video editing, but challenges remain in achieving high temporal consistency. To address this, we introduce Vidco-3DGS, 3D Gaussian Splatting (3DGS)-based vidco refiner designed to enhance temporal consistency in zero-shot video editors. Our approach utilizes a two-stage 3D Gaussian optimizing process tailored for editing dynamic monocular videos. In the first stage, Video-3DGS employs an improved version of COLMAP, referred as MC-COLMAP, which processes original videos using Masked and Clipped approach. For each video clip, MC-COLMAP generates the point clouds for dynamic foreground objects and complex backgrounds. These point clouds utilized to initialize two sets of 3D Gaussians (Frg-3DGS and Bkg-3DGS) aiming to represent foreground and background views. Both foreground and background views are then merged with a 2D learnable parameter map to reconstruct full views. the second stage, we leverage the reconstruction ability developed in the first stage to impose the temporal constraints on the video diffusion model. This approach ensures the temporal consistency in the edited videos while maintaining high fidelity the editing text prompt. We further propose recursive and ensembled refinement. revisiting the denoising step and guidance scale used in vidco diffusion process with Video-3DGS. To demonstrate the elficacy of Video-3DGS on both slages, conduct exlensive experiments across two related tasks: Video Reconstruction and Video Editing. Video-3DGS trained with 3k iterations significantly improves video reconstruction quality (+3 PSNR, +7 PSNR increase) and training efficiency (x1.9, x4.5 times faster) over NeRF-based and 3DGS- based state-of-art methods on DAVIS dataset, respectively. Moreover, it enhances video editing by ensuring temporal consistency across 58 dynamic monocular videos. Project website is available here.
Work done during internship RyteDance.

---

Published in Transactions on Machine Learning Research (03/2025)
Video-3DGS
Video Reconstruction
GT Video
Deformable-
30GS
Video 3DGS
Video Editing
Original Video
Driginal Video
Text2vid
Text prompt: "neon lights"
TokenFlow
Text prompt: "through on obstacle Mars"
Text2Vid-Zero Video-3DGS
TakenFlow Video-3DGS
The proposed Video-3DGS expands the capabilities of 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) to dynamic monocular video scenes, enhancing temporal consistency in both video reconstruction and video editing. For instance, it consistently captures and reconstructs dynamic objects such as riders and horses (upper section), while also enriching style smoothness in scenarios like drift-car sequences (bottom left) and ensuring structure consistency (bottom right). Regions of interest highlighted by white dashed reclangles.
Figure 1:
1 Introduction
Advancements in diffusion-basod generative models (Ho et al., 2020; Rombach et al., 2022) have significantly improved text-driven image editing capabilities (Brooks et al., 2023). Building on this success, there has been considerable effort to extend these technologies to the video domain (Ho et al., 2022; Singer et al., 2022), which holds practical potential across broad range of applications, including Film/Entertaiin EEntertainment and AR/VR. However, training vidco diffusion model from scratch using vidco datascts is not effective for two main reasons: (i) it lacks a well-const video training dataset to handle the diverse distribution of videos in the wild, and (ii) discarding pre-trained image diffusion models would incur a penalty in generating high-quality edited frames for vidoos. To address the limitations, zero-shot training-free vidco editing methods (Khachatryan et al., 2023; Geyer al., 2023; Kara et al., 2023) that are built on pre-trained image diffusion model have recently been introduced to not only increase the video editing quality but also improve
2

---

Published in Transactions on Machine Learning Research (03/2025)
wild VDdeo-3D6S stage)
Video-3DGS (2nd stage)
Frg-3DGS
sho
vidio adhor
foreground
video recon
3
background points
video edit
MC-COLMAP
Bkg-3DGS
Figure 2: The overall pipeline of Video-3DGS. We aim to design a video-level 3D Gaussian Splatting framework to reconstruct the video scenes (1st stage), which enables high temporal consistency vidco editing (2nd stage). Specifically, Video-3DGS is empowered by the proposed MC-COLMAP that effectively obtains 3D points for foreground moving objects. The background 3D points modeled with spherical-shaped random points, surrounding the foreground points. Video-3DGS utilizes two sets of 3D Gaussians (Frg-3DGS and Bkg-3DGS) to represent foreground and background 3D points, respectively. A 2D learnable parameter map merges the foreground and background views, rendered from each set of 3D Gaussians. The merged views enable high-fidelity video reconstruction. Then, we leverage this reconstruction capability into zero-shot video oditor to enhance temporal consistency while maintaining high fidelity text prompt.
efficiency. Nevertheless, these zero-shot video diffusson models still exhibit. low temporal consistency in dynamic vidcos due to their limited understanding of individual vidco scenes. A natural question thus emerges: Is it possible to design a simple yet effective blug-and-play module to enhance the temporal consistency of each edited video from any zero-shot video editors?
To answer the question, we carcfully design Vidco-3DGS, innovative approach that leverages per-scene representation power of 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) to enhance lemporal consistency ability for zero-shot video diffusion models. Our approach aims to utilize the structural preservation ability of 3DGS, which is known for explicitly representing and associating multiple views with unified 3D Gaussians. As illustrated in Figure 2, Video-3DGS employs a two-stage 3DGS optimization process to reconstruct dynamic monocular videos and refine the initially edited ones, ensuring enhanced temporal consistency and visual coherence.
The first stage begins by addressing the limitations of 3DGS in representing dynamic monocular video scenes, where multiple objects move against complex backgrounds. To solve this issue, methods adopting 3DGS for dynamic monocular videos are emerging. Deformable-3DGS (Yang et al., 2023) employed the deformation network to condition 3DGS with time-variable, while 4DGS (Wu et al., 2023a) additionally added HcxPlane (Cao & Johnson, 2023) encoder before the deformation network. Despite these attempts to exlend 3DGS for monocular video scenes, it important to acknowledge the persisting limilations, which are summarized follows. (i) The fragile dependency on the underlying SfM method, e.g., COLMAP (Schönberger & Frahm,, 2016) for deriving 3D points from the entire dynamic videos. (ii) The difficulty of accurately representing monocular videos with single set of 3D Gaussians. Video frames, particularly those with dynamic motion and intricate background, are infeasible to be represented by single set of 3D Gaussians, even with the deformation network. As we can observe from Figure 1, Deformable-3IGSS (Yang et al., 2023) shows unsatisfying video reconstruction results for dynamic moving objects. Therefore, we first design the framework of Video-3DGS to address the complexities of dynamic monocular video scenes. One of the key distinctions of the proposed method lies in its ability to excel in "wilder" monocular vidco scenarios (e-g., videos with larger object motion). To this end, we initially simplify the video sequences with two decomposition strategies. First, spatial decomposition, powered by an off-the-shelf open vocabulary video segmentor, mitigates background clutter 1. Second, temporal decomposition breaks down the entire video sequence into multiple shorter video clips with overlapping frames between neighboring clips. Given these
I Bbackground containing complex regions that clutter the foreground objects (e.g., trees in the background).
3

---

Published in Transactions on Machine Learning Research (03/2025)
decompositions, we can effectively extract 3D points of masked foreground moving objects in each video clip by introducing MC-COLMAP. Meanwhile, the cluttered background is modeled with spherical-shaped random 3D points surrounding the pre-extracted 3D points of foreground moving objects. For each video clip, Vidco-3DGS utilizes two sets of 3D Gaussians (Frg-3DGS and Bkg-3DGS) to represent foreground and background 3D points, respectively. We additionally implement multi-resolution hash encoding (Müller, 2021) with deformation networks (Yang et al., 2023) on both Frg-3DGS and Bkg-3DGS, which can boost performance and efficiency. Last but not least, to obtain the final rendered 2D outputs from both 3D Gaussians, adopt straightforward and effective 2D learnable parameter map to merge the two rendered images (rendered views from Frg-3DGS and Bkg-3DGS), resulting in faithful video frame representations.
The subsequent stage focuses on seamlessly integrating a pre-optimized Video-3DGS into existing zero-shot video editors. The primary advantage of using a pre-optimized Video-3DGS lies in its ability to apply structured constraints of 3D Gaussians across multiple video frames. More specifically, we maintain the structural components of Frg-3DGS and Bkg-3DGS a fixed state while selectively fine-tuning the color parameters, such as spherical harmonic coefficients, alongside a 2D learnable parameter map. This fine-tuning process is designed to capture and replicate the style of the initially edited video frames. By applying fixed 3D Gaussian structures with style updates in cach clip and smoothing transitions between overlapping frames across neighboring clips, we effectively minimize style flickering throughout the entire video, enhancing temporal consistency. We have also observed that existing zero-shot video editors exhibit sensitivity to variations in parameters, such as the number of denoising steps and the scale of image or text guidance. These variations can significantly impact the editing outputs, as demonstrated in Figure 3. To optimize effectiveness and reduce parameter sensitivity, we introduce recursive and ensembled video editing strategy- This involves interspersing Vidco-3DGS between split denoising steps and updating styles using multiple videos edited under different guidance scales. This further exploration helps to stabilize the editing outcomes across varying parameter settings.
To validate the proposed two-stage Vidco-3DGS optimization, we tested on corresponding video tasks: vidco rection and video editing. For video reconstruction, our approach surpasses both NeRe-based and 9DGS-based state-of-the-ad the-art methods across 28 DAVIS videos. We then demonstrate the applicability of Video-3DGS's video scene reprosentation to the video editing task in 58 challenging monocular vidcos sourced from the CVPR 2023 LOVEU Text-Guided Video Editing (TGVE) challenge (Wu et al., 2023b). Video-3DGS Zeeoo
consistently enhances editing quality across three off-the-shelf video editors et al., 2023), TokenFlor foow (Geyer et al., 2023), and RAVE (Kara et al., 2023)).
2
Method
The meta-architecture of Video-3DGS aims to design a vidco-specific 3DGS, which serves as a plug-and-play refiner for initially edited video from zero-shot video editors. This process is structured into two seamless stages. The first stage of Video-3DGS (Section 2.2) aims to represent and reconstruct original videos with two integrated components: MC-COLMAP (Section 2.2.1) and Doregrouud/Bacckroound /Background 3DGS (Section 2.2.2). MC-COLMAP plays a pivotal role in generating masked clip-level foreground and background 3D points. Subsequently, for each clip, two sets of 3D Gaussians, Frg-3DGS and Bkg-3DGS, are initialized and optimized based on these points. Additionally, 2D learnable parameter map is employed to merge the foreground and background views rendered from Frg-3DGS and Bkg-3DGS. The resulting views accurately represent the video frames, facilitating the video reconstruction. Section 2.3 details the second stage, where we transform the optimized Video-3DGS into a plug-and-play temporal refiner for video editing.
2.1 Preliminary
Diffusion Model Diffusion probabilistic models (DPM), introduced by (Sohl-Dickstein et al., 2015) and further developed in (Ho et al., 2020) represent a class of generative models designed to approximate a data distribution q through a progressive denoising process. The model starts with a Gaussian i.i.d noisy image 2T ~ (0, I) and the diffusion model 69 incrementally denoises this image until it reaches a clean image 2n drawn from the target distribution q. The Deterministic Sampling algorithm (DDIM) described in (Song et al., 2020) initializes the noise process, termed DDIM inversion, by starting from clean image zo- This
4

---

Published in Transactions on Machine Learning Research (03/2025)
video editing after N₂/ 2
video editing after Na
"Jrucks drive recetrack, autumn, fall colors."
guidance scale:
1.0
guidance scale:
1.5
video editor
(e.g., Text2Video-Zero)
Figure 3: We revisit the key hyperparameters in the video diffusion process: the denoising step (Na) and the guidance scale. Employing a higher denoising step combined with a lower guidance scale (c.g., similar to the image guidance scale in Text2Video-Zert (Khachatryan et al., 2023)) results in greater fidelity to the editing prompt but compromises struutuaal and temporal consistency, and vice versa. This analysis confirms that, the zero-shot video editor model highly sensitive to these hyperparameters.
approach effectively facilitates image editing through diffusion models. Subsequently, the technique has been adapted for video editing by incorporating temporal constraints. This extension employs several strategies to ensure consistency across frames, inchuding the use of a 3D UNet with cross-frame attention as explored in Text2Video-7ero (Khachatryan al., 2023), diffusion feature propagation as discussed in TokenFlow (Geyer et al., 2023), and the grid trick detailed in RAVE (Kara et al., 2023).
3D Gaussian Splatting We also present overview of 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023). The method involves the initialization of set of 3D Gaussians upon 3D point clouds. The point clouds are derived from Structure from Motion (SfM) techniques, exemplified by COLMAP (Schônberger & Frahm, 2016) with the input V a series of consecutive frames x; (i.e., V = (x:) in monocular video sequence. Then, 3D Gaussians are trained with the function of G(z, r, s, a, SH) containing center position x, 3D covariance matrix obtained from quaternion and scaling s, opacity o, and spherical harmonic cocfficient. SH.
2.2 Video-3DGS (1st Stage): Reconstructing Videos with 3DGS
2.2.1
MC-COLMAP
We denote the conventional COLMAP function as R, which is designed to derive a 3D point cloud p, along with the associated camera data e (encompassing both intrinsic (cin) and extrinsic (c parameters). Furthermore, it outputs a feedback signal status, which serves as an indicator of the COLMAP system's success in roconstructing 3D points from the entire video frames V. That is,
p, C, status R(V)
(1)
However, the presence of foreground moving objects and cluttered background in video frames poses a significant challenge in extracting consistently matched 3D points across the entire video frames. This difficulty lcads to inaccurate and sparse reconstructions in vidoo outputs, is evident in our experiments. To lackle these challenges, we propose MC-COLMAP, revised version of COLMAP, specifically designed to progressively process video frames. Tt effectively minimizes motion and complex background through two key strategies: spatial decomposition and temporal decomposition.
Spatial Decomposition To reduce the background cluttering effect, we adopt an off- the-shhf open- vocabulary video object segmentation network S (e.g., DEVA (Cheng al., 2023)) to extract the segmentation masks for foreground moving objects in the video V as follows:
vf S(V, class)
(2)
where yf and class denote the extracted segmentation masks of foreground moving objects and the user- guided text prompt (a required input for the segmentation network to specify the target object), respectively.
5

---

Published in Transactions on Machine Learning Research (03/2025)
Clip #1
Clip
Monocular
video
fflled, SPM with frames
Open-vocabulary
MO-COLMAP
MC-COLMAP
vOs
Recon
LRecon
Frg-3DGS 6
Bkg-3DGS 3DGS 6)
Frg- Frg-3DGS c rendering 1
3DGS
clip-level
optimization
rendering f
rendering A
rendering 1
marged
merged
Figure 4: The proposed Video-3DGS (1st stage) comprises two key components. First, to effectively capture 3D point clouds and corresponding frame viewpoints from dynamic monocular videos, we introduce Masked and Clipped COLMAP (MC-COLMAP), This module spatially and temporally decomposes video frames, facilitating the extraction of clip-level foreground points through progressively processing clips. Additionally, we initialize spherical-shapel random background points conditioned on the foreground points. Second, with these two sets of point clouds, we introduce two distinct sets of 3D Gaussian Splatting (3DGS): Frg-3DGS and Bkg-3DGS, optimized separately for foreground and background points, respectively. Subsequently, we employ straightforward merging operation to combine the rendered outputs of Frg-3DGS and Bkg-3DGS. We optimize the merged rendered outputs for each clip using the reconstruction loss.
For example, in the top left Figure 4, we can consistently extract 'Swan' across frames in monocular video using spatial decomposition strategy.
Temporal Decomposition The segmentation masks Vf for foreground objects, which span the entire video sequence, present processing challenge for COLMAP due to their intricate motion dynamics. To mitigate this complexity, stemming from diverse motion patterns throughout the video sequence, we partition the video sequence into multiple shorter video clips. This division ensures that objects exhibit reduced motion within each clip, thus facilitating more manageable processing for COLMAP. Formally, we address this by splitting y into multiple M clips (1/)
Rather than evenly dividing the video sequence into M clips, each containing k frames, we introduce a progressive scheme to address potential failure cases, such as when the foreground object remains static within clip, making the & frames inadequate for point cloud extraction or registration in SfM. Specifically, we begin with the first clip initialized with the first & frames (i.e., v fx (h)))) If the COLMAP function Rt fails to process the clip (i.e,, statusi * 'Success'), we iteratively include one additional consecutive frame into the current clip until COLMAP returns a 'Success' status. Subsequently, the next clip commences with the last frame of previous clip, also initialized with frames. This process continues until the entire video sequence processed. We term the resulting pipeline as MC-COLMAP, which applies COLMAP in a Masked and Clipped manner. Our MC-COLMA system (denoted as RMC) yields multiple sets of masked and clipped 3D point clouds, along with their corresponding views from M clips demonstrated in the top right of Figure 4 and following equation:
k)) RArc(S(V, class))
(3)

---

Published in Transactions on Machine Learning Research (03/2025)
We validate the effectiveness of MC-COLMAP in Table 1 and Table 3, demonstrating that MC-COLMAP not only improves the success rate over the original COLMAP, but also provides better-initialized 3D points for constructing 3D Gaussians.
2.2.2 Foreground and Background 3DGS
Foreground 3D Gaussians For each video clip, the 3D point clouds for foreground moving objects, derived from Equation (3), serve as initialization for optimizing foreground 3D Gassuians. This process yields set of 3D Gaussians {GfTS)M tailored for those foreground point clouds with the corresponding camera views, ((p), - With this, we successfully represent the foreground objects using 3D Gaussians, and the subsequent step involves modcling the cluttered background.
1j-1:
Background 3D Gaussians For each video clip, to simply model the cluttered background, we utilize spherical-shaped random point clouds {p, Bkg و
Jj=1, surrounding the previously extracted foreground 3D points. These background random point clouds serve as the initialization for optimizing background 3D Gaussians, yielding a set of 3D Gaussians {G/ha > two syper-parametend
1==1- Notably, the spherical-shaped random point clouds are defined by number of points BBg and its radius r₃. Our method is empirically found robust to those pyper-parameters and we fix Bakg = 60k points and ra to times larger than the foreground points' distance that is measured by the maximum Euclidean distance between foreground points.
Deformable 3D Gaussians Following (Yang et al., 2023), both the foreground and background 3D Gaussians are enhanced with the deformable network, which effectively leverage the time information, but we extend them for clip-level processing, resulting in:
Frg-3DGS: ((5,)))) + +5 55 , SH M Bkg-3DGS: (G) (lag,
(4)
85} (G +5 5Bhg o, SH
where 6y is the deformation within the j-th clip to transform center Ij. rotation rj, and scale 8j, according to each clip center and normalized time. The superscript Frg and Bkg denote foreground and background, respectively. We implement the deformation network with 4D) multi-resolution hash encoding (Müller, 2021). Given those two sets of Frg-3DGS and Bkg-3DGS, we reformulate them as Clip-3DGS. where each clip contains its corresponding Frg-3DGS and Bkg-3DGS. That is, Clip-3DGS {G of fj-1- process those 3D Gaussians in a clip-by-clip manner.
G.kkn M We then
Merging Foreground and Background Views with 2D Learnable Parameters For the j-th clip, we leverage the differentiable point-based rendering technique (Wiles et al., 2020), as outlined in (Kerbl et al., 2023), to process the two sets of 3D Gaussians G9 and G This enables us to generate two distinct rendered images for each frame within the j-th clip. Specifically, rendering the i-th video frame produces two images, {I,', (f 1Bkg), derived from Frr and G respectively. To seamlessly merge these images, both of dimensions height H and width W, we introduce a straightforward yet powerful merging technique. This method employs a 2D learnable parameter E RxW, facilitating pixcl-wise merging with corresponding learnable parameters initialized to a value of 0.5. Formally, we have:
merged = a; x FTg + (1 (z) x 1, Bkg
(5)
where pmergedd is the merged result for the i-th video frame.
Through the merging operation, we derive merged images for all video frames by optimizing N different a values (N represents the total number of frames in the video).
Training Losses To optimize Frg-3DGS and Bkg-3DGS, adopt the reconstruction loss, denoted Lrecon, which comprises two components: Ln and LSSIM, akin to the approach outlined in (Kerbl et al., 2023). These components are calculated by comparing three rendered images the foreground FFr, the background Bkg, and the merged image pmerged against their respective ground truth images.
Video Reconstruction The reconstruction of vidco can be achieved by sequentially rendoring the Frg-3DGS and Bkg-3DGS for each clip. Subsequently, the rendered images from each frame are merged using pre-trained alpha parameter (Fquation (5)).
7

---

Published in Transactions on Machine Learning Research (03/2025)
recurs editing
ensemble editing
update
guidance
optimized Video-3DGS
scale:
video editor
guidance
scale:
append
denoising step: Na N,
guidance
dencising step:
scale:
Ngf N,
XN.
Figure 5: The overview of Video-3DGS (2nd stage) a. plug-and-play refiner for video editing begins with fine-tuning the spherical coefficient of the optimized Video-3DGS produced using an off-the-shelf video editor with the default hyperparameters: denoising step N₂ and a guidance scale. This method, referred to as the ainle-phase refiner with Video-3DGS, is further enhanced by our findings in Figure 3. We split Na into a recursive number N, and fine-tune the spherical coefficient parameters against multiple outputs from varied guidance scales, aiming for improved temporal consistency and high fidelity the editing text. This advanced approach is named the Recursive and Ensembled (RE) refinement.
initially edited video, which is
2.3 Video-3DGS (2nd Stage): Plug-and- Play Refiner for Editing Videos
Single-phase Refiner We employ pre-trained 3D Gaussians to preserve the original structure and ensure temporal consistency in video editing. Edited vidco frames, denoted as
(Tedited N JEzl
(6)
can be generated by any zero-shot video editor. However, these edits may introduce style inconsistencies or cause original objects to be changed. To address this issue, we fix the positional parameters (æ,, r, 8) and deformation parameters (6) of both Frg-3DGS and Bkg-3DGS. At the same time, we adjust the color value (SH) and opacity (a) parameters to match the edited style. This update is achieved by minimizing the reconstruction loss between the rendered images {1 [.merged v and edited frames (Tedited] j==1 for updating corresponding SH and a follows:
min SH.0
Lrecon (T merrged redited)
(7)
Adjusting the color values ensures that corresponding regions in the rendered images remain consistent within each clip. Furthermore, by refining overlapping frames between adjacent clips, we achieve smoother style transitions and improved temporal consistency throughout the video. Given that we adopt Video-3DGS following single-phase of vidco editing, Video-3DGS (2nd stage) can be appropriately referred as a single-phase refiner.
Recursive and Ensembled Refiner In the single-phase refiner, update SH and by minimizing the reconstruction loss between the rendered images and the edited frames. However, when the off-the-shelf video editor uses an excessive number of denoising steps (Na) with editing prompts, some edited frames may lose essential structural details. To counteract this, we split N₂ into multiple phases (with N, = for efficiency) and provide the detailed explanation for each phase.
First Phase: we first apply the signle-phase refinement Equation (7) on the initially edited frames (e.g., after Na/2 denoising steps). This produces set of intermediate refined frames.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Metrics
Method
success rase
PSNR
SSIM
Time
Robust-Dyn (Tán ot nl., 2023)
28 28
26.8
0.796
746m
NeRF-based
NLA (Kasten et al, 2021)
25
/ 28
27.3
0.795
309m
CoDeF [Ouyang sl, 2023)
0,869
28m
Nerv (Chen al., 2021b)
/ 28
34.6
0,980
20m
3DGS-bese
3DGS [Kerb] es ab. 2025)
20 / 28
24.8
0.832
10m
Deform-DDG (Yang es sl., 2023)
20 / 28
30.6
0.919
50m
Video-9DGS (3k)
3DGS-based (ours)
Video-3DGS (5k)
28
/ 28
41.2
0.989
22m
Video-3DOS (10k)
28 / 28
45.8
0.985
56m
Table 1: Average quantitative results of DAVIS videos for Video Reconstruction. 'success rate' indicates the proportion of videos, out of the total 28 videos, that can be successfully reconstructed. Our method, Video-3DGS, demonstrates efficient training and significantly outperforms NeRF-basse and 3DGS-based methods. We report Video-3DGS results trained with 3k, 5k, and 10k iterations.
Inilialization for the Next Phase: we use the intermediate edited frames from above first phase as input for DDIM inversion (as in TokenFlow (Geyer et al., 2023) and RAVE (Kara et al., 2023) or an image encoder (as in Text2Vidco-Zan (Khachatryan ct al., 2023)) to initialize the subsoquent phase.
Second Phase: Then, we complete the remaining Na/2 denoising steps for further video editing and follow with final single-phase refinement on the newly edited frames.
We refer to this overall procedure as recursive refinement with multiple phases. Additionally, the guidance parameters in the diffusion process (e.g., image guidance in 2Vxd2oio,,,,,r,,, text guidance in TokenFlow, and ControlNet guidance in RAVE) play a crucial role in the quality of the edits. As shown in Figure 3, different guidance scales can lead to significant variations. To reduce sensitivity to these scale changes and combine the benefits of multiple scales, we perform an ensemble update of SH and using edited videos generated at three different scales during each recursive phase. The ensemble of edits from one phase is then incorporated into the next, allowing each phase to benefit from previous refinements. For simplicity, we denote the 2nd-stage Video-3DGS with Recursive and Ensembled refinement as Video-3DGS (RE), as illustrated in Figure 5.
3
Experimental Results
In this section, we evaluate Video-3DGS two tasks: Video Reconstruction (Section 3.1) and Video Editing (Section 3.2). We provide the details of datasets and metrics used for those two tasks (Appendix B), ablation studies (Appendix D), and qualitative results (Appendix E) in the Appendix.
3.1 Video Reconstruction
Baselines As Video-3DGS aims reconstruct videos by learning scene representatious, we compare our approach with two representative methods: NeRF-based and 3DGS-basod reprosentations. For NeRF-based methods, we select four Srate-of-the-art baselines: 1) NLA (Kasten et al., 2021), which proposes video reconstruction and editing method using layered atlases powered by the NeRF framework. 2) CoDEF (Ouyang et al., 2023), which leverages a canonical space of deformation fields to reconstruct and edit videos. 3) RobustDyn (Liu et al., 2023), an advanced view reconstruction and synthesis model that estimates camera poses diverse settings. 4) Nerv Chen al. (2021b), which proposes neural representation to encode videos in neural networks. For 3DGS-based methods, we first consider the original 3DGS method (Kerbl al., 2023), which lacks modules specific to video scenes. Additionally, we select. Defornable-3DGs (Yang et al., 2023), state-of-the-art approach that utilizes deformation network on 3D Gaussian, serving another strong bascline.
Quantitative Results As illustrated in Table 1, our comprehensive video reconstruction experiments encompass 28 videos sourced from DAVIS. We note that the four Nese-based methods generally exhibit limitations in both reconstruction quality (measured by PSNR and SSTM) and efficiency (training time), primarily attributed to their implicit neural representation. Conversely, the 3DGS method (Kerbl al., 2023) demonstrates significant reduction in training time (less than 10 minutes in average of 20 videos),
9

---

Published in Transactions on Machine Learning Research (03/2025)
Method
Zest
+Vilee-3DGS
+Vibbo-DGS (RE)
Datnaet
WarpSSMM
Qunr
WarpSSIMr
QoneT
WarpSSIMf
Quart
DAVIS
(-13.0%)
21.0 (+0.9%)
0.899
(+20.8%)
22.3 (+2.2%)
Videovo
0.773
21.9
0.902 (-12.9%) 12.9%)
22. 1 (+0.29%
0.926 (5.3339)
23.1 (+1.2%)
Youtube
(-144%)
20.4 (+0.0%)
0.922
(+.2.1%)
21.1 (+1.3%)
Method
Token Flow
VVidro-DDSS
VViroo3DCS (RR)
Datase
WarpSSIM1
Qonrt
WorpSSIM
QuacT
WirpSSMP
Qute
DAVIS
0.85G
22.9
0.909
(-5. (-5.1%)
25.9 (+1.0%)
0.912 (+5.7%)
21.8 (+1.9%)
Videovo
0.897
22.6
0.933
(+4.6%
23.5 (+0.9%%
0.987 (+..9%%
29.8 (+1.2%)
Youtube
0,818
22.1
0.923
(-7.5%)
28.1 (+1.0%)
0.923 (+7.5%)
21.2 (+2.15)
Method
RAVE
VVdeo-J1XSS
VVsteo3DDS (IRE)
Datases
WarpSSIMf
Quar?
WarpSSIM
Quact
WarpSSIM
Qutit
DAVIS
0.872
23.4
0.908
(-3.6%)
24.1 (+4.79%
0.913 (+4.19%)
24.8 (+1.5%)
Videovo
0.913
(-1.1%)
23.5 (+1.1%)
(45.1%))
23.6 (+1.2%)
Youtube
0.855
21.7
0.918 (-0.3%
22.9 (+1.2%)
0.921 (+6.0%)
23.5 (+18%)
Table 2: Quantitative results of Vidco Editing on three different datascts: DAVIS, Videvo, Youtube. Each dataset contains four editing calegories: style change, object change, background change, multiple change. We present the results for average WarpSSIM and Qedit of four categories, with text color indicating the effect of Video-3DGS. Our Video-3DGS increases both temporal consistency and overall video editing performance across all initial video editors. f: the higher, the better.
0.72
Usee Pieloceuee
Prefoenno
0.6
0.63
0.4
0.4 0.37
0.4
0.9g9.96
0,20
1125
0.25
0.23
0.2
0.2
Tokou
RAVE
Tox: MiileV Zaeno
TelonnFlow
RAVE
Basellan
Utw/ Vidoo-DDS
UBEqui
BBw Vadoo-11X38
Video-IDGS
Figure 6: User study of Video-3DGS on three video editors.
owing to its explicit 3D Gaussian representation and efficient rasterization. However, the 3DGS method notably exhibits performance degradation (average PSNR of 20 videos: 24.8), as it is tailored for static scenes and lacks design considerations for dynamic video scenes. On the other hand, the state-of-thes bascline Reformable-3DGs, which employs a deformation field for time dimension on 3DGS, shows improvod video reconstruction quality (average PSNR of 20 videos: 30.6); nevertheless, this enhancement comes the expense of compromised training efficiency (50 minutes in average of 20 videos). Furthermore, due to the fundamental issue in COLMAP, they cannot conduct reconstruction from being used for wild video datasets. On the other hand, powered by the proposed MC-COLMAP and framework of BB--3DGS/Bkg--Deee
8 videos, which hinders 3DGS
Video-3DGS can reconstruct all 28 videos in high quality with shorter training time (iteration 3k: 37.6 PSNR with 11 minutes; iteration 5k: 41.2 PSNR with 22 minutes; ileration 10k: 45.8 PSNR with minutes; all results are measured by taking average over 28 videos). Upon training for 3k iterations, Video-3DGS significantly surpasses both the NeRF-based SoTA, Nerv (Chen et al., 2021b), and the 3DGS-based SoTA, Deform-3DGS (Yang et al., 2023), in terms of video roconstruction quality and training efficiency. Specifically, it achieves an improvement in PSNR by +3 and +7 over Nerv and Deform-3DGS, respectively. Furthermore, Video-3DGS demonstrates a notable improvement in training time efficiency, being 1.9 times faster than Nerv and 4.5 times faster than Deform-3DGS.
3.2 Video Editing
Importance of 3D Gaussians for Video Editing Task The primary goal of Video-3DGS is to enhance the temporal consistency of video editing by leveraging 3D Gaussian representations with a two-stage approach. Specifically, in the 1st stage, our Video-3DGS reconstructs videos using 3D Gaussians to establish temporal correspondence between frames within each clip and link neighboring clips with overlapping frames. Tt.
10

---

Published in Transactions on Machine Learning Research (03/2025)
motivates us to prove the video reconstruction capability of Video-3DGS, showcasing its ability to preserve the content and structure of original raw video without need of any 3D task exploration. Afterwards, we are able to exploit this reconstruction ability for video editing tasks in the 2nd stage. We provide explicit evidence of 3D Gaussians' suitability for temporal correspondence as shown in this video link, where we visualize the correspondence for the first five groups of 3D Gaussians. As shown in the visualization, Video-3DGS ensures the correspondenees of similar areas across frames using 3D Gaussians of the same group. We can enhance this further by incorporating the visualization of pixel trajectories tracked by the 3D Gaussians. This enhancement is demonstrated in the provided video link.
Basclines We meticulously choose three zero-shot vidco editing methods as basclinc comparisons. 1) Text2Video-Zero (Khachatryan et al., 2023), which extends Instruct-pix2pix (Brooks et al., 2023) to the video domain by inserting temporal attention within the diffusion model. 2) TokenFlow (Geyer et al., 2023), which achieves temporal smoothness through the propagation of diffusion features using inter-frame correspondences. 3) RAVE (Kara et al., 2023), which employs a noisc-shufling strategy and grid trick for enhancing video editing capabilities.
Quantitative Results We assessed Video-3DGS's editing capability top of those three zero-shot video editors in Table 2. Video-3DGS significantly enhances temporal consistency (WarpSSIM score) across various editing scenarios spanning three different datasets. Consequently, this improvement gemerally yields superior final video editing results (Qeair). It proves that Video-3DGS generally provides robust temporal editing guidance to existing video editors. We also observe that our proposed recursive and ensembled refinement. can further improve both temporal consistency and video editing results on all of video editors and datasets.
User Study To explore user preferences for video editing outcomes, we conducted a user study detailed in Figure 6. We enlisted 20 participants and presented them with 30 randomly paired videos (10 pairs for each of the adopted off-the-shelf video editors) derived from the LOVEU video results. Participants were tasked with selecting their preferred edited video, taking into account fidelity to the provided text prompt (for editing) and temporal consistency. Consistent with the findings in the left panel of Figure 6, we observed that users generally favored Video-3DGS guided video editing results over the original edited videos. Moreover, in the right panel of Figure 6, we conducted another user study show the effectiveness of recursive and ensembled refinement in editing with Video-3DGS.
4 Discussion & Limitations
While the proposed Vidco-3DGS exhibits remarkable performance in both video reconstruction and editing lasks, certain challenges persist.
In video reconstruction, our Video-3DGS still relies on the success of MC-COLMAP even though it is more robust version compared to original COLMAP. Following are scenarios when MC-COLMAP can struggle or fail. Signéficant Occlusions of Foreground Objects: When a foreground object becomes fully or partially occluded in consecutive frames, ACCCOLMAP'ss decomposition process can struggle to track and re-identify the object. Excessive Motion Blur: Rapid camera object movements can induce motion blur, making it difficult for feature detection algorithms to match points across frames accurately. If the blurred regions prevent consistent feature extraction, the reconstruction pipeline may fail to build coherent model for certain frames or produce large alignment errors. We further acknowledge that MC-COLMAP for Video-3DGS requires a preprocessing step for foreground object extraction using an off-the-shelf panoptic segmenter. However, this is notably simpler compared to the five-stage preprocessing pipeline of shape-of-motion Wang et al. (2024a), which involves mask estimation, metric depth, monocular depth, camera estimation, and 2D tracking. Furthermore, we studied the influence of different variants and settings of the off-the-shelf segmenter video reconstruction quality to ensure robustness as discuss more in " "Ablations on Segmentation Modules" of Appendix D.1.
In video editing, unsatisfactory results may arise if the underlying video editor fails or if significant changes to object shapes are required, which contradicts the inherent property of 3D Gaussians prioritizing the preservation of original structure. As a future direction for Vidco-3DGS, we aim to develop an architecture capable of manipulating 3D Gaussians in a more flexible manner to accommodate desired shape changes. Nevertheless, our work represents a pioneering effort in extending 3DGS to enhance the temporal consistency
11

---

Published in Transactions on Machine Learning Research (03/2025)
of imitially edited videos from any zero-shot video editor. We envision that this will inspire further rescarch in zero-shot video editing community.
Unlike other conventional neural representation papers, our work does not explore the task of novel view synthesis. Instead, our method focuses on video reconstruction and editing by encoding and fitting videos with 3DGS. We anticipate that our Video-3DGS will also serve as fundamental framework for 4D novel view synthesis in future research.
5 Conclusion
We introduced Video-3DGS, a novel two-stage approach tailored to edit dynamic monocular video scenes by reconstructing them with 3D Gaussian Splatting (3DGS). The first stage of Video-3DGS lies its capacity to spatially and temporally decompose vidcos, strcamlining motion representation across clips. It surpasses previous state-of-the-art methods in video reconstruction. The second stage of Video-3DGS leverages the reconstruction ability to refine the edited videos from three different. zero-shot video editors, which enhances both temporal consistency and editing results. We anticipate significant positive societal impacts from our method, as it consistently performs well in representing diverse videos and refining outputs from zero-shot video editors. We expect it to benefit, various applications, such as Entertainment with video synthesis and AR/VR.
Acknowledgemmnn This work was supported part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea Government (MSIT) (Artificial Intelligence Innovation Hub) under Grant 2021-0-02068. This rescarch also partially supported by the KAIST Generrainn Collaborative Lab Project.
References
Alex Andonian, Sabrina Osmany, Audrey Cui, YeonHwan Park, Ali Jahanian, Antonio Torralba, and David Bau. Paint by word. arXiv preprint arXiw:2103.1099: 2021.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, and Jiang Bian. Uniedit: A unified tuning-free framework for video motion and appearance editing. arXiv preprint or 240 13185, 2024.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In CVPR, 2021.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR, 2023.
Duygu Ceylan, Chun-Hao Huang, and Niloy J. Mitra. Pix2video: Video editing using image diffusion. In ICCV, 2023.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconotruction from multi-view stereo. orXiv preprint arXiv:2100 15595, 2021a.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. NewrIPS, 2021b.
Ho Kei Cheng, Scoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Young Lec. Tracking anything with decoupled video segmentation. In ICCV, 2023.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. Garf: Gaussian activated radiance fields for high fidelity reconstruction and pose estimation. or preprint or oxxiv2204.057333 2022.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Malthieu Cord. Diffedit: Diflusion-baset semantic image editing with mask guidance. arXio preprint arXis:2210.11427, 2022.
Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv preprint ar xrXi:2211..12..0 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalixing text-to-image generation using textual inversion. ar preprinl arXiv:2208.01611 2022.
Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. arXiv preprint Xiv:211033 10380, 2021.
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint urXiv:2307.10373, 2023.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. Xi preprinl ar arXiv:2110.08955 2021.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208. 01626, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NewrIPS, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. NeurIPS, 2022.
Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, Minsu Cho, and Jaesik Park. Self-calibrating neural radiance fields. orXiv preprint Xrv:221008. 136226088.3326, 2021.
Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M. Rehg, and Pinar Yanardag, Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models. orXie preprint arXiv:2312.0p 0425,, 2023.
Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. Relu fields: The little non-linearity that could. In SIGGRAPH, 2022.
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM Transactions on Graphics (TOG), 40(6):1 12, 2021.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering, ACM Transactions Graphics, 42(4), 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zere Text-to-image diffusion models are zero-shot video generators. Xi preprint Xiv:2330. 13439, 2023.
Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A plug-and-play framework for any rideo-to-video editing tasks. arXiv preprint arXiv:2303.14400 14468, 2024.
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellacrt, and Thomas Funkhouscr. Panoptic neural fields: A semantic object-aware neural scene representation. arXiv preprint arXiv:2205.04 04334, 2022.
Byeonghyeon Lee, Howoong Lee, Xiangyu Sun, Usman Ali, and Eunbyung Park. Deblurring 3d gaussian splatting. arXie preprint arXiv:2401.008. 2024.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-hng image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. arXiv preprint arXie:2104.06. 06405, 2021.
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In CVPR, 2023.
Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro V. Sander. Deblur-nerf: Neural radiance fields from blurry images. arXiv preprint 1rXi::2111114222 2022.
Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf the wild: Neural radiance fields for unconstrained photo collections. arXiv preprint Xiv: 02268, 2021.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint erXhv:2108.0107% 2022.
Andreas Meuleman, Yu-Lun Liu, Chen Gao, Jia-Bin Huang, Changil Kim, Min H. Kim, and Johannes Kopf. Progressively optimized local radiance fields for robust view synthesis. In CVPR, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99-106, 2021.
Thomas Müller. tiny-cuda-nn, 4 2021. URL htttps://githiii
Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation ficlds for temporally consistent video processing. arviv preprinl ar 230: 200220000 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NewrIPS, 2019.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Corkine-Horming and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: XXiv:170400077: 2017.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint orXic:2303.095: 2023.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. arXiv preprint 137.14403,,374 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arziv:2208.122: 2022.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mottio revisited. In CVPR, 2016.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-vidco generation without text-video data. arXiv preprint arXiv:2209. 14799, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics.. In ICML, 2015.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
Dan Wang, Xinrui Cui, Septimiu Salcudean, and Z. Jane Wang. Generalizable neural radiance fields for novel view synthesis with transformer. arXiv preprint ar 2rxi:22000.055 053977 2022.
Qianqian Wang, Vickie Ye, Hang Gao, Weijia Zeng, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. 2024a.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, pp. 1-20, 2024b.
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. aiviv preprint urv: 1911..8800,4. 2020.
Guanjun WU, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXio preprinl Stv:2310.08525 2023a.
Jay Zhangjie Wu, Xiuyu Li, Difei Gao, Zhen Dong, Jinbin Bai, Aishani Singh, Xiaoyu Xiang, Youzeng Li, Zuwci Huang, Yuanxi Sun, Rui He, Feng Hu, Junhua Hu, Hai Huang, Hanyu Zhu, Xu Cheng, Jie Tang, Mike Zheng Shou, Kurt Keutzer, and Forrest landola. Cvpr 2023 text guided video editing competition. arXiv preprint arXiv:2310.16000 2023b.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXie preprint arXiv:2309. 111:1309.1311 2023.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. arXiv preprint 110::2003.,4024, 2021.
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. arXie preprint arXiv:2311.16493, 2023.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV, 2021.
Allan Zhou, Moo Jin Kim, Lirui Wang, Pete Florence, and Chelsea Finn. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. arXiv preprint arXix:2301.08 00.5566 2023.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Appendix
In the appendix, we provide additional information listed below:
Appendix A provides the related works.
Appendix B provides the details of used datascts and metrics.
Appendix C provides the implementation details.
Appendix provides extensive ablation study on the proposed Video-3DGS.
Appendix E provides qualitative results on video reconstruction and video editing.
Appendix F provides the licenses of the assets used.
Appendix G discusses broader impact.
A Related Work
In this part, we introduce the literature related to the core part of Video-3DGS (1st stage): neural rendering for scene representation (Appendix A.1), 3D Gaussian Spatting-ban based representation (Appendix A.2), dynamic scene representation (Appendix A.3), and spatial temporal decomposition neural rendering (Appendix A.4). Regarding Video-3DGS (2nd stage), we briefly review the literature on Text-to-lmage editing (Appendix A.5), and Zero-shot Text-to- Video editing (Appendix A.6).
A.1 Neural Rendering for Scene Representation
The neural radiance field (NcRF) (Mildenhall et al., 2021) is one of the advanced methodologies that represent scenes using implicit neural rendering with MLP. It has seen significant. advancement recently, with major improvements in several key areas. Firstly, the speed of both training (Karnewar et al., 2022) and inference (Garbin et al., 2021) has been substantially increasod, allowing the current models to be trained and deployed within minutes (Barron et al., 2021; Yu et al., 2021; Roiser et al., 2021; Martin-Brualla et al., 2021). Further advancements have addressed image artifacts, including motion blur (Ma et al., 2022), exposure and lens distortion (Jeong et al., 2021), enhancing the quality and realism of rendered scenes. The necessity for precise camera calibrations is becoming less stringent, thanks to methods that allow for local camera. refinement (Chng et al., 2022; Lin et al., 2021). More recent research has also extended NeRF applications beyond traditional uses, exploring areas such as generalization (Wang et al., 2022; Chen et al., 2021a), semantic understanding (Zhi et al., 2021; Kundu et al., 2022), robotics (Zhou et al., 2023), and 3D image style transfer (Gu et al., 2021). These developments indicate a broadening scope and increasing versatility of NeRF models in capturing and rendering complex scenes.
A.2 3D Gaussian Spatting-based Representation
Building upon the advancements Neural Radiance Fields (NeRF) (Kerbl et al., 2023) for scene representation, another innovative approach that has gained attention the utilization of 3D Gaussian distributions model scenes. This methodology, often referred to "Gaussian Scene Representation" leverages the mathematical properties of Gaussian distributions to efficiently capture the volumetric density and color information of 3D scones. Similar to the trend in NeRF, there have been several advancements in performance and efficiency. For example, some of them apply the module used in NeRF to 3D Gaussian-based view synthesis. (Lee el al., 2024) leverages a compact MLP to adjust the covariance of 3D Gaussians, which can effectively reconstruct sharp details from blurry images and Mip-splatting (Yu et al., 2023) address alias issuc 3DGS as similar (Barron et al., 2021). Besides the advancements in quality, (Fan et al., 2023) enhances the efficiency and compactness of 3D Gaussian Splatting for neural rendering by pruning insignificant Gaussians.
16

---

Published in Transactions on Machine Learning Research (03/2025)
A.3 Handling Dynamic Scene Representation
Despite the significant progress in NeRF and 3DGS for static scene representation, both approaches encounter substantial challenges when applied to dynamic scenes. The core limitation stems from their inherent design, which is primarily tailored for stalic environments, thus struggling to adapt to changes in scene geomelry and appearance over time. To address these challenges, recent research endeavors have begun exploring extensions and alternatives to NeRF and 3D Gaussian Splatting that incorporate temporal information into the scene modeling process. For instance, deformation network has been widely used for encoding time dimension both in the framework of NeRF and 3DGS, (Fan et al., 2023) and (Yang et al., 2023) respectively, More developments for dynamic scene representation are introduced in NeRF. Robust-Dyn (Liu et al., 2023) enhances the robust.ness of dynamic radiance field reconstruction by jointly estimating static and dynamic scene components along with camera parameters. NeRV Chen et al. (2021b) utilizes neural implicit representation to encode videos by simply fitting a neural network input vidco frames. Yet, despite their advancements, due to the fundamental limitation in the NeRF framework, these approaches continue to grapple with challenges related to achieving fine detail, maintaining temporal consistency across frames, and requiring extensive optimization times. Driven by these challenges, the proposed Video-3DGS seeks to investigate the application of 3DGS for dynamic scenes captured through monocular video, exploring its efficacy and potential improvements.
A.4 Spatial Temporal Decomposition
Our proposed Video-3DGS decomposes the dynamic video spatially and temporally to represent original video effectively. Similarly, ProgressiveNeRF (Meuleman et al., 2023) proposes method of decomposing video into spatial and temporal components through progressively optimized local radiance fields. It represents a significant advancement in the field of robust view synthesis and largo-scale scene reconstruction. The primary distinction between our method and the ProgressiveNeRF approach lies in how we manage camera poses and frame integration across video clips. While ProgrossiveNeRF dynamically updates camera poses and continues to append frames until the camera's ' pose extends to the boundary of a high-resolution, uncontracted space, our method treats each clip independently. We utilize overlapping frames solely to ensure consistency of 3D Gaussians corresponding to overlapping frames across neighboring clips, with each clip maintaining its unique camera poses. However, all clips can still be rendered onto the same image plane via a differentiable rendering with each distinctive 3D Gaussian and camera pose. Additionally, our process repeats until MC-COLMAN signals a 'Success' status marking a clear difference in the criteria used determine the sufficiency of frames for each clip. Furthermore, ProgressiveNeRF uses several dependencies (e.g., depth estimation and optical flow) and many losses for better temporal decomposition. However, we only need segmentalion masks and simple reconstruction losses (I.1 and SSIM) as in the original 3DGS paper. We also provide extensive visual comparison with ProgressiveNeRRF in this link.
A.5 Text-to-lmage Editing
Text-to-image editing has significantly advanced with the development of generative models, particularly latent diffusion models (Rombach et al., 2022). These models have demonstrated remarkable capabilities in generating high-quality images guided by textual prompts. Methods such as DreamBooth (Ruiz et al., 2022) and Textual Inversion (Gal et al., 2022) have achieved notable success by fine-tuning pre-trained models for specific editing tasks. Recent approaches, like Prompt-to-Prompt (Hertz al., 2022) and DiffEEdit (Couairon et al., 2022), leverage attention mechanisms within these models to enable localized and detailed image edits without extensive retraining. Techniques like SDEdit (Meng et al., 2022) utilize stochastic differential equations to guide image synthosis, enabling precise control over the generated content. In addition to these, Blended Diffusion (Avrahami el al., 2022) presents a method to blend newly generated content seamlessly into existing images. This technique ensures that the synthesized additions are coherent with the original image context, thus maintaining a high level of realism. Another notable approach is Paint by Word (Andonian et al., 2021), which introduces a framework where users can interactively edit images by providing textual descriptions for specific regions. This method utilizes a combination of mask prediction and image generation
17

---

Published in Transactions on Machine Learning Research (03/2025)
to accurately reflect the user's intent. Despite significant advancements in text-to-image editing, the progress in texi-to-video editing technology has been relatively slow.
A.6 Zero-shot Text-to- Video Editing
Zero-shot text-to-video editing leverages pre-trained text-to-image models edit videos without requiring extensive retraining. This approach aims to overcome the limitations of traditional vidco editing methods, such as video-specific training (Ho et. al., 2022) and atlas learning (Kasten et al., 2021), which are often time-consuming and require significant manual effort. Recent advancements in this domain have introduced various techniques to achieve high-quality and temporally consistent video edits. For instance, RAVE (Kara et al., 2023) employs a noise shuflling stralegy to enhance temporal consistency by leveraging spalio-temporal interactions between frames. This method integrates spatial guidance through ControlNet (Zhang et al., 2023) and operates efficiently in terms of memory requirements, making it suitable for editing longer videos. RAVE's approach significantly reduces processing time compared to existing methods, achieving roughly 25% faster editing rate while maintaining high visual quality. Other notable zero-shot methods include Pix2Video (Ceylan et al., 2023) and FateZero (Qi et al., 2023), which utilize sparse- attention and atlention feature preservation, respectively, to maintain motion and structural consistency across video frames. (Khachatryan et al., 2023) synthesizes and edits videos by integrating cross-frame attention and controlling the fidelity to structure of original video with image guidance scale. TokenFlow (Geyer et al., 2023) enforces consistency by propagating diffusion features across frames based on Inter-frame correspondences in zero-shot manner. These techniques collectively demonstrate the potential of zero-shot. video editing in producing temporally coherent and visually appealing videos without the need for extensive training video-speeific datasets. Despite the recognized improvements in efficiency and editing capabilities offered by zero-shol video editors, they still suffer from subpar temporal consistency and overall video editing quality due to a lack of per-scene understanding, This limitation motivates us to design plug-and aad--ndd-lly -layy refiner for zero-shot video editors (we select representative three editors: Text2Video-Zerr / TokenFlow RAVE), aimed at enhancing per-scene understanding and improving the quality of the edited videos.
B Datasets and Metrics
B.1 Datasets
Video Reconstruction To assess the quality of video reconstruction in real-world monocular video scenes, we curated a dataset comprising 28 representative videos from the DAVIS dataset (Pont- et al., 2017), each with resolution of 480p. These videos feature a varied array of foreground moving objects, encompassing humans, vehicles, and animals, together with cluttored backgrounds. Belows are the key features of DAVIS dataset:
Resolution: videos are available two resolutions, 480p and 1080p. Following prior works, we utilized 480p.
Each video has different number of frames ranging from 40 80
FPS: 24
Video sequences: the dataset includes 50 video sequences. Among them, we selected 28 videos that feature a varied array of foreground-moving objects, encompassing humans, vehicles, and animals.
Vidco Editing For vidco editing, we adopt the datasots and text prompts used in the CVPR 2023 workshop, LOVEU Texi-Guided Video Editing (TGVE) challenge (Wu et al., 2023b). They curated a datasel consisting of 76 videos (480p) in-the-wild from DAVIS, Youtube, and Videvo Videos. Each video is edited according different types of text prompts acquired from BLIP-2 (Li et al., 2023), including: style change, object change, background change, and multiple change. As noted in the main paper, we tested our framework in 58 videos with editing prompts, totaling 232 video editing scenarios. Below are key details.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Type
COLMAP
Masked COLMAP 25/28
MC COLMAP
COLMAT
20/28
28/28
processing time
12mn30s
2mm10s
2mnle
Type
Deformahle-3DD 3DGS
FDGS
Prg-3DGS
BDg-3DGS
Fog 3DGS + Bkg 3DGS (11m)
3DGS
Iterstion training
10l (30m)
(4m)
(15m)
G0l (31m)
10k
(34m)
(21m)
PSNR
30.6
32.9
35.3
37.3
35.1
38.0
37.6
41.2
Table 3: Ablation study on MC-COLMAP. The colored cells indicate the best, second best, and third best performances (best viewed in color). Our MC-COLMAP not only successfully generales 3D points for all videos, but also enables the high reconstruction quality.
Resolution: 480x480
Frame number: DAVIS: 32/
Youtube: 128 / Videvo: 32
FPS: 24
B.2
Evaluation Metrics
Video Reconstruction To evaluate video reconstruction quality, two principal metrics utilized: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). These metrics collectively appraise the fidelity of the reconstructed video in comparison to the original. Additionally, efficiency is gauged through the metric of Training Time, which quantifics the required training time, thus indicating the computational efficiency of the reconstruction algorithm.
Video Editing To assess video editing quality, we utilize WarpSSIM, which computes the mean SSIM score between the edited video warped by the optical flow (Teed & Deng, 2020) (derived from the source video) and corresponding original edited video. This metric offers valuable insight into the temporal consistency of post-editing. Furthermore, we employ Qedit (Kara et al., 2023), a comprehensive video editing metric that combines the WarpSSIM score with the CLIPScore text, providing a multiplicative assessment of the overall video editing performance.
C Implementation Details
Our framework is based on Pytorch (Paszke et al., 2019) and 3D Gaussian Splatting (Kerbl et al., 2023), while adopting depth incorporated differentiable Gaussian rasterization (Yang et al., 2023). During training, two sets of 3D Gaussians (both Frg-3DGS and Bkg-3DGS) in each clip are sequentially optimized with three different total numbers of training iterations: a total of 3k, 5k and 10k iterations, each targeting distinct. time-accuracy tradc-off (e.g., 10k iterations for the best quality but the slowest training timc). We employ the same training hyper-paramelers (e.g., learning rale, deformation network setting) specified by (Yang et al., 2023). With optimized 3D Gaussians for each clip, video reconstruction proceeds through sequential differential rendering (Kerbl et al., 2023) of the clips. For video editing, 3D Gaussians optimized for original videos undergo fine-tuning for 1k iterations to update the spherical harmonic coefficients (SH) and opacity (a)) guided by the initial edited videos (obtained from an off-the-shelf zera-shot video editor). Each experiment. uses a single A100 GPU.
D Ablation Study
D.1 Video-3DGS (1st Stage)
We present comprehensive analysis of MC-COLMAP. Table 3 compares two baselines: conventional COLMAP, which processes full video frames, and Masked COLMAP, which employs only spatial decomposition. Conventional COLMAP encounters more failure case (8 out of 28 vidcos fail obtain 3D points) and exhibits slower processing time (12m30s). Among the successful cases (20 videos), reconstruction using the state-of-the-att DefSrmable-3DGS (Yang et al., 2023) performs suboptimally in
Analysis on MC-COLMAP
19

---

Published in Transactions on Machine Learning Research (03/2025)
Scene
blackswan
boat
drift-turn
horsejump- high
Clip #1
17(1)
11(1)
10(1)
10(1)
Clip #2
11(1)
13(1)
11(1)
11(1)
Clip #3
11(1)
17(1)
11(1)
11(1)
Clip #4
11(1)
20(5)
11(1)
11(1)
Clip #5
4
22
11(1)
Clip #6
IT(1)
Clip #7
Table 4: Analysis on statistics of MC-COLMAP. We show the number of frames for each clip in different
scenes. The number in the parentheses refers to the number of overlapping frames between neighboring clips.
Metrics
PSNR
SSIM
Method
Frg-3DGS (w/ fore)
35.1
0.923
Frg-3DGS (w/ rdn)
36.5
0.951
Frg-3DGS (w/ fore-trdn)
37.5
0.957
Frg-3DGS + Bkg-3DGS
41.4
0.987
Table 5: Analysis on Frg-3DGS and Bkg-3DGS. 'fore' and 'rnd' denote the foreground and random 3D points, respectively. Utilizing a single set of Frg-3DGS can benefit from both types of 3D points. However, performance can be enhanced further by employing two sets of Frg-3DGS and Bkg-3DGS, tailored specifieally for foreground and background random 3D points, respectively.
both training time and PSNR compared to our Frg-3DGS, powered by multi-resolution hash eucoding-basse deformation network. It is worth noting that this study utilizes only one set of 3D Gaussians, without spatial decomposition for foreground and background points. Masked COLMAP demonstrates improved success rate (only 3 out of 28 videos fail to obtain 3D points) and enhances the processing speed by mitigating the effects of cluttered backgrounds through spatial decomposition. Additionally, reconstruction quality, as measured by PSNR., is improved. Finally, our proposed MC-COLMAP, which incorporates both spatial and temporal decomposition, successfully captures 3D points from all 28 videos. Furthermore, it enables the best reconstruction quality of 41.2 PSNR with 5k training itcrations, significantly outperforming the other settings.
MC-COLMAP Statistics As shown in Table 4, to enhance comprehension of MC-COLMAP, we present statistics that illustrate the distribution of clip length across four representative scenes from the challonging DAVIS datasel. We have selected a default. value of k=10 for our experiments and found out that altering k either 15 does not markedly impact our results. We synchronize different 3DGS sets across video clips through the use of overlapping frames, which indicated in the above table numbers within parentheses. Initially, we designate the count of overlapping frames to be one, although this number be adjusted based on the number of frames added to each clip. It also shows behaviors for slow or fast motion in video. For example, the scenes "blackswan" and "boat" " feature comparatively static and slow movements, whereas "drift-tume turn" and "horsejump-highe are characterized by highly dynamic actions. Each scene begins with an initial set of 10 frames for MC-COLMAP processing. is noted that fast-moving scenes are typically decomposed into or frames. In contrast, scenes with slower motion demand greater number of frames to accuralely extract 3D points for MC-COLMAP analysis.
additionally include the ablation study using different vidco segmentation modules. There are several open-vocabulary vidco object segmentation models available, and we have chosen the most representative models for our experiments, including DEVA model with SAM, and Mobile-SAM. We experimented with three settings: (a) DEVA with SAM and user prompts (i.e., user nceds provide the object class names), (b) DEVA with SAM and VIPSeg foreground categorics (i.c., we use the foreground categories defined by VIPSeg dataset without any user prompts), and (c) DEVA with Mobile-SAM and VIPSeg foreground categories. As shown in this video, our Video-3DGS is robust.
A Ablations on Segmentation Modules
20

---

Published in Transactions on Machine Learning Research (03/2025)
DAVIS (180p)
PSNR
SSIM
WarpSSIM
Qedit
Youtube
PSNR
SSIM
WarpSSIM
Qedit
CoDeF
28.7
0.904
0.926
23.7
CoDeF'
25.5
0.795
0.925
21.8
Video-3DGS (3k)
0.986
Video-IDGS (3k)
40.8
0.986
0.501
23.2
0.324
21.2
RAVE
0.872
23.4
TokeuFlow
0.815
22.1
RAVE- Vidro-IDGS
0.908
24.1
ToleuFlo++ Video---DDS
0.923
23.1
ELVE--Vido--DDGG (RE)
0.913
24.8
PeleuFPoo++Vibee333223 (RE)
0.923
21.2
Table 6: Comparison of the proposed Video-3DGS method with the CoDEF Ouyang et al. (2023) framework in the context of video editing tasks. The results demonstrate that Video-3DGS exhibits strong compatibility with zero-shot video editors, yielding superior or comparable performance to CoDEF across several video editing benchmarks.
to the change of segmentation modules. Furthermore, it can also prove that our MC-COLMAP effectively reconstructs foreground moving objects regardless of their prodefined categorics, thereby achieving comparable reconstruction capabilities.
Analysis on Frg-3DGS and Bkg-3DGS In Table 5, we demonstrate the necessity of employing two sets of Frg-3DGS and Bkg-3DGS for vidco reconstruction. We begin with single set of Frg-3DGS that leverages only the foreground 3D points (denoted as Frg-3DGS w/ fore), resulting in 35.1 PSNR and 0.923 SSTM. By initializing the single set of Frg-3DGS with the spherical-shaper random points (denoted as Frg-3DGS w/ rdu), the performance improves to 36.5 PSNR and 0.951 SSIM. The inclusion of random 3D points uniformly captures both foreground and background points, thus enhancing performance. Furthermore, employing both foreground 3D points and spherical-shaped random 3D points (Frg-3DGS w/ fore-trdn) yields further improvement. Finally, utilizing two sets of Gaussians, Frg-3DGS and Bkg-3DGS, each specific to foreground and background 3D points respectively, significantly enhances performance to 41.4 PSNR and 0.987 SSIM.
2D Learnable Parameter for Merging Foreground and Background Views We visualized the 2D learnable parameter (i.e., a in Equation (5)) in different video scenarios in the following link. As shown in the videos, the learned a focuses on the foreground regions, while the learned (1 - a) focuses on the background regions.
Model Size The overall size of the model is determined by both the number of clips and the size of the Frg-3DGS and Bkg-3DGS for each clip. For example, consider the "blackswan" scene. the Video-3DGS (3k iteration) necessitates, on average, a 15MB Frg-3DGS and 60MB Bkg-3DGS per clip. Consequently, the cumulative model size amounts to 375MB. In contrast, the conventional 3DGS (30k itcration) employs a singular, substantial 3DGS model, approximately 300MB in size, which is a result. of the densification technique employed during long iterations. Despite the relatively big model size required for 3DGS, since it represents and stores the scene in an explicit manner (point cloud), we can take advantage of the efficient training and inference capability of 3DGS in videos. It should be noted that the aggregate size of the Video-3DGS model escalates with an increasing number of clips. Therefore, pivotal objective for future research is to minimize the overall size requisite for Video-3DGS implementation. One of the naive ways of achieving this is opting for a larger k value so that we can effectively reduce the final model size, as this leads to a decreased number of clips.
D.2 Wideo-3DGS (2nd Stage)
Comparison With NeRe- Video Editing In addition to Table 2 where we show the compatibility of Video-3DGS with zero-shot video editors, we compare with training- video editor that utilizes NeRF Mildenhall et al. (2021) CoDEF Ouyang et al. (2023) as in Table 6. We can observe that Video- 3DGS with zero-shot video editors (e.g. RAVE or TokenFlow) show comparable or better results than "CoDeF+ + Video-3DGS in terms of both reconstruction and editing ability. Qualitative Results with CoDEF can be found in this link. As shown the qualitative results, CoDEF struggles to preserve the content and structure of the original video due to its application of ControlNet on a canonical image. In contrast, our Video-3DGS effectively maintains the original content and structure.
21

---

Published in Transactions on Machine Learning Research (03/2025)
Metries
WarpSSIM
Quiet
Method
UniEddic (Bai et nl, 2024)
0.779
19.95
Video-3DGS
0.830
21.15
AnyV2V (Ku et al, 2024)
0.737
19.85
+Videe-3DGS
0.836
22.25
Table 7: Additional results on compatibility of Video-3DGS with video editors using video generation models. Our Vidco-3DGS shows consistent compatibility with video editors using vidco generation model by increasing the editing quality. We used 16 DAVIS videos with 32 frames and initially edited them with style-change prompts from LOVEU (Wu et al., 2023b) benchmark.
VideoVooooeerr
+
TokenFlow Video-3DGS
+
Video-3DGS
RAVE
+ Video-3DGS
Editing time
1mll
6m17
2m550s
16m7s
7mn20s
I 66m5s
Table 8: Ablation study on time cost of Video-3DGS ('style change' in DAVIS).
Comparison With Editors Using Video Generation Models Recently, a new trend in video editing has emerged by leveraging video generation models. These approaches can be broadly categorized into two types: those that use image-to-video (I2V) priors and those that employ text-to-video (T2V) priors. To illustrate, we selected a representative method from each and V2V (Ku et al., 2024) (12V-based). Due to UniEdit's high memory requirements, we divided DAVIS videos (32 frames each) into four chunks and applied the LaVie (Wang et al., 2024b) T2V model to each chunk as a prior for video editing. For V2y, we edited the first frame using Instruct-pix2pix (Brooks et al., 2023) and then propagated the edits to the remaining frames using its 12V model. After obtaining the initial outputs from each methods, we refined the results with our Video-3DGS. As shown in Table 7, Video-3DGS demonstrates strong compatibility with recent methods that use video generation priors, particularly when processing DAVIS videos with style-change prompts from the LOVEU benchmark.
(Bai et al., 2024) (T2V-based)
Time Cost Analysis Similar to Table 1, we also explore the time efficiency of video editing when incorporating Video-3DGS atop existing zero-shot video editing frameworks in this section in Table 8. This assessment spans three critical phases: optimizing Frg-3DGS and Bkg-3DGS with the initial video, updating SH and a for the initially odited video, and rendering to produce a refined edit. We can demonstrate that integrating Video-3DGS as an enhancement module consistently requires approximately 6 minutes, irrespective of the underlying editing techniques, affirming its efficiency reasonable investment in processing time.
Analysis on Recursive and Ensembled Refinement After observing the effectiveness of Video-3DGS (RF) in Table 2, we conducted an ablation study on its components, as shown in Table 9. Each component individually improves performance compared to a singlo-phase refiner. The 'Recursive' component increases WarpSSIM by 1.5% and Qutit by 0.4%. Similarly, the 'Ensembled' component euhances the refiner, resulting in 7% increase in WarpSSIM and 0.4% increase in Qedit. Our Video-3DGS (RE), which combines both recursive and ensembled strategics, achieves the highest scores in both WarpSSTM (0.899) and Qedit (22.3).
D.3 Relationship between Video-3DGS (1st Stage) and Video-3DGS (2nd Stage)
We examine the correlation between vidco reconstruction and editing quality in Table 10. We introduce two versions of Video-3DGS: model A, where all clips utilize the same deformation field (thus reducing the representation capacity), and model B, our default setting. As depicted in the table, model B achieves superior reconstruction quality compared to model A, resulting better performance across all three off-the-shelf video editors. We also conducted qualitative comparison with 3DGS (Kerbl et al., 2023) to further prove the positive relationship between video reconstruction and video editing in this link.
22

---

Published in Transactions on Machine Learning Research (03/2025)
Text2Vid-Zero
Video-3DGS
WarpSSIM
Quei
Dataset
WarpSSIMt
Quart
Remursive
Ensemblod
0.827
21.0
0.812
21.1
0.897
21.4
0.899
22.3
Table 9: Analysis the components of recursive and ensembled Video-3DGS.
Test2Vioo
Video-3DGS
TelesFFow
Nideo-IDGS
Video-IDSS
Ealting
WargSSIM Qeeit
WarpSSSI Qudit
WarpSSMM
Qedi
WarpSSiM
Qudit
WarpSSIM
Quet
WarpSSIM
6.701 21.1
7
0,509 /
7
0.574
Table 10: The relationship between video reconstruction and video editing on DAVIS. Across all three off-the-shelf video editors (Text2Video-7enss Zero, TokenFlow, and RAVE), model (with better reconstruction ability) yields better video editing results than model A.
E
Qualitative Results
In this section, we provide more qualitative results for both video reconstruction (Appendix E.1) and video editing (Appendix F.2).
E.1 Video Reconstruction
For video reconstruction, we visualize qualitative results in Figure 7, Figure and Figure 9. As shown in the figures, the proposed Video-3DGS consistently demonstrates higher reconstruction quality. Video comparisons with 3DGS (Kerbl et al., 2023) and Deformable-3DGS (Yang et al., 2023) are provided in the following link
E.1.1 Spatial Decomposition
We also provide visual analysis on the spatial decompasition in the following video link. Our method succossfully extracts the corresponding 3D points of moving objects using MC-COLMAP. We also found that the spatial decomposition can work well in a scene with multiple foreground objects.
E.2
Video Editing
For video editing, we provide the visualization results for the 'Single-phar Refiner' and 'Recursive and Ensembled Refiner'.
E.2.1 Single-phase Refiner
To show the visual effectiveness of Video-3DGS as singlo-phase refiner, we visualize qualitative results Fig- ure 10, Figure 11, and Figure 12, which adopt Tex(2Video-Zero (Khachatryan et al., 2023), TokenFlow (Geyer et al., 2023), and RAVE (Kara al., 2023) as the underlying video editor, respectively. As shown in the figures, Vidco-3DGS effectively enhances the temporal consistency in the edited results across all three video editors. We further provide the video analysis of Video-3DGS as plug-and-play refiner to show better visual comparison in this link.
E.2.2 Recursive and Ensembled Refiner
We further conduct the visual analysis on Iideo-3DGS as Recursive and Ensembled (RE) refiner. First, similar Figure 3, we provide more vidco results showing the sensitivity of video editors hyperparameter changes in this link, It confirms that different hyperparameter values in the deployed video editors results in different outputs. Then, show the editing visual improvement brought by Video-3DGS (RE) in here.
23

---

Published in Transactions on Machine Learning Research (03/2025)
F
Asset Licenses
The licenses of the assets used in the experiments are denoted as follows:
Dalasets:
DAVIS2017 (Pont-Tuset et al,, httge://barrcccaallngggggo////e
LOVEU-TGVE-2023 (Wu al., 2023b): http:///ggtthbbc///////bbb//ataa
Codes:
3D Gaussian Splatting (Kerbl et al., http:///wg.g..ttooo/////cc./o/ta
Text2Video-Zero (Khachatryan et al., 2023): http::/gggtta..o...aaaaaaaaa/aas
TokenFlow (Geyer al., htt00//ttttrrp///aaaaa.////////a
RAVE (Kara et al., 2023): http:::/ggattnb..m///ehhgaaa//
Models:
Stable
Diffusion v 1.5 Models (Rombach et al., 2022): https///angchgfaacccco/mmmme
stable-diffusioo---1-
G
Broader Impact
This paper presents Video-3DGS, which leverages 3D Gaussian Splatting reconstruct and edit dynamic monocular videos. We anticipate significant positive societal impacts from our method, as it consistently performs well in representing diverse videos and refining outputs from zero-shot video editors. We expect it to benefit various applications, such as Entertainment with video synthesis and A./VR.
24

---

Published in Transactions on Machine Learning Research (03/2025)
GT video
3
NLA
CoDeF
Robust -Dyn
3DGS
Deform- 3DGS
Video- 3DGS
(3k)
Video-
3DGS
(10k)
Figure 7: Qualitative results for vidco reconstruction.
25

---

Published in Transactions on Machine Learning Research (03/2025)
GT video
NLA
CoDeF
Robust -Dyn
3DGS
Deform- 3DGS
Video-
3DGS
(3k)
Video-
3DGS
(10k)
Figure 8: Qualitative results for vidco reconstruction.
26

---

Published in Transactions on Machine Learning Research (03/2025)
GT video
NLA
CoDeF
Robust -Dyn
3DGS
Deform- 3DGS
Video-
3DGS
(3k)
Video-
3DGS
(10k)
Figure 9: Qualitative results for vidco reconstruction.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Original video
3
Initial edited (Text2Video Zero)
VVdeo-3DGS
prompt:
Text prompt: man dancing on a wave.
truck drifts muddy aff-road track, sepia toned.
Original
Initial edited video (Text2Video-Zen) Zero)
Wideo-3DGS
Text prompt:
prompt:
Man wearing wet suit surfing on the ocean, golden hour lighting.
An Audi Q7 trail, anime style.
Figure 10: Qualitative results single stage refiner for video editing by adopting Text2Video-Zero.
28

---

Published in Transactions on Machine Learning Research (03/2025)
Original video
3
3
Initial edited lTokenflow)
Viidoo 3DGS
Text prompt: car drifts snowy road.
Text prompt:
A surfing made of aurora borealis,
Original video
Initial edited video (ToienFlow)
Wideo-3DGS
Text prompt:
Text prompt:
The squirrel climbing down wooded park, Van Gogh style
Several goldfish swim tank, impressionist style.
Figure 11: Qualitative results of single stage refiner video editing by adopting TokenFlow.
29

---

Published in Transactions on Machine Learning Research (03/2025)
Original video
Initial edited video (RAVE)
-Video-3DG
Text prompt: A car driving on birthday cake
Text prompt:
Eric Jones Santana & Ivona Ariel dancing salsa among people on the roof skyscraper city lights the background.
ADELTA
3
3
Original video
3
3
AELATL A MONIIRI
V
0
(RAVE)
VI
0
BEALS
AELIA
7
-Video-3GGS
Text prompt:
Text prompt: car drifts on racetrack, neon lights.
A Delta Airlines aircraft descends onto the runaway during cloudless morning, sepia toned.
Figure 12: Qualitative results of single stage refiner for vidco editing by adopting RAVE.
30


Reviews:
[{"review_id": "t48AaKjbCs", "reviewer": "Reviewer_k6rY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method effectively reduces flickering and inconsistent object rendering in video editing through structured 3D constraints.\n2. The method is a plug-and-play module that is compatible with off-the-shelf video editing frameworks.\n\nWeaknesses:\n1. Can the author provide more intuition on why it is necessary to introduce MC-COLMAP for 3D reconstruction and then do Gaussian splatting from 3D back to 2D in video editing refinement? Will the error accumulate during this 2D-3D-2D process?\n2. While it makes sense that video editing methods based on zero-shot adaptation of image diffusion models (e.g. Tokenflow, T2V-Zero) lack the video generation prior and can thus introduce temporal artifacts, recently there has been a trend of using video generation models (e.g. text-to-video generation model or image-to-video generation model) as a prior for video editing. Examples include UniEdit [1] (using a T2V model), AnyV2V [2] and I2VEdit [3] (using I2V models). These methods effectively overcome the lack of temporal understanding problems and reduce the flickering artifacts. It is necessary to compare Video-3DGS against these methods and see if the proposed 3D reconstruction and Gaussian splatting pipeline outperform these methods (or if using them together can further enhance the editing quality).\n\n[1] Bai, Jianhong, et al. \"Uniedit: A unified tuning-free framework for video motion and appearance editing.\" arXiv preprint arXiv:2402.13185 (2024).\n[2] Ku, Max, et al. \"AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks.\" Transactions on Machine Learning Research (2024).\n[3] Ouyang, Wenqi, et al. \"I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models.\" SIGGRAPH Asia 2024 Conference Papers. 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Video-3DGS, a novel approach that enhances temporal consistency in video editing by leveraging 3D Gaussian Splatting (3DGS). The method follows a two-stage optimization process: (1) video reconstruction, where it refines dynamic monocular videos using a modified MC-COLMAP pipeline to extract structured foreground and background 3D representations, and (2) video editing, where the refined video representations serve as constraints to improve the temporal stability of zero-shot video editors.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please provide results with more recent video editing baselines, as mentioned above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nU1LeI6BH1", "reviewer": "Reviewer_9M7g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The approach of decomposing the video into point clouds appears original and effective for content reconstruction.\n- The reconstruction quality surpasses that of chosen baselines while requiring less training time.\n- Comprehensive evaluation and detailed ablation studies.\n\n**Weaknesses**\n- The proposed method requires an off the shelf panoptic segmenter to separate background and foreground. \n- The quality of edited videos is constrained by the success of the quality of the content decomposition into point clouds.\n- Due to the reconstruction objective to build the point clouds, the method doesn’t allow for shape modification of edited objects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of zero-shot video editing by introducing a novel method called Video-3DGS, a 3D Gaussian splatting-based video refinement technique designed to improve temporal consistency in zero-shot video editors. Video-3DGS represents video content as point clouds, separating it into foreground and background components, which are further modeled as two sets of 3D Gaussians. This point cloud decomposition provides reconstruction constraints that guide a pre-trained diffusion model to maintain temporal coherence in edited videos. The proposed approach achieves high reconstruction quality and efficient training, effectively enhancing the temporal consistency of zero-shot video edits.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Questions**\n- Specifically, what is the condition for the MC-COLMAP to return a successful / failed status?\n- Are there some scenarios where the video decomposition might fail (despite the MC-COLMAP)? If so, could you provide examples of failed video decompositions to better understand the extent of their impact?\n- How many frames can the method edit? How does the number of frames impact the quality of the editing?\n- The explanation of the second stage for video editing in Section 2.3 is somewhat challenging to follow. I recommend rewriting it to provide a clearer and more intuitive description of the method.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RgczI1Zpxq", "reviewer": "Reviewer_274s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- **Practical system design** that decomposes the problem into a multi-stage optimization process.\n- **Improved performance** on benchmark datasets.\n\n**Weaknesses:**\n\n- The proposed method **primarily focuses on system integration**, resulting in limited technical novelty. Each technical component (MC-COLMAP, Deformable GS, video editing) is essentially a straightforward extension or hyperparameter tuning, and does not offer substantial new insights.\n- The work is **more of an engineering case study** and may not be ideally suited for an ML-focused venue; it might be better positioned in a more applied CV venue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- **Novel pipeline for video editing** based on 3DGS and a diffusion model.\n- The method ensures **temporal consistency** by introducing 3DGS-based explicit scene modeling and a zero-shot video style change using an off-the-shelf video diffusion model.\n- **Experimental results** demonstrate **competitive or improved performance** on benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Because the **technical contribution of the work is fundamentally marginal**, a more extensive evaluation is necessary to validate the practical usefulness of the method. In particular, please include comparisons with:\n\n- **Dynamic NeRF/3DGS methods** such as RodyNeRF and Shape-of-Motion.\n- **Other video editing methods** such as InstructNeRF2NeRF and Gaussian Editor.", "broader_impact_concerns": "Nothing", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Scope of Contributions. 
- Two reviewers (k6rY and 9M7g) acknowledge that the proposed framework is more engineering-focused yet find its 3DGS-based pipeline for monocular video reconstruction and editing to be valuable. Both highlight the empirical improvements on flicker and temporal consistency, which are long-standing challenges for zero-shot video editing. While reviewer 274s is less convinced of the paper’s novelty, they do not dispute that the approach is carefully validated and obtains improved results.

Empirical Validation. 
- The authors addressed the request for additional comparisons with recent video editors (e.g., UniEdit, AnyV2V, Slicedit). The results indicate that Video-3DGS effectively improves temporal stability when combined with these methods. Furthermore, the authors performed additional experiments to benchmark the approach against dynamic reconstruction methods (e.g., Robust-Dyn, partial comparisons with Shape-of-Motion) and showed that 3DGS-based pipeline typically yields higher PSNR in video reconstruction while being faster or requiring fewer complex preprocessing steps.

Addressing Reviewer Concerns.
- Novelty. While 274s still considers the paper’s technical innovations incremental, TMLR guidelines allow acceptance when a system’s thorough empirical validation and demonstration of practical utility make it a worthwhile reference for the community.
- Clarity and Additional Evaluations. Reviewers requested (and the authors provided) clarifications on the second-stage editing process, success/failure conditions of the MC-COLMAP pipeline, and comparisons with more specialized video-editing models. This bolstered confidence in the robustness and general usability of Video-3DGS.

Remaining Minor Points for Revision.
- The text could further emphasize limitations (e.g., shape manipulation is not straightforward with 3DGS, dependence on panoptic segmentation, potential failures in highly dynamic scenes).
- Further clarifications around MC-COLMAP might help readers who wish to replicate the decomposition and understand failure cases in even more detail.

Conclusion. 
- With two reviewers leaning toward acceptance and one leaning toward rejection primarily due to novelty concerns, the consensus is that the method and its evidence-based improvements will still interest the TMLR community. The paper should be accepted, pending minor clarifications and polishing in the final draft.
Decision: Accept (poster)