4D Gaussian Splatting with Scale-aware Residual Field and Adaptive Optimization for Real-time rendering of temporally complex dynamic scenes

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 08:50:48

Abstract: Reconstructing dynamic scenes from video sequences is a highly promising task in the multimedia domain. While previous methods have made progress, they often struggle with slow rendering and managing temporal complexities such as significant motion and object appearance/disappearance. In this paper, we propose SaRO-GS as a novel dynamic scene representation capable of achieving real-time rendering while effectively handling temporal complexities in dynamic scenes. To address the issue of slow rendering speed, we adopt a Gaussian primitive-based representation and optimize the Gaussians in 4D space, which facilitates real-time rendering with the assistance of 3D Gaussian Splatting. Additionally, to handle temporally complex dynamic scenes, we introduce a Scale-aware Residual Field. This field considers the size information of each Gaussian primitive while encoding its residual feature and aligns with the self-splitting behavior of Gaussian primitives. Furthermore, we propose an Adaptive Optimization Schedule, which assigns different optimization strategies to Gaussian primitives based on their distinct temporal properties, thereby expediting the reconstruction of dynamic regions. Through evaluations on monocular and multi-view datasets, our method has demonstrated state-of-the-art performance.

Main paper: 
 introduction: 
 The reconstruction of dynamic scenes is pivotal for immersive imaging, driving advancements in various multimedia technologies such as VR, AR, and metaverse. Rendering dynamic scenes from any time, position, and viewpoint is crucial for enhancing user experiences with multimedia products, like free-viewpoint video and bullettime effects. Our objective is to reconstruct a continuous 4D space from a discrete temporal video sequence. However, this endeavor faces several challenges. Firstly, the reconstruction quality acts as a bottleneck for widespread adoption, requiring accurate capture of spatial dimensions and temporal variations in dynamic scenes. Additionally, there's a growing demand for real-time interaction in multimedia products to boost user engagement, highlighting the importance of achieving real-time rendering. Nevertheless, existing methods struggle to achieve both high-quality reconstruction and real-time rendering simultaneously, precisely the issue our approach aims to tackle.Recent advancements in dynamic scene reconstruction have been achieved through methods based on NeRF [23] and 3DGS [14] . NeRF employs an implicit field to model static scenes and achieves photo-realistic view synthesis. Many extensions of NeRF to dynamic scenes either utilize deformation fields and canonical fields to model the motion of objects relative to canonical frames over time [10, 20, 27, 31, 35, 38] , or decompose the 4D volume into spatial-only and spatial-temporal spaces [5, 7, 19, 34] , representing space through combinations of dimensionally reduced features. While significant progress has been made in rendering quality, these methods face a significant disadvantage in rendering speed. The emergence of 3DGS has enabled real-time rendering of dynamic scenes. Some methods [13, 18, [45] [46] [47] have attempted dynamic scene modeling based on 3DGS. However, they either struggle to model temporally complex scenes such as object appearances and disappearances [13, 45, 46] or overlook the spatiotemporal information in the scene [18, 47] , resulting in disadvantages when dealing with temporally complex dynamic scenes.To address the aforementioned challenges, we propose SaRO-GS, aiming to achieve real-time rendering while maintaining highquality reconstruction of temporally complex dynamic scenes. SaRO-GS comprises a set of Gaussian primitives in 4D space and a Scaleaware Residual Field. Each Gaussian receives a unique optimization schedule based on its distinct temporal properties through an Adaptive Optimization strategy. To address the issue of slow rendering speeds, Gaussian primitives in 4D space can be projected to 3D based on their temporal properties and residual features obtained from the Scale-aware Residual Field. Then we can achieve real-time rendering leveraging the fast differentiable rasterizer introduced by 3DGS. For high-quality modeling of temporally complex scenes, we employ the following strategies: Firstly, each 4D Gaussian primitive possesses temporal properties, including temporal position and lifespan. The lifespan allows us to model the appearance and disappearance of objects in dynamic scenes, while the temporal position of Gaussians spans the entire temporal range, rather than being fixed at frame 0 as in previous methods. Additionally, we incorporate scale information of Gaussian primitives into the Residual Field to accommodate their ellipsoidal nature. By encoding the region that the Gaussian primitives occupy rather than just their position, we ensure accurate feature extraction and align with the self-splitting behavior of Gaussian primitives. Thirdly, we introduce an Adaptive Optimization strategy, where unique optimization strategies are assigned to each Gaussian primitive based on its temporal properties, facilitating faster reconstruction of dynamic regions.We extensively evaluated our approach on monocular and multiview dynamic scene datasets, comprising both real and synthetic scenes. Both quantitative and qualitative results demonstrate that our method achieves high-quality rendering in real time and effectively handles temporal complexities in dynamic scenes. Our contributions are summarized below.• We propose a Scale-aware Residual Field, incorporating the scale information of Gaussian primitives. This results in a more precise spatiotemporal representation, considering Gaussian primitives' ellipsoidal nature and self-splitting behavior. • We introduce an Adaptive Optimization strategy, assigning unique optimization schedule to Gaussians based on their unique temporal properties, enhancing the reconstruction of dynamic areas. • Our SaRO-GS excels in managing temporally complex scenarios, delivering state-of-the-art performance in both the reconstruction quality and rendering speed. It achieves an 80x rendering speed improvement compared to NeRF-based methods as shown in Fig. 1 . SaRO-GS is versatile, applicable to both monocular and multi-view scenarios and can also achieve dynamic scene segmentation without any prior knowledge. 
related work 2.1 neural scene representations: 
 In recent years, there has been widespread interest in the representation of static scenes based on Neural Radiance Fields. NeRF [23] , as a representative work, adopts a purely implicit approach, modeling static scenes as a radiance field and synthesizing images through volume rendering. The photo-realistic view synthesis capability of NeRF has inspired a series of works across various domains, including enhancing rendering quality [3, 39, 40, 51, 53] ,different camera trajectories [4, 43] ,sparse inputs [2, 25] ,accelerating training and rendering [8, 11, 12, 24, 32-34, 36, 48, 52] , as well as human body reconstruction [29, 30, 44, 50] and large-scale scene modeling [22, 37] , among others.Recently, there has been a breakthrough in high-quality view synthesis and real-time rendering with 3D Gaussian Splatting [14] and relate works [16, 49, 54, 55] , garnering significant attention in the scene reconstruction field.Expanding static scene representation to dynamic scenes is not a simple task. Some methods [10, 20, 27, 31, 35, 38] have made progress based on deformation fields, modeling the entire scene as a canonical field and a deformation field. They use the deformation field to represent the association between sampled points under different frames and the static canonical. Other methods [5, 7, 19, 34] reduce the dimensionality of the 4D space by decomposing it into a set of planar grids, projecting the 4D sampled points onto planes to obtain corresponding features. This approach effectively models temporal correlations through spatiotemporal planes. Furthermore, some methods [17, 42] adopt a streaming strategy to model residuals between adjacent frames, which is suitable for real-time transmission and decoding. However, these rendering approachs based on NeRF requires dense sampling along rays during rendering, limiting the possibility of real-time rendering.There are some concurrent works based on 3D Gaussian representations for dynamic scenes. [21] uses an online strategy to model dynamic scenes frame by frame, [45] uses a Hex-planes to model the changes of Gaussian primitives over sampling time. Both of them struggle to handle temporal complexities, such as significant motion and object appearances/disappearances. [47] introduces a time dimension to Gaussian, enabling 4D Gaussian to be decomposed into a conditional 3D Gaussian and a marginal 1D Gaussian. In comparison, we propose an Scale-aware Residual field to model the residual of Gaussian primitives projected from 4D to 3D, employing an explicit-implicit blending approach that better incorporates spatiotemporal correlations. Other methods like [18] are not suitable for monocular scenarios, and [46] cannot be utilized in multi-view scenes. In contrast, our approach, validated through experiments, demonstrates promising results in both single-view and multi-view scenarios. 
preliminary 3.1 3d gaussian splatting: 
 Given a set of input images of a static scene along with their corresponding camera parameters, 3D Gaussian Splatting (3DGS) initiates the reconstruction of the static scene from an initial point cloud, employing 3D Gaussians as primitives. This approach enables high-quality real-time novel view synthesis. In 3DGS, each Gaussian primitive encompasses a set of attributes, including 3D position 𝜇 3𝑑 , opacity 𝛼, and covariance matrix Σ. A 3D Gaussian G can be represented as:𝐺 (𝑥) = 𝑒 -1 2 (𝑥 -𝜇 ) 𝑇 Σ -1 (𝑥 -𝜇 ) (1)For optimization convenience, 3DGS employs a scaling matrix 𝑆 and a rotation matrix 𝑅 to represent covariance, stored as a 3D vector 𝑠 for scaling and a quaternion 𝑞 for rotation.EQUATIONAdditionally, 3DGS utilizes SH coefficients to represent view-dependent color.Based on the fast differentiable rasterizer implemented by 3DGS, we can achieve rapid image rendering through Gaussian Splatting.To get the rendering images from a given viewpoint, we should first project 3D Gaussian primitives to 2D. Specifically, for a given viewpoint transformation matrix 𝑊 and a projection matrix 𝐾, we can obtain the covariance and the position in 2D space:Σ 2𝐷 = (𝐽𝑊 Σ𝑊 𝑇 𝐽 𝑇 ) 1:2,1:2 𝜇 2𝐷 = 𝐾 ( 𝑊 𝜇 (𝑊 𝜇) 𝑧 ) 1:2 (3)where 𝐽 is the Jacobian of the affine approximation of the projetive transformation. And we can get the 2D Gaussian G 2𝑑 based on Eq. 1.After sorting the Gaussian primitives in 2D space based on depth, we can obtain the color of the specified pixel in the image:EQUATIONHere, 𝑐 𝑖 represents the view-dependent color obtained by combining the SH coefficients of G 𝑖 with the viewing direction.Previous works modeling dynamic scenes using plane field encoders mostly employed hexplanes 𝑃, which encompass spatialonly planes 𝑃 𝑠𝑜 = {𝑃 𝑥,𝑦 , 𝑃 𝑦,𝑧 , 𝑃 𝑥,𝑧 } and spatiotemporal planes 𝑃 𝑠𝑡 = {𝑃 𝑥,𝑡 , 𝑃 𝑦,𝑡 , 𝑃 𝑧,𝑡 }. Each plane 𝑃 𝑖,𝑗 in 𝑃 is a 𝑀 × 𝑁 × 𝑁 twodimensional grid, where 𝑀 represents the feature dimension and 𝑁 represents the spatiotemporal resolution of the grid. For a 4D sample point 𝑞 = (x,y,z,t), we perform interpolation based on its projected coordinates on the six grids to obtain the corresponding feature for this point:EQUATIONHere, 𝑓 (𝑞) is an 𝑀-dimensional feature, 𝜓 𝑏𝑖 represents bilinear interpolation, and 𝜋 𝑖,𝑗 (𝑞) denotes the projected coordinates of sample point 𝑞 on 𝑃 𝑖,𝑗 . 
method: 
 In this section, we initiate with the presentation of overall pipeline of SaRO-GS in Sec. 4.1. Subsequently, we explore the Scale-aware Residual Field in Sec. 4.2, with a particular emphasis on its consideration of scale information for Gaussian primitives. Following this, we elaborate on the Adaptive Optimization strategy employed in Sec. 4.3. Finally, we delineate the loss function and regularization terms utilized in our approach in Sec. 4.4.To represent a dynamic scene, we utilize a set of Gaussian primitives G 4𝐷 in 4D space alongside Scale-aware Residual Field M, as shown in Fig. 2(a) . Each 4D Gaussian primitive G 4𝐷 𝑖 possesses a temporal position 𝜏 𝑖 ,it is learned alongside its 3D position (x,y,z), forming a 4D location 𝜇 4𝐷 = (𝑥, 𝑦, 𝑧, 𝜏). Together with the initial attributes Σ 4𝐷 , 𝑐 4𝐷 and 𝛼 4𝐷 , a 4D Gaussian primitive and its residual feature 𝑓 can be represented as follows:EQUATIONEQUATIONΣ 4𝐷 , 𝑐 4𝐷 and 𝛼 4𝐷 respectively represent the initial covariance, color, and opacity of the Gaussian primitive in 4D space. Similar to 3D Gaussian, we employ quaternion rotation 𝑞 4𝐷 = (𝑞 𝑎 , 𝑞 𝑏 , 𝑞 𝑐 , 𝑞 𝑑 ) and scaling vectors 𝑠 4𝐷 = (𝑠 𝑥 , 𝑠 𝑦 , 𝑠 𝑧 ) to represent covariance, and utilize SH (Spherical Harmonics) coefficients to depict view-dependent color.To address complex temporal scenarios such as object appearance and disappearance, each Gaussian primitive should have a lifespan to indicate how long it can survive in the temporal domain. In order to effectively integrate the Scale-aware Residual Field M with our 4D Gaussian primitives and leverage the spatiotemporal characteristics of M, we employ a tiny MLP F 𝑤 to perform inference on 𝑓 𝑖 and compute the lifespan 𝜎 𝑖 of G 4𝐷 𝑖 :EQUATIONTherefore, in our 4D space, each Gaussian primitive G 4𝐷 𝑖 can obtain a residual feature 𝑓 𝑖 and a lifespan 𝜎 𝑖 through M. {G 4𝐷 𝑖 , 𝑓 𝑖 , 𝜎 𝑖 } fully represents both the initial attributes and temporal characteristics of a Gaussian primitive in 4D space.Once the sampling time 𝑡 0 are given, we need to project the Gaussian primitives from 4D space to 3D space, as shown in Fig. 2(b ). We first need to examine whether G 4𝐷 𝑖 still survives at the current sampling time 𝑡 0 . We adopt a Gaussian-like state function 𝛾 (𝑡) to model the state of G 4𝐷 𝑖 as it varies with the sampling time 𝑡:EQUATION)Where 𝑡 represents the sampling time, and 𝜏 𝑖 represents the temporal position of G 4𝐷 𝑖 . In practice, 𝑘 is set to 4. As the sampling time 𝑡 gradually moves away from the temporal position of G 4𝐷 𝑖 , 𝛾 (𝑡) decreases from 1. When the sampling time reaches the Gaussian lifespan 𝜎 𝑖 , 𝛾 (𝑡) will decrease to 0.01, indicating that G 4𝐷𝑖 is nearly inactive at 𝑡, which means it should be invisible when projected into 3D space. So for a given sampling time 𝑡 0 , We can utilize this state function to represent the opacity of G 4𝐷 𝑖 in 3D space after projection:EQUATIONApart from opacity, other features of G 4𝐷 𝑖 also vary with sampling time 𝑡 when projected into 3D space. We can utilize a set of MLPs F 𝜃 to decode the residual feature 𝑓 𝑖 of G 4𝐷 𝑖 at the sampling time 𝑡, thereby obtaining the residual of the projected attributes as they vary with the sampling time 𝑡.EQUATIONΔ𝜇 𝑖 (𝑡), ΔΣ 𝑖 (𝑡) and Δ𝑐 𝑖 (𝑡) respectively represent the residuals of position, covariance, and color. Here we decode using 𝑡 -𝜏 𝑖 instead of just 𝑡, as we aim to obtain the residual relative to the initial attribute of G 4𝐷 𝑖 in 4D space, where G 4𝐷 𝑖 is temporally positioned using 𝜏 𝑖 .Therefore , we can obtain the attributes of projected G 3𝐷 𝑖 at a given time 𝑡 0 :EQUATIONEQUATIONEQUATION𝜇 4𝐷 𝑖 [: 3] represents extracting the xyz components of 𝜇 4𝐷 𝑖 as the initial 3D location, and ΔΣ 𝑖 can be decomposed into ΔΣ 𝑖 [: 3] and ΔΣ 𝑖 [3 :], representing the residuals of the three-dimensional scaling vector and quaternion rotation about G 4𝐷 𝑖 , respectively. The changes to Σ 4𝐷 𝑖 and 𝑐 4𝐷 𝑖 are achieved by adjusting the corresponding quaternion rotation, scaling vectors, and SH coefficients.Therefore, according to Eq. [8] [9] [10] [11] [12] [13] [14] , Gaussian primitives in 4D space evolve within their lifespan as the sampling time varies, projecting into 3D space at a given sampling time. Then, based on Eq. 34, we employ 3DGS to render 3D Gaussians, obtaining rendered images from a given camera viewpoint, as shown in Fig. 2(c ).To fully integrate the spatiotemporal information of the scene and save computational resources, we adopt hexplanes to represent our Scale-aware Residual Field M, which consists of spatial-only planes and spatiotemporal planes, as described in Sec. 3.2.However, neglecting the size of Gaussian primitives and solely projecting them onto planes based on their 4D positions for feature extraction would lead to incorrect residual features. First, Gaussian (a)When size information is considered, the features of the split Gaussian remain similar to its parent Gaussian. (b)Otherwise, the split Gaussian will have features different from its parent Gaussian primitives can be approximated as ellipsoids. Therefore, when projecting a Gaussian primitive onto a spatial-only plane, we obtain an elliptical region instead of a single point, as in the current NeRFbased approach. Thus, the corresponding feature for a Gaussian primitive should be a combination of all the areas it occupies in the plane. Secondly, if we follow the self-splitting strategy of 3DGS and split a large Gaussian primitive into smaller ones, they would have different residual features, significantly deviating from those of their parent primitive, which contradicts our original intention, as shown in Fig. 3 . So, finding an appropriate method to encode the region projected by Gaussian is crucial.We propose a scale-aware Residual Field to address the aforementioned issue, which decompose the 4D space into three spatial-only planes 𝑃 𝑠𝑜 and three spatiotemporal planes 𝑃 𝑠𝑡 . Given that the size of Gaussian primitives impacts their projection only within the spatial-only plane 𝑃 𝑠𝑜 , we specifically consider only employing scale-aware encoding within them, as shown in Fig. 2 .For every spatial-only planes 𝑃 𝑖,𝑗 , we employ a MipMap stack to represent features at different spatial scales in the scene. The level 0 of the MipMap stack 𝑃 0 𝑖,𝑗 is a feature map with shape 𝑀 × 𝑁 × 𝑁 ,which has the smallest spatial scale 𝑠 0 among all levels. And remaining levels 𝑃 𝑙 𝑖,𝑗 in the Mipmap stack are obtained by computing thumbnails based on the features of the previous level, where the width and height are reduced by a factor of 2 each. Taking 𝑃 𝑥,𝑦 as an example, the relationship between their spatial scale is as follows:EQUATIONThe variables B 𝑚𝑎𝑥 and B 𝑚𝑖𝑛 respectively represent the maximum and minimum values of the scene's bounding box and 𝑠 𝑙 is the spatial scale of level 𝑙 in the MipMap stack. In practice, we only store and optimize the features in level 0 MipMap 𝑃 0 𝑥,𝑦 , while the remaining levels are dynamically computed and generated during forward inference. This way, we possess the capability to encode features at different spatial scales within the scene.Meanwhile, for a Gaussian primitive with a scaling s in 4D space, when projected onto the spatial-only plane 𝑃 𝑥,𝑦 , it results in a 2D ellipse with axes (𝑠 𝑥 , 𝑠 𝑦 ).Therefore, based on the projected axes of the Gaussian primitive on 𝑃 𝑥,𝑦 and the corresponding base spatial scale 𝑠 0 𝑥,𝑦 of the MipMap stack, we can determine the spatial scale level associated with this Gaussian primitive:EQUATIONTo maintain the highest possible accuracy, we choose the minimum value among them as the final spatial level 𝑙 = 𝑚𝑖𝑛(𝑙 𝑥 , 𝑙 𝑦 ). So now we can obtain the two MipMap features that are closest to its spatial level:𝑃 Here, 𝜓 𝑡𝑟𝑖 ,𝜓 𝑏𝑖 represent trilinear interpolation and bilinear interpolation respectively. Through experiments, we found that summation is a more effective way to combine features in our Scale-aware Residual Field compared to others.Due to the varying temporal locations and lifespans of Gaussian primitives in this 4D space, each Gaussian primitive is sampled with different probabilities over observed time. Dynamic primitives, in order to represent the temporal complexity of the scene, often have a smaller lifespan, resulting in a lower sampling probability compared to static primitives. Primitives with less exposure over the entire temporal domain would have smaller gradients during the backward propagation of the loss function. The gradient value is crucial in the 3DGS framework, as it needs to exceed a threshold to densify the corresponding primitive and optimize the currently imperfectly reconstructed regions. Hence, applying the same optimization and densification strategy directly to each primitive with 3DGS [14] may lead to optimization imbalance.To tackle the aforementioned issue, we propose an Adaptive Optimization strategy, which dynamically adjusts the learning rate and densify gradient threshold for G 4𝐷 𝑖 based on its sampling probability across the observable time range. Specifically, We can use 𝛾 𝑖 (𝑡) of G 4𝐷 𝑖 to calculate the temporal integral within the observable range, representing its sampling probability . The larger the integral, the more the Gaussian primitive's lifespan intersects with the observable range, making it more likely to be sampled. Definite Integral of the Time Domain Distribution. Based on state function 𝛾 𝑖 (𝑡) of G 4𝐷 𝑖 , we can compute its integral over the time domain.EQUATION𝐹 (𝑡) represents the CDF (cumulative distribution function) of each Gaussian primitive. Here, 𝐼 denotes the definite integral over the entire time domain from 𝑡 𝑠𝑡𝑎𝑟𝑡 to 𝑡 𝑒𝑛𝑑 , where 𝑡 𝑠𝑡𝑎𝑟𝑡 and 𝑡 𝑒𝑛𝑑 normalize to 0 and 1, respectively. Since 𝛾 𝑖 (𝑡) is a Gaussian-like function, it is challenging to compute precise definite integral values. Inspired by [26] , we derive the approximate cumulative distribution function as:EQUATIONHere, 𝛼 1 = 0.070565992, 𝛼 2 = 1.5976. Please refer to the appendix for details. Therefore, we can obtain the definite integral of Gaussian primitives over the time domain using Eq. 20 with minimal computational complexity.Intergral-based Per-Gaussian Optimization Schedule. After get the temoral intergral of each Gaussian, we can dynamically adjust the learning rates and gradient thresholds for densification control on a per-primitive basis, aiming to achieve rapid reconstruction of dynamic regions:EQUATIONHere, 𝜅 𝑖 and 𝑙𝑟 𝑖 respectively represent the densification threshold and the learning rate of G 4𝐷 𝑖 , while 𝐼 𝑖 and 𝐼 𝑚𝑎𝑥 respectively denote the timeporal integral of G 4𝐷𝑖 and the maximum time-domain integral among all Gaussian primitives. We adjust its densification threshold each time densification control is required. Additionally, for the learning rate, we dynamically adjust it every 50 iterations based on Eq. 23, involving parameters related to 4D position, scaling, rotation, and zeroth-order SH coefficients of G 4𝐷 𝑖 .Regularization term for scaling residuals: When the sampling time 𝑡 equals the temporal position of G 4𝐷 𝑖 , we aim for minimal variations in the attributes of Gaussian primitives projected into 3D space compared to their initial values in 4D space. Excessive reliance on attribute residuals during the projection process may neglect the optimization of their initial values. Additionally, we strive to minimize the disparity between the initial scaling values of Gaussian primitives in 4D space and their values in 3D space. This ensures that the Scale-aware Residual Field can effectively integrate accurate scale information. To achieve this goal, we propose a regularization term L 𝑆𝑅 concerning scaling residuals:EQUATIONHere, ΔΣ 𝑖 (𝜏 𝑖 ) [: 3] represents the residuals of scaling of G 4𝐷 𝑖 obtained during the projection process when the sampling time is G 4𝐷 𝑖 's temporal position 𝜏 𝑖 , where 𝑛 is the total number of Gaussian primitives in the current 4D space. Total Loss Function Following [14] , we use the loss between the rendered image and the ground truth image, which includes an L 1 term and a L 𝐷 -𝑆𝑆𝐼 𝑀 term. Combined with our regularization terms, the overall loss function is formulated as:EQUATIONThe settings for 𝜆 1 and 𝜆 2 can be referred to in Sec. 5. 
implementation details: 
 We implemented our work using the PyTorch [28] framework and open-source code based on 3DGS. We utilized the nvidiffrast [15] library to compute the MipMap stack in the Scale-aware Residual Field, ensuring computational efficiency. We use Adam optimizer and retained certain implementations from 3DGS, including the fast differentiable rasterizer, hyperparameters, and opacity reset strategy. In the loss function Eq. 25, we set 𝜆 1 to 0.2 and 𝜆 2 to 0.8. Detailed hyperparameter settings can be found in the appendix. We conducted training and testing of our model using a single RTX 3090.To enhance our rendering speed, we adopted a lossless baking strategy for the model during rendering. Since the features extracted from the Scale-aware Residual Field are independent of the sampling time, we can pre-compute the features for each Gaussian primitive in the 4D space. Thus, during rendering, the conversion of Gaussian primitives from 4D space to 3D space incurs only the overhead of MLP inference. Additionally, during rendering, for a given sampling time 𝑡 0 , we filter out points where the survival status 𝛾 (𝑡 0 ) < 0.001. These filtered points indicate that they are inactive and invisible at time 𝑡 0 ), thereby reducing the computational overhead during MLP Decoder and Splatting. After testing, our baking strategy has enabled us to double our rendering speed. 
experiments 6.1 datasets: 
 Synthetic Dataset. We chose D-NeRF [31] as our evaluation dataset for monocular scenes. D-NeRF is a monocular synthetic dataset consisting of eight scenes with large-scale movements and real non-Lambertian material dynamic objects, which imposes a challenge on model performance. Real-world Datasets. We selected Plenoptic Video dataset to evaluate our performance in multi-view real dynamic scenes. Plenoptic Video dataset consists of six real-world scenes, each captured by 15-20 cameras. Each scene in the dataset encompasses complex We have employed a variety of evaluation metrics to assess our model. For rendering quality, we utilize PSNR, SSIM, DSSIM, and LPIPS, and for rendering speed, we measure FPS. All evaluation results are averaged across all scenes in the dataset. For monocular scenes in the D-NeRF dataset, we compare our method against the current state-of-the-art methods [5-7, 9, 31, 45, 47] in the field. The quantitative evaluation results are listed in Tab. 1. The dynamic scene representation method based on NeRF [5-7, 9, 31] struggles with achieving real-time rendering due to the need for dense ray sampling during rendering. However, our method achieves a rendering speed of 182 FPS and exhibits considerable improvement in rendering quality. Compared to existing dynamic Gaussian methods [45, 47] , our approach demonstrates superior performance in handling complex temporal scenes and achieves a certain level of enhancement in rendering quality. Qualitative comparison results can be seen in Fig. 4 . In the standup scene, our method demonstrates better reconstruction of details (such as facial and hand features).For multi-view scenes in the Plenoptic Video dataset, our evaluation results are presented in Tab. 2. Our method outperforms all compared methods in rendering quality while maintaining superior [45] based on deformation fields has shortcomings in modeling complex temporal situations such as object appearance and disappearance, which we can effectively address. Although Spacetime-GS [18] achieves higher rendering speeds, it is only applicable to multi-view scenes, whereas our method is suitable for both multi-view and single-view scenarios. A qualitative comparison of rendering quality with Gaussian methods can be seen in Fig. 5 . Our approach demonstrates more accurate reconstructions in temporally complex scenes (such as the inverted coffee scene with details on the heads and hands) and richer details (such as the circular decorations on the upper garment and reflections on the clippers in the cut beef scene).Temporal properties of 4D Gaussian. Each Gaussian primitive in 4D space possesses temporal properties, including temporal position and lifespan. These Gaussian primitives can be arbitrarily distributed across the temporal domain, giving us an advantage over previous deformation-based methods in handling intricate temporal information. To demonstrate this, we fix the temporal position of each point to frame 0 and integrate the 3D position of Gaussian primitives with the sampling time to extract features in the Redisual field. 3 and Fig. 6 , show a notable decline in performance when temporal position and lifespan are disregarded. Consideration of Spatial Scale in the Residual Field. In our Residual field, we take into account the size information of each Gaussian, improving the accuracy of the features obtained for each Gaussian and aligning with the splitting of Gaussian primitives. To evaluate this approach, we refrain from encoding the projection area in spatial-only planes and conduct assessments on the D-NeRF dataset. As depicted in Tab. 3 and Fig. 6 , not incorporating the size information of Gaussian primitives results in a reduction in the reconstruction quality of dynamic scenes. Adaptive Optimization. To address the imbalance in optimization between dynamic and static regions in the scene, we introduce the Adaptive Optimization strategy. Without this strategy, the reconstruction ability of 4D Gaussians for moving regions would decrease. This is supported by Tab. 3. Without this strategy, Gaussian primitives struggle to distinguish between dynamic and static regions, leading to artifacts as shown in Fig. 6 (e).Regularization of scaling residuals. We conducted evaluations without regularization of scaling residuals to validate the effectiveness of this regularization term. Without this constraint, Gaussian primitives overly rely on the Scale-aware Residual field, neglecting optimization of their own initial attributes. Additionally, it leads Our method achieves high-quality, real-time reconstruction of temporally complex scenes. However, it has limitations. Firstly, the use of explicit and implicit mixing, such as 4D space plane decomposition and MLP inference, may reduce training speed. Secondly, while applicable to both monocular and multi-view scenarios, accurate pose estimation is required. Thus, in scenarios lacking precise pose calibration, such as handheld smartphone videos, performance may degrade. Enhancing reconstruction quality in such scenarios is a future optimization direction. 
dynamic-static segmentation: 
 Static primitives are observable throughout the entire observation period, whereas dynamic primitives, aiming to represent the temporal complexity of the scene, are only visible near their temporal positions. Therefore, dynamic primitives have a shorter lifespan, while static primitives have a much longer lifespan. Consequently, we can segment the scene into dynamic and static parts based solely on their lifespans, without any prior knowledge, as illustrated in Fig. 7 . Our method accurately segments dynamic foreground elements and dynamic lighting effects (e.g., shadows of people in the image). This aspect also underscores the interpretability of our method. 
conclution: 
 In this paper, we propose SaRO-GS as a novel approach for representing dynamic scenes, enabling real-time rendering while ensuring high-quality reconstruction, especially in temporally complex scenes. SaRO-GS utilizes a set of 4D Gaussian primitives to represent dynamic scenes and leverages 3D Gaussian Splatting for real-time rendering. Additionally, we propose a Scale-aware Residual Field to encode the region occupied by Gaussian primitives, resulting in more accurate features and align with the self-splitting behavior of Gaussian primitives. Furthermore, we introduce an Adaptive Optimization strategy to enhance the model's ability to reconstruct high-frequency temporal information in dynamic scenes. Experimental results in both monocular and multi-view settings demonstrate that our approach achieves state-of-the-art rendering quality while enabling real-time rendering. 
: 
 . Our approach achieves higher-quality reconstruction in temporally complex scenes (a) while maintaining real-time rendering (b), with a certain improvement in performance(c). Additionally, we achieve dynamic scene segmentation without any prior information (c).Reconstructing dynamic scenes from video sequences is a highly promising task in the multimedia domain. While previous methods have made progress, they often struggle with slow rendering and managing temporal complexities such as significant motion and object appearance/disappearance. In this paper, we propose SaRO-GS as a novel dynamic scene representation capable of achieving realtime rendering while effectively handling temporal complexities in dynamic scenes. To address the issue of slow rendering speed, we adopt a Gaussian primitive-based representation and optimize the Gaussians in 4D space, which facilitates real-time rendering with the assistance of 3D Gaussian Splatting. Additionally, to handle temporally complex dynamic scenes, we introduce a Scale-aware Residual Field. This field considers the size information of each 
 
 Appendix:  


Decision: The paper presents a novel method focused on reconstructing real-time dynamic scenes in video sequences. Experiments are conducted on both monocular and multi-view datasets.

Since the first round of ratings, all the reviewers agreed on the good quality of the paper in terms of writing, novelty of the contribution and convincing analysis and results presented. Only one of the four reviewers is not expert in the topic. 
Two reviewers (Reviewer vBfU and i357) raised some concerns (e.g., ablation study quality, flickering in the video submitted as supplementary material and about some choices made by authors) but they have been all successfully addressed in the rebuttal. Authors also presented additional analysis and results in the submitted rebuttal.

Given the above remarks, I believe the paper presents a relevant contribution for ACM Multimedia community. My suggestion is also to have it as an oral presentation.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel 4D-GS method to represent dynamic scenes using a scale-aware residual field and an adaptive optimization schedule. The research gaps and proposed solutions are well-presented, and experiments on both monocular and multi-view scenarios demonstrate the effectiveness and robustness of the method.
strength_weakness: strength_weakness: The paper is well written and clearly presents their ideas
Their approach is very novel and effectiveness
Their performance achieves sota performance on public monocular and multi-view datasets
limitations: limitations: 1.Does Figure1 (d) provide the comparison on the rendering quality and Speed on D-Nerf dataset ？ If so, the comparison with the 4DGS  seems to be unfair as they are rendered with the solution of 800
2.The lifespan allows to model the disappearance of objects, but I wonder how it handles the sudden appearance of new objects
3.The adaptive optimization schedule is designed based on the temporal properties, why such a design can benefit the performance of performance? What is the motivation behind this?
4.It seems to be unfair to compare the efficiency with different resolutions as can be seen in table 1? Can you explain the reasons for this?
5.Table 2 does not show the sota rendering speed
6.I feel the ablation study section can be improved. Detailed experiments is preferred to validate the effectiveness of each component, starting from the baseline, which does not adopt any proposed strategies
paper_summary: paper_summary: This article focuses on the task of reconstructing dynamic scenes in video sequences, with a focus on tracking the appearance and disappearance of objects. Lifespan is introduced as a control to better represent the Gaussian function of objects at any moment.
strength_weakness: strength_weakness: The lifespan proposed in this paper is innovative. At the same time, judging from the pictures and tables in the experiment, this paper has certain advantages over other algorithms in terms of effectiveness.
limitations: limitations: 1. From the perspective of the effect, it seems that the introduction of the lifespan parameter has caused shaking and flickering in the video.
2. Will it cause a significant increase in the number of parameters, for example, an object only appears briefly, but its corresponding Gaussian function remains unchanged.
paper_summary: paper_summary: This paper proposes a novel Gaussian-based method to reconstruct dynamic scenes from video sequences. They introduce a Scale-aware Residual Field to handle temporally complex dynamic scenes and propose an Adaptive Optimization schedule to expedite the reconstruction of dynamic regions. Based on their design, the method exhibits high-quality reconstruction and high-efficient rendering.
strength_weakness: strength_weakness: They achieve high-quality reconstruction and real-time rendering simultaneously.

The analysis and design of scale-aware residual field and adaptive optimization, including the spatial projection and leveraging different optimizations for different gaussians, are convincing.

Exhaustive experiments demonstrate its superiority compared to previous methods in terms of reconstruction quality.

The exposition is correct, and the technique can be reproduced.

The writing is clear.
limitations: limitations: The adaptive optimization seems to be capable of accelerating the training, but there is no experiemnt to validate this.

I am not familiar with this field and I cannot find severe limitation of this paper.
paper_summary: paper_summary: The paper proposes SaRO-GS as a dynamic scene representation for real-time view synthesis of dynamic scenes. The method uses a Gaussian primitive-based representation and optimizes the Gaussians in 4D space. To handle temporally complex dynamic scenes, SaRO-GS introduces a Scale-aware Residual Field, which can produce the lifespan and temporal offsets of each Gaussian. In addition, the paper presents an Adaptive Optimization Schedule to adaptively choose the densify gradient threshold and learning rate of each Gaussian. Experiments are conducted on both monocular and multi-view datasets.
strength_weakness: strength_weakness: 1. The paper is well-structured and easy to follow.
2. For the monocular D-NeRF dataset, the proposed method outperforms prior arts such as 4DGS and 4DGS-Realtime by a noticeable margin in rendering quality. The rendering speed is also faster than 4DGS.
3. The paper conducts a detailed ablation study on the proposed components, which demonstrates the effectiveness of each component.
limitations: limitations: 1. The idea of Eq. 9-10 is highly related to the temporal opacity proposed in [18]. Hence, it is suggested to reference [18] at these paragraphs.
2. Line 497-499 mention that the size of Gaussian primitives impacts their projection only within the spatial-only planes. However, from my understanding, the size should also impact the projection on the spatiotemporal planes. For example, s_x and \sigma_i can impact the projection on the x-t plane.
3. In Table 2, NeRFPlayer, HyperReel and Spacetime-Gs use different DSSIM computation setting than the other methods, which will lead to higher DSSIM value. Therefore, the paper should mention that it is unfair to directly compare their DSSIM value with the others'. [18] provides a description on the different DSSIM computation settings, and reports the DSSIM metrics of Spacetime-Gs for each computation setting.
4. In the supplementary video, there is flickering near fast-moving objects (e.g., the cooking torch).
5. Missing references to some advances in view synthesis:
"Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction", "Zip-nerf: Anti-aliased grid-based neural radiance fields", "Neurbf: A neural fields representation with adaptive radial basis functions"