QUEEN: QUantized Efficient ENcoding for Streaming Free-viewpoint Videos

Authors: sharath  girish
tianye  li
abhinav  shrivastava
david  luebke
shalini  de mello
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-01 02:07:41

Abstract: Online free-viewpoint video (FVV) streaming is a challenging problem, which is relatively under-explored. It requires incremental on-the-fly updates to a volumetric representation, fast training and rendering to satisfy realtime constraints and a small memory footprint for efficient transmission. If acheived, it can enhance user experience by enabling novel applications, e.g., 3D video conferencing and live volumetric video broadcast, among others. In this work, we propose a novel framework for QUantized and Efficient ENcoding (QUEEN) for streaming FVV using 3D Gaussian Splatting (3D-GS). QUEEN directly learns Gaussian attribute residuals between consecutive frames at each time-step without imposing any structural constraints on them, allowing for high quality reconstruction and generalizability. To efficiently store the residuals, we further propose a quantization-sparsity framework, which contains a learned latent-decoder for effectively quantizing attribute residuals other than Gaussian positions and a learned gating module to sparsify position residuals. We propose to use the Gaussian viewspace gradient difference vector as a signal to separate the static and dynamic content of the scene. It acts as a guide for effective sparsity learning and speeds up training. On diverse FVV benchmarks, QUEEN outperforms the state-of-the-art online FVV methods on all metrics. Notably, for several highly dynamic scenes, it reduces the model size to just 0.7 MB per frame while training in under 5 sec and rendering at ~350 FPS.

Main paper: 
 introduction: 
 The dynamic world that we perceive around us is not 2D, but rather 3D. Unlike 2D videos, which are ubiquitous, the question of how to effectively capture, encode and disseminate free-viewpoint videos (FVV) of dynamic 3D scenes, which can be viewed at any instance of time and from any viewpoint, has intrigued computer vision and graphics researchers for much time. Free-viewpoint video transmission, if achieved, has the potential to transform and enrich user experience in profound ways by offering novel immersive experiences, e.g., FVV video playback and live streaming, 3D video conferencing and telepresence, gaming, virtual spatial tutoring and teleoperation, among others.The underlying problem of reconstructing FVV involves learning a 6D plenoptic function of a dynamic scene P (x, d, t) from sparse multiple views acquired over a window of time, with x ∈ R 3 being a position in 3D space, d = (θ, ϕ) a viewing direction and t an instance of time. Neural volumetric representations, which learn a 5D plenoptic function of a scene P (x, d) at a fixed time instance, e.g., neural radiance fields (NeRFs) [54] and its variants [2, 7, 18, 55] present a compact and high-fidelity representation for 3D scenes. NeRFs have also been extended to dynamic 4D scenes [1, 5, 19] providing a powerful tool for reconstructing FVV. However, NeRFs require compositing dense information across a 3D volume and hence are slow to train and render. Recently, 3D Gaussian Splatting (3D-GS) [29] has emerged as a promising technique with significantly faster training and rendering speeds in comparison with NeRFs, and they have also been extended to dynamic 4D scenes [42, 84, 89] . While these representations accurately model 4D scenes, they are trained in an offline fashion requiring full multi-view video sequences to learn temporal relationships between frames. They also require long training times to achieve high reconstruction quality and are mostly not streamable.Online FVV, e.g., for broadcast and teleconferencing applications, presents additional challenges versus offline. It requires incremental on-the-fly updates to volumetric representation at each time-step of the dynamic scene, fast training and rendering times to maintain real-time operation, and small packet sizes per frame to enable effective transmission on bandwidth-limited channels. Consequently, the more challenging problem of online FVV reconstruction remains relatively under-explored. Notable prior solutions are those based on NeRFs using voxel grids [37, 78] or triplanes [85] to learn 3D representations that are updated on-the-fly. Unsurprisingly, they suffer from slow rendering speeds. Recently, Sun et al. proposed 3DGStream [69] , which uses 3D-GS to model a 3D scene along with InstantNGP [55] to model its geometric transformation over time. It achieves high rendering speeds but imposes heuristic structural constraints on the volumetric representation to achieve efficiency, which compromises model expressiveness and quality.In this work, we propose a novel QUantized and Efficient ENcoding (QUEEN) framework, which uses 3D-GS for online FVV. Similarly to prior approaches [69] , we also learn Gaussian attribute residuals between consecutive time-steps. To reduce memory requirements, however, [69] learns only a subset of the Gaussian attributes at each time-step, limiting model expressiveness. Our first insight, therefore, is to model residuals for all attributes instead, which does not compromise quality. However, encoding all Guassian attributes increases the per frame memory requirement and hence necessitates a means to compress them more effectively. Our second insight, then, is to learn to directly compress the Gaussian residuals in proportion to the real-time scene dynamics, e.g., motion and illumination changes. This contrasts with existing methods [37, 69, 78, 85] that employ a single fixed-sized structure, e.g., a voxel-grid, a triplane, or hash encoding at all time-steps, and the result is higher efficiency in terms of model size, training speeds, and rendering speeds. Lastly, we also exploit temporal redundancies across time-steps to limit computations to the highly dynamic parts of the scene only and achieve further efficiencies.Specifically, to achieve this, we propose a learned quantization-sparsity framework to simultaneously learn and compress Gaussian attribute residuals for each time-step. We quantize all attribute residuals, except Gaussian positions, via an end-to-end trainable integer-based latent-decoder. Once learned, we efficiently encode the integer latents via entropy coding to achieve high compression factors. For position residuals that exhibit greater sensitivity to quantization, we propose a learned gating mechanism to sparsify them, which identifies the static (corresponding to 0 value) and dynamic Gaussians and retains the sparse dynamic ones only at full precision. Finally, to achieve further efficiencies in terms of training time and storage, we utilize the differences between the 2D viewspace Gaussian gradients of consecutive frames to initialize our learnable gates, and to selectively render local image regions corresponding to highly dynamic scene content.We evaluate our approach, QUEEN, on two benchmark datasets, containing diverse scenes with large geometric motion and illumination changes. QUEEN outperforms all prior state-of-the-art approaches for online FVV and significantly reduces the per-frame memory cost (∼10×), all while achieving higher reconstruction quality, as well as faster training and rendering speeds. Extensive ablations show the efficacy of the various components of our approach.To summarize, our key contributions are:• We propose a Gaussian residual-based framework to model 3D dynamic scenes for online FVV without any structural constraints, which allows free learning of all 3D-GS attribute residuals, resulting in higher model expressiveness.• We introduce a learned quantization-sparsity framework for compressing per-frame residuals, and we initialize and train it efficiently using viewspace gradient differences that separate the dynamic and static scene content.• On various challenging real-world dynamic scenes, we surpass existing state-of-the-art approaches on all metrics: reconstruction quality, memory utilization, as well as training and rendering speed.2 Related Work 
traditional free-viewpoint video: 
 Ever since early FVV work such as [27] , a series of geometry-based FVV methods [12, 57] has been pushing for high reconstruction quality and streamable performance. However, their rendering and compression quality rely on the accuracy of a sophisticated pipeline of geometry reconstruction [20, 28] , tracking [36] , and texturing [61] . They also require high-end hardware for capturing complex and dynamic appearance [6, 14, 26] . Purely image-based rendering [10, 13, 35, 53] relaxes the requirement for geometric accuracy. Although methods such as [4, 96] support view interpolation with layered representations in the dynamic setting, they require a high count of views as input to ensure interpolation quality.Offline Methods. Compared to the traditional representations, the emergence of neural representations [47, 54, 87, 67, 72] opened a new door for capturing FVV for dynamic humans [21, 31, 32, 39, 63, 83, 95] and monocular videos [22, 43, 44, 73, 75, 86] . In this work, we focus on general dynamic scenes [93] from multiple views to push the quality of streamable FVV without requiring a strong human prior [40, 48] or a very constrained input. [16, 62, 74, 81] model the scene dynamics via explicit deformation. Although suitable for motion analysis, they inevitably face a trade-off between motion accuracy and visual quality [74] . To tackle this, [41, 52, 59 ] use a spatialtemporal formulation via time-conditioned latent codes to implicitly encode the 4D scene, enabling reconstruction of topological changes and volumetric effects. [5, 19, 66] factorize the 4D scene into multiple space-time feature planes and achieves higher model compactness and training efficiency. [68, 76] decompose the 4D scene into static and dynamic volumes. [1, 46, 77, 88, 94] incorporate efficient NeRF representations [9, 19, 92] for higher fidelity. Although, these NeRF-based method achieve high compactness, they suffer from low rendering efficiency, even when converted [68] to a more efficient NeRF formulation [9, 55] . Seeing their great potential for efficiency, recent works extend 3D Gaussian representations [29] to dynamic scenes, with temporal attributes [42, 90] , generalized 4D Gaussians [89] and a hybrid representation [84] . While these methods achieve high quality in modeling 3D dynamic scenes, they, together with the aforementioned NeRF-based methods, are mostly offline, i.e., they require all the input video and a long time for training, which is inherently difficult for streaming applications.Online Methods. Online reconstruction for FVVs is relatively under-explored, as it imposes additional challenges of on-the-fly reconstruction using only local temporal information instead of the full recordings. Furthermore, toward the goal of streamable FVVs, the encoding system is evaluated by multiple metrics including compression rate, encoding and rendering speed and visual quality.[50] tracks dense 3D Gaussians by solving their motion over time. Visual quality and dynamic appearance is not their focus. [23] models motion by rendering scene dynamics, however their method is not optimized for efficiency. [45] focuses on generalizable NeRF reconstruction and shows good promise to adapt to a new frame but has a high memory footprint due to an MVSNet-style neural network [8, 91] . [37] accelerates training and rendering speed with a special tuning strategy and sparse voxels, however, their representation still has high temporal redundancy. [82] proposes an incremental training scheme with natural information partitioning and achieves high compression, but its encoding is slow. Several works [79, 80, 85] use video codec-inspired encoding paradigms for data efficiency. [79] achieves a decent compression rate and near interactive rendering with compact motion and residual grids. However, their training requires 10 minutes per frame. [80] focuses on real-time decoding, streaming and rendering instead of on-the-fly encoding. [85] performs grouped training on a hybrid representation of triplanes and volume grid. While achieving high compression rate, their fixed encoding paradigm and aggressive quantization limits their reconstruction quality along-with low rendering speeds. [69] is the closest work to ours for streaming FVV via 3D-GS. They encode the position and rotation residuals via an Instant-NGP [55] can approximate only small changes in new scene content or lighting variations. Our work updates and compresses all 3D-GS attributes freely without any structural constraints while still obtaining much better memory costs via our quantization-sparsity framework along with better training times.Several works propose a variety of compression methodologies for reducing the memory, training time, or rendering speed of standard static scene 3D representations. [9, 71] decompose NeRFs via lowrank approximations. [15, 38] prune voxels along with vector quantization by [38] . [25, 70] compress multi-resolution feature-grids via codebook/vector quantization. A large number of approaches target 3D-GS compression [17, 24, 34, 58] and acceleration via pruning. While these approaches can be applied to static representations on a per-frame basis, their trivial frame-wise application would result in extremely high training costs as well as large memory per frame. To enable streaming, our work, instead, explicitly focuses on effectively leveraging the temporal redundancies across frames by compressing the residual information between them to achieve greater efficiency. 
queen: quantized efficient encoding for streaming fvv: 
 A solution for streamable FVV must have low-latency encoding (training) and decoding (rendering), and low data bandwidth (memory) for transmission on a common network infrastructure. Motivated by these constraints, we aim to generate streamable FVVs with compact representations that are fast to train and render incrementally. In this section, we first provide an overview of 3D-GS (Sec. 3.1). In Sec. 3.2, we propose a compression framework to efficiently represent and train Gaussian attribute residuals at each time step. Sec. 3.3 discusses utilizing an approach based on viewspace gradient differences to achieve greater efficiencies. An overview of our method is shown in Fig. 1 .Our efficient representation for dynamic scenes is based on 3D Gaussian Splatting (3DGS) [29] . Given multi-view images I, a 3D scene is modeled by a set of Gaussians with attributes A.Representation. The shape of each Gaussian i is defined by its mean p i ∈ R 3 and covariance matrix Σ i . The covariance matrix is represented byΣ i = R i S i S T i R T i, where R i is a rotation matrix parameterized by a quaternion vector q i ∈ R 4 , and the scale matrix S i is a diagonal matrix with elements s i ∈ R 3 . Each Gaussian also contains opacity o i ∈ [0, 1] and spherical harmonic coefficients h i for view-dependent appearance with dimensions based on the number of degrees.Rendering. For rasterization, 3D Gaussians are projected into 2D Gaussians for any given view. Given a camera with intrinsic matrix K and viewing transform W, the 2D mean and covariance are:EQUATION)where Π(• ) denotes the perspective projection and J is the Jacobian of the affine approximation of the projective transform [97] . The image color ĉ at pixel location x is obtained by blending N depth-sorted Gaussians with their view-dependent RGB color value c i computed from h i :EQUATION)where α i is the conic opacity of Gaussian i at pixel location x multiplied by the Gaussian opacity o i .Training. With a differentiable rasterizer [29] , the attributes A = {p i , q i , s i , o i , h i } N i=1 are optimized to produce renderings Î = R(A) that fit to input images I by optimizing a reconstruction loss which combines the D-SSIM loss [65] and L 1 loss with a hyperparameter λ:L = λL D-SSIM + (1 -λ)L 1 .(3)Given a multi-view image sequence {I t } T -1 t=0 , our goal is to reconstruct the dynamic scene via Gaussian attributes A t for each time-step t. We model the attributes based on the trained attributes from the previous time-step t -1 asEQUATION)where R t consists of learnable residuals for each attribute (in Fig. 1 (gray block)). For time-step t = 0, we perform vanilla Gaussian splatting training to obtain attributes A 0 . This sequential formulation allows us to freely and adaptively update the residuals R t on-the-fly with incoming streaming training views, without any structural constraints as in prior works [69] . However, representing the 4D scene with uncompressed residuals is still highly inefficient. As residuals have low magnitudes in comparison with the attributes themselves, they can be efficiently compressed, for which we propose a novel quantization-sparsity framework.There exists spatial redundancy within the Gaussian attributes of the same time-step. Nearby Gaussians have highly correlated residuals for shape, orientation and appearance. To reduce the storage cost of the residuals, we propose to utilize a quantization framework during training [24] .At each time-step t, we represent the residuals via quantized latents and a shared compact decoder. Specifically, to obtain the residuals for each category2 r i ∈ R M , we maintain corresponding quantized integer latents l i ∈ Z L for each Gaussian i. These latents are passed through a shared linear decoder D with learnable parameters D ∈ R M ×L to obtain the decoded attribute residual r i . Such a compact decoder has small time and memory costs due to few parameters and arithmetic operations. To allow differentiable training of the integer latents via gradient optimization, we use a continuous approximation li ∈ R L instead. li are rounded to the nearest integer values for the forward pass but can still receive backpropagated gradients via the Straight-Through Estimator (STE) [3] :EQUATIONThe continuous latents L = { li } N i=1 , and the shared decoder's parameters D are learnable during training. After adding the decoded residuals to the previous time-step's attributes (Eq. 4), the standard rasterization process (Eq. 2) is used to obtain the rendered image. This differentiable quantization module is trained end-to-end with the main training process by optimizing the reconstruction loss. Post-training, we entropy code the quantized latents L and directly store the decoder D. Entropy coding results in as much as 10× reduction in model size from 44 to 4 MB without quality degradation.Sparse Representation. While most of the attribute residuals can be quantized effectively with our proposed method in Sec 3.2.1, we observe that the position residuals are sensitive to quantization and require high precision during rendering 3 . Storing all the full-precision position residuals, however, still results in high per-frame memory costs. To tackle this, we propose a learned gating methodology, which enforces sparsity in the residuals instead of quantization. This mechanism allows us to set a vast majority of the position residuals to zeros, while maintaining full-precision non-zero values. Specifically, we represent the positional residual for each Gaussian i as ∆p i = g i • l pi , where the scalar g i is the learnable gate variable and l pi ∈ R 3 is the learnable pre-gated residual in full precision during training. After training, the sparse ∆p i can be efficiently stored via sparse matrix formats [60] to reduce memory costs. Thus, our goal is to encourage the sparsity for the variable g i across allGaussians. This goal also aligns with the observation that a large portion of a dynamic scene is static or nearly static, which can be leveraged to attain high compression performance.Hard Concrete Gate. Sparsity can be induced via L 0 or L 1 norm regularization penalties. However, L 1 norm induces shrinkage, i.e., lowers the magnitude of even non-zero values. L 0 norm is the ideal sparsity loss without shrinkage, but is computationally intractable with non-differentiability and combinatorial complexity. To enforce sparsity, we instead propose to use the hard concrete gate [49] . For each Gaussian i, the concrete gate [51] is a continuous relaxation of the Bernoulli distribution: ĝi = Sigmoid( log αi /τ), (6) where α i is a learnable parameter and τ is the temperature parameter. Although the concrete gate approximates the discrete Bernoulli gate, it does not include the end points {0, 1}, which does not directly result in sparsity. The hard concrete gate "stretches" the range of the concrete gate to the interval (γ 0 , γ 1 ) and then applies a hard-sigmoid:EQUATIONThis includes the end points {0, 1} for g i , required for achieving sparse residuals.Sparsity Loss. The hard concrete gate formulation leads to a convenient regularization loss for encouraging L 0 sparsity [49] in the gates g i :EQUATION)Here, we treat α = {α i } N i=1 to be learnable parameters for all N Gaussians at a given time-step and {τ, γ 0 , γ 1 } as hyperparameters that are shared for all Gaussians and all time-steps. Real-world dynamic scenes contain high amounts of temporal redundancy with only a fraction of the content changing between consecutive time-steps. The proposed quantization-sparsity framework can learn to identify Gaussians corresponding to static scene content and set their residuals to 0. However, they still forward/backward pass through static regions resulting in wasted training computation. Additionally, initializing the gates with 1s requires more iterations for convergence. We thus propose a proxy metric to identify Gaussians, which are static or dynamic at the start of training. We use this metric to initialize our gates while also identifying dynamic image regions to perform local rendering in, during training.Viewspace Gradient Difference. The ground-truth (GT) training images contain information of the dynamic scene content, which we leverage to separate static and dynamic Gaussians. A simple pixel difference between consecutive frames does not account for illumination changes and is a noisy signal for the geometric position residuals. 3D-GS utilizes 2D viewspace gradients ∂L ∂p ′ to identify poorly fitted Gaussians based on the reconstruction loss L between the rendered ( Î), GT image (I).More concretely, after training Gaussians at time-step t -1 with an MSE loss, L t-1 and 2D Gaussian means p ′ t-1 , we compute the MSE loss for the next time-step L t and then compute the gradient difference. The score vector d t is the average of gradient differences across all training views v:d t = 1 V V -1 v=0 ∂L (v) t ∂p ′ (v) t-1 - ∂L (v) t-1 ∂p ′ (v) t-1 , L (v) t-1 = L I (v) t-1 , Î(v) t-1 , L (v) t = L I (v) t , Î(v) t-1 . (9)As shown in Fig. 2 , d t identifies the dynamic scene regions while factoring out the noise from imperfect reconstructions at time-step t -1. We use the norm of the score vector |d ti | to initialize the gate parameters. We define the probability of a gate being active for Gaussian i at time-step t asEQUATION)We set p i in Eq. 8 to be d ti to solve for the initial α i . This initialization leads to better convergence by identifying Gaussians corresponding to 0 position residuals at the start of training itself.Adaptive Masked Training. In addition to gate initialization, we propose to utilize the score vector d t for an adaptive masked training scheme. We split the Gaussians into static or dynamic parts by applying a threshold t d on the norm of d t . We render dynamic Gaussians for each training view to identify corresponding dynamic image regions. We then only render and backpropagate through these pixel locations. We perform this masked training for a fraction of the full training iterations, and find it to improve training speeds with little to no loss of reconstruction quality.Initial Frame Reconstruction. For an incrementally updating online approach, it is important to make sure the initial frame is well reconstructed. COLMAP used to initialize the positions of Gaussians can result in sparse 3D points for regions with sparse camera views. Hence, we use an off-the-shelf monocular depth estimation network to estimate point locations in these empty regions and predict a more complete initial point cloud. Further details and results are in the supplementary.End-to-end Training. We train separate decoders and quantized latents {L c , D c |c ∈ {q, s, o, h}} for all attributes except position. For position, we learn the gate parameters α and positional residuals L p . All variables are end-to-end differentiable. The total loss function that we minimize is the reconstruction loss (Eq. 3) and the sparsity gate regularization loss (Eq. 8):EQUATIONwhere λ reg controls tradeoffs between memory and reconstruction quality. By simultaneously quantizing while training we achieve high compression while maintaining quality, unlike [85, 79] with post-training compression that lead to quality degradations. We also apply the 3D-GS densification stage at each time-step and is sufficient in modeling new or finer scene content. 3DGStream [69] adds Gaussians relative to the first time-step only, which limits their approach to small scene changes.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We provide details including computational resources (hardware), training times and dataset specifications in the implementation details in the main paper and the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
datasets and implementation: 
 We evaluate our method on two challenging FVV video datasets. (1) Neural 3D Videos (N3DV) [41] consists of six indoor scenes with forward-facing 20-view videos.(2) Immersive Videos [4] consists of seven indoor and outdoor scenes captures with 46 cameras. In both datasets, the central view is held out for testing. We implement QUEEN on [29] . We train for 500 and 350 epochs for the first time-step, and for 10 and 15 epochs for the subsequent time-steps, for N3DV and Immersive, respectively, on an NVIDIA A100 GPU. One epoch contains all training views. We evaluate visual quality in terms of average frame-wise PSNR, SSIM, and LPIPS (VGG) across all videos. We also compute the average storage size and training time for each time-step, and the rendering speed. Additional details are provided in the supplementary materials.We compare QUEEN against state-of-the-art existing online FVV methods (3DGStream [69] , StreamRF [37] and TeTriRF [85] ) on N3DV and Immersive (Tab. 1). 3DGStream [69] is the overall best-performing prior method. Since 3DGStream [69] was originally run on an older NVIDIA V100 GPU on N3DV, we re-run 3DGStream on an NVIDIA A100 GPU on both N3DV and Immersive and denote it as 3DGStream* in Tab. 1 for consistency with QUEEN. For brevity, in Tab. 1 we additionally compare against only selected top-performing offline FVV methods. We include a more extensive comparison to all existing offline FVV methods in the supplementary (Tab. 8). Lastly, we evaluate three variants of QUEEN: QUEEN-s (small), QUEEN-m (medium) and QUEEN-l (large), with residuals trained for 6, 8 and 10 epochs, respectively. From Tab. 1, on N3DV, QUEEN-l results in the best quality among all online FVV methods and achieves a 10× reduction in storage size compared to 3DGStream. Although TeTriRF requires less memory than QUEEN, it has much worse quality (-1.5dB) and rendering speed (4FPS), and higher training time (39 sec). On Immersive, which contains more pronounced scene changes than N3DV, we limit our comparisons to 3DGStream with longer iterations. TeTriRF requires long convergence times to achieve reasonable reconstruction quality, limiting their training feasibility. QUEEN-l significantly outperforms 3DGStream, obtaining +4dB PSNR, 5× smaller size, and lower training times. These results on the more challenging scenes from the Immersive datasets reveal the structural constraints brought by the heuristic compression design of 3DGStream. In contrast, our quantization-sparsity framework shows higher flexibility and quality in capturing changing appearances and scene density as well as learning compact and effective representations.In Fig. 3 we compare the reconstruction results of the various methods. On N3DV, we reconstruct finer details than 3DGStream, e.g., the hand and the dog, and minimize artifacts such as the tongs in the top scene or the coffee and metal tumbler in the bottom scene. TeTriRF produces blurry outputs, e.g., the cap or metal tumbler in the bottom scene. On Immersive, we better model illumination changes and new scene content such as the person (first patch) and the flame (third patch) in the top scene or the face in the bottom scene (second patch) versus 3DGStream. Effect of Updating Appearance Attributes. To ablate the importance of updating the Gaussian appearance attributes (color and opacity) per frame in QUEEN, we run experiments on N3DV for 2 settings: (1) learning only geometric attribute residuals (position, scale and covariance) with appearance residuals set to zero and (2) learning all residuals per frame (Tab. 3). Updating only geometric attributes results in a drop of 0.4 dB PSNR versus updating all attributes.Visually, for the Flame Steak scene in N3DV (Fig. 4 ), updating all attributes results in the highest quality, while fixing opacity introduces artifacts at the edge of the flamethrower. Fixing color, additionally, results in a significant drop in PSNR (-0.6 dB) producing a discolored flame (rightmost column).  
conclusion: 
 We proposed QUEEN, a framework to model 3D dynamic scenes for online FVV using 3D-GS.We utilized an attribute residual framework, which freely updates all parameters leading to better modeling of complex scenes. We show that the residuals can be successfully compressed via our learned quantization-sparsity mechanism, which adapts to the dynamic scene content to achieve very small model sizes, improved training and rendering speeds, and improved visual quality. In future work, we aim to extend QUEEN for sparse view reconstruction or sequences with long duration. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: Please see Appendix D where we discuss the potential societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Our method is evaluated on several datasets. We followed their license and we have credited and cited their work and datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] .Justification: N/A.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: N/A.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: N/A.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide supplementary results (Appendix A), additional implementation details (Appendix B), discussion on limitations and future work (Appendix C) and the broader impact (Appendix D) of our approach. We recommend the reader to watch the supplementary video hosted on our project website: https://research.nvidia.com/labs/amri/projects/queen for a visual comparison of the results of the various methods as well as more details of this project.A.1 Quantization vs. GatingWe evaluate the effect of the gating framework in comparison to the quantization framework for the position residuals. We perform gating or quantization on the position residuals while quantizing all other attributes. Results, averaged on the N3DV dataset, are shown in Figure 7 . We consider the effect of varying different loss coefficients to trade off between accuracy and memory.In Figure 7 we explore the tradeoff between PSNR and size by varying the number of training iterations. We can also control the amount of sparsity in the scene by varying the λ reg loss coefficient.As visualized in Fig. 8 (b), we find that increasing λ reg leads to higher sparsity or lower memory but also lower reconstruction quality.We further experiment with an additional regularization loss to reduce the entropy of the latents. We observe that lower entropy corresponds to lower memory, but also lower reconstruction quality. While learnable probability models can successfully reduce entropy, as shown by [25] , these models have higher time and memory costs during training. We instead observe that the probability distribution of the various attribute residuals at each time-step is unimodal and is close to a Laplacian or Gaussian distribution. As a unimodal distribution has entropy proportional to the variance [11] , we enforce a loss on the standard deviation of the latents with a tradeoff parameter λ std controlling the effect of this regularization loss. Fig. 8 (a) shows results on the N3DV dataset by varying λ std . We observe that increasing λ std reduces the entropy costs, leading to lower memory costs, but lower reconstruction quality, and vice versa.We provide additional analysis on the effect of latent dimension for the various attributes in Table 6 .In general, latent dimension does not have a significant effect on reconstruction quality or model size. Increasing the latent dimension can lead to lower per-dimension entropy due to our learnable quantization framework and hence still maintains the overall total size for the latent. We find that varying the total number of iterations (Appendix A.1) or the entropy loss/variance coefficient (Appendix A.2) are more effective knobs for trading off between quality-memory or quality-time. Frame DifferenceFigure 9 : Adaptive Sizes. Our quantization-sparsity framework automatically allocates more memory for frames with larger scene changes (higher frame difference). Notice that the frame difference spikes in the Immersive scenes (bottom row) correlate with frame-wise model size, which increases to allow for modeling of more temporal variations.A key advantage of our quantization-sparsity framework is its adaptability to scene content. We visualize the per frame sizes for 2 scenes each from N3DV and Immersive along with the visual frame difference between consecutive frames in Figure 9 . We see that our approach allocates variable model sizes for each frame unlike 3DGStream [69] , which uses a fixed-sized InstantNGP structure. Additionally, higher frame differences result in larger model sizes and vice versa, as seen in the top row. This shows that our method is capable of allocating more bits to frames with large scene changes. This is especially evident from the spikes in Immersive's scenes in the bottom row, which correlate with the model size.Next, we show the stability of our approach at recovering from large scene variations corresponding to the frame difference spikes as mentioned above. We visualize the reconstructed test-view PSNR for each frame, for 2 scenes each from N3DV and Immersive, along with the frame difference between consecutive frames in Figure 10 . A large L1 error such as around frames 175 (top left), frames 225 (top right), frames 75 (bottom left) or frames 90 and 290 (bottom right) does lead to drops in PSNR. However, our PSNR recovers in subsequent frames showing the stability of our framework with large scene variations present.A.5 Effect of Improved Point Cloud Initialization.Consistent geometry for the 3D scene in the first frame is important to learn accurate residuals for the attributes of the subsequent frames. The COLMAP-generated point cloud initialization can be incomplete for regions that are textureless or are not sufficiently captured in multiple cameras. This is visualized in the top row of Figure 11 . The boundaries of the scene consist of limited training view cameras as shown by the white box leading to sparse or no points in these regions by COLMAP initialization. The densification stage in 3DGS is unable to recover from this producing erroneous rendered depth or geometry and also leads to low quality image reconstruction. Frame Difference Therefore, we propose to use an off-the-shelf monocular depth estimation network [64] to predict a more complete initial point cloud. However, due to the scale-shift ambiguity of monocular depth estimation, we align the predicted monocular depth with the true scene depth from existing COLMAP points. To do so, we estimate the 2D pixel locations p ′ i of each COLMAP point i by projecting points from 3D world space to 2D screenspace.EQUATIONwhere Π(.) denotes the perspective transform described in Eq. 1 of the paper. We then query the pixel locations p ′ i in the monocular depth image to obtain the corresponding depth values ẑi . This is aligned with the GT depth value from the COLMAP 3D points z i by a least-squares optimization to obtain the scale and shift parameters α, β. We then obtain the aligned dense depth map αẑ i + β.To identify regions with empty COLMAP initializations, we iterate over training views and render the corresponding image along with an alpha mask calculating the accumulated transmittance at each pixel location. Mask values below a threshold t z (0.10 for N3DV and 0.03 for Immersive) are identified to obtain the corresponding pixel locations containing few COLMAP points. We then use the aligned depth values corresponding to these pixel locations to re-project back into the world space.As seen in the row of Figure 11 , the depth map initialization produces more dense points in empty regions. These points maintain consistent depth with existing COLMAP points. Such an initialization results in improved image reconstruction quality with ∼3dB improvement in PSNR for the corresponding view while also producing better consistent depth or geometry. As we utilize the network depth at initialization only, we do not require high-quality depth networks and a coarse depth is sufficient for sampling new points. The training can then learn to move the to produce finer scene depth. This also results in minimal increases in training time as it's a one-time operation at the initial time-step and produces a small number of additional points in empty regions.We show quantitative results for 2 configurations with and without depth map initialization for the datasets of N3DV and Immersive in Tab. 7. We show PSNR for the central test view as well as for the first train view for all scenes. We also show the number of points at the end of training the first frame with and without initialization along with the average training time per frame for the full scene. The depth initialization does not significantly affect the test view PSNR on N3DV as it corresponds to the central view with a large overlap with many training views. In contrast, the PSNR for the training view (also in Fig. 11 ) improves considerably (+1.3dB) with the depth initialization highlighting the efficacy of the approach in improving geometry for regions with sparse camera views. Additionally, there is almost no overhead cost as we obtain similar number of Gaussian points at the end of training the first frame. Training time for the full video is also only marginally higher. On Immersive, we observe a 0.5 dB improvement in PSNR for the test view while the train view PSNR shows minor improvements. Again, the number of Gaussians and training time is not significantly affected by the initialization with additional depth based points.We show comparisons to additional offline FVV baselines on the N3DV dataset in Table 8 for the sake of completeness. This is a superset of the Tab. 2 in the main paper. A majority of the works compute SSIM using the scikit-image implementation, which tends to produce higher values different from our SSIM implementation similar to MipNeRF [2] . Since the two are not comparable, we exclude implementation numbers computed with scikit-image. We also only show results for LPIPS on VGG for the methods that use it. This is consistent with how we compute LPIPS. In addition to the average quantitative results over the full datasets of N3DV and Immersive in Tab. 8, we show results for each scene in both the N3DV and Immersive datasets of various frame-wise metrics, including, PSNR, SSIM, LPIPS, size, training time and rendering time (FPS) in Tables 9 and 10 .In addition to the extensive quantitative analysis in the paper as well as supplementary, we conduct an A/B user study to measure the perceptual quality of our video reconstructions. For each vote, we show a pair of randomly chosen rendering results (from a test view that is not used in training) by our method and one of the baseline methods (3DGStream [69] and TeTriRF [85] ). We also show the ground truth video as a reference for the participants to make the decision. We ask the participant to choose the method that more faithfully matches the reference video. In total, we collected 285 responses from 15 participants within the timeline of the rebuttal. For the N3DV dataset, 76.67% of users preferred our method over 3DGStream and 96.67% preferred our method over TeTriRF. On the Google Immersive dataset, 97.14% of users preferred the reconstructions from our approach over that of 3DGStream. This showed that the participants strongly prefer our results in comparison to the baseline methods for both datasets.Training. Our implementation of QUEEN builds on that of [29] . We train the Gaussians for 500 and 350 epochs for first time-step, and for 10 and 15 epochs for the subsequent time-steps, on N3DV and Immersive, respectively, with each epoch consisting of all training views. We set the SH degree to 2 for N3DV and 3 for Immersive. We set the score vector threshold t d = 0.001 for all experiments. We additionally dilate the image mask by a 48 × 48 kernel to include neighboring image regions while rendering as larger Gaussians can depend on multiple pixel locations. We perform masked training for 30% of the iterations for N3DV and 65% for Immersive. We perform masked training for only a fraction of iterations as updating Gaussians rendered only at masked locations can alter the rendered pixels at unmasked location as well. We thus allow fine-tuning on the full image to account for any changes in the unmasked image regions. Sparse Gating. We learn the parameters α with the Adam optimizer [30] . The position residual learning rate is set to 0.00016 for N3DV and 0.0005 for Immersive. Other hyperparameters are provided in Table 11 .Quantization. For both datasets, we set the learning rate of the decoder parameters to be 0.001 for the color and rotation attributes and 0.0001 for opacity and scaling, optimized with the Adam optimizer. Other hyperparameters are provided in Table 12 .Densification. For N3DV, we perform densification for subsequent frames from epoch 6 until 80% of the epochs with an interval of 2 epochs and a gradient threshold of 0.00125. For Immersive, we perform densification on the 8th epoch with a gradient threshold of 0.00125. First-frame Quantization. Uncompressed Gaussians have large memory costs even for the first frame. However, quantizing all attributes results in quality degradations [24] . We therefore apply learnable quantization only on the first frame's high-frequency spherical harmonic coefficients (excluding the DC component) using the hyperparameters mentioned above. For example, on N3DV, we reduce the first frame's size from 47 to 17 MB with no quality degradation.Storage. Post-training, we convert the learned parameters to the final compressed form. For the quantized residuals of rotation, scale, opacity and appearance, we convert the continuous latents L c to the integer form and then apply entropy encoding and store this further-compressed representation E c as well as the decoder D c for all four categories c ∈ {r, s, o, h}. Our entropy coding approach flattens our integer latent matrix for each attribute before encoding. For example, for L-dimensional latent attributes for N gaussians, we flatten the matrix to obtain a vector with L*N elements. This integer vector is then encoded using standard entropy coding approaches such as arithmetic coding. The number of bits for storing each attribute residual matrix is therefore dependent on the scene content as it relies on the amount of motion. This number can be fractional, on average, which is the standard for the entropy coding algorithm of arithmetic coding or Huffman coding [33] . For example, for the Sear Steak scene in the N3DV dataset, on average, we require 0.68 bits for all the quantized attributes (corresponding to 0.5MB/frame). This depends on the entropy of the latents itself, which varies with changing scene motion (Figure 9 ).For the sparse gates, we store the positional residuals as a sparse matrix with the indices from binarized gate variables I = {i|g i ̸ = 0}, and the full-precision residual vectors only if its corresponding gate is on, E p = {l pi |i ∈ I}. Both operations add a negligible computation overhead. This corresponds to the coordinate format (COO) for storing sparse matrices where the non-zero values are stored in FP-32 precision along with their integer index locations. We set different hyperparameters for the two datasets in in Tables 11 and 12 to account for the widely varying amount of scene motion between N3DV and Immersive datasets. The Immersive dataset contains larger and more complex scene motions (e.g., a person entering and leaving the scene) while N3DV contains relatively minor motions. We found that a higher learning rate for the position residuals allows Gaussians to adapt to the highly dynamic scenes. The gating hyperparameters in Table 11 for N3DV are set to utilize this prior information about the dataset where the stretch hyperparameters γ 0 and γ 1 are set closer to 0 to enforce more sparsity in the position residuals. Additionally, the Immersive dataset itself consists of a wide variety of indoor/outdoor scenes at varying scales/scene motion/illumination. We use the same set of hyperparameters for each scene achieving good reconstruction quality for every scene (Table 10 ) showing its generalization capability.Tables 11 and 12 list out the different hyperparameters for quantizing and sparsifying residuals for both N3DV and Immersive datasets. To test the sensitivity of reconstruction quality to hyperparameters, we train our method on the N3DV dataset with two sets of hyperparameters. The first configuration uses the stated hyperparameters for N3DV from Tables 11 and 12 , while the second utilizes the hyperparameters corresponding to Immersive while also matching the learning rate for the position residuals (0.0005). We show results on N3DV datasets in Table 13 . We see that the Immersive dataset's hyperparameter configuration still achieves similar PSNR as the original hyperparameters for N3DV. While the model size is higher (1.49 MB) with the Immersive configuration compared to the original configuration (0.60 MB), it is still much lower than the prior state-of-the-art 3DGStream (7.8 MB) while maintaining higher reconstruction quality in terms of PSNR. prior work, we downsample videos by a factor of 2 to obtain a resolution of 1280 × 960. We evaluate on 7 scenes (Welder, Flames, Truck, Exhibit, Face Paint 1, Face Paint 2 and Cave) with the central view held out for testing. We extract the first 300 frames for all scenes except for Truck, which consists of 150 frames. We undistort the fisheye views into perspective views using the distortion parameters. We train and evaluate on the perspective views with pinhole camera parameters.Baselines. 3DGStream [69] : We use the official codebase4 from 3DGStream [69] . We use the same default configuration for N3DV as provided by the authors. For Immersive, we reduce the gradient threshold for densification to 0.0075 to allow for more Gaussians while increasing the training iterations for Stage 1 and 2 to be 450 and 250 iterations, respectively. TeTriRF [85] : We use the official codebase 5 from TeTriRF [85] for all experiments on N3DV.Measuring FPS. We compute FPS for all frames of the video and report the median value in our experiments including the time taken to decode the residuals. Note that the decoding is a one-time operation per step and rendering for a preceding frame can be performed while learning the residuals of a subsequent ones to achieve even higher speeds.For efficiency and on-the-fly training, we encode sequences by learning inter-frame residuals. However, for FVVs of long duration or drastic scene update, per-frame training will face challenges in reconstruction capability. Unlike offline reconstruction, per-frame training does not have access to future-frame information. This setup limits the capability to effectively reason about large scene changes (e.g., topological changes and highly varying appearance) [56] . If an object suddenly appears or disappears, it is more difficult to (de-)allocate and update scene parameters to capture such changes. In the context of Gaussian splatting, it is challenging to schedule densification and pruning of the Gaussians. Future work could address this by designing an efficient keyframing technique for identifying large changes in the scene and allocate longer training times accordingly.Furthermore, most current FVV encoding paradigms rely on the input multi-view videos for reconstruction. Exploring a general prior of the dynamic scenes, e.g., generative video models, is a promising direction for reducing the dependence on coherent multi-view input, as the video prior could regularize the reconstructed FVV to capture reasonable scene dynamics even if some input views or frames are missing. Moreover, extending our approach to a single or sparse view scenario is a challenging yet important problem for further democratizing streamable FVV. We will leave these directions for future work.We consider our work as a neutral technology. This proposed method reconstructs free-viewpoint videos from user-provided video inputs. As we highlighted in the introduction, this technology can improve many aspect of people's lives, such as through healthcare (tele-operation) and communications (3D video conferencing). There is indeed a possibility that this work can be misused. Since our reconstruction completely relies on the video inputs, the most likely cases of misuse are those where the input video (provided by the users) have negative impacts.NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: Yes, the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).r ∈ R, belong to one of five categories of Gaussian attributes: position, rotation, scale, opacity and color.Concurrent work[58] also discovered that positional attributes are sensitive to compression.https://github.com/SJoJoK/3DGStreamhttps://github.com/wuminye/TeTriRF 


Decision: The paper introduces QUEEN, a framework for efficient streaming of free-viewpoint videos using 3D Gaussian Splatting. It focuses on high-quality reconstruction, fast training, and reduced model size. QUEEN outperforms state-of-the-art methods on various benchmarks, particularly in dynamic scenes.
We are glad to accept the paper, based on the reviewers' comments which are all positive.
We encourage the authors to include the suggested revisions by the reviewers in their camera-ready version (e.g., clarify the quantization process and experimental details, provide a more detailed analysis of the results, address hyperparameter sensitivity, ensure code release for reproducibility, and discuss limitations and future directions).

Reviews: review: paper_summary: paper_summary: This paper introduces QUEEN, a framework designed to enable fast encoding (training) and decoding (rendering) for online free-viewpoint video (FVV) streaming using 3D-GS. Achieving high frame generation quality alongside real-time seamless streaming and rendering is challenging due to the intensive computation and large data volumes involved in this process. Towards this, QUEEN first proposes to apply all attribute residuals, unlike the state-of-the-art works that only use a subset, thereby maintaining high video quality. QUEEN also incorporates several components such as attribute residual quantization and position residual sparsity to accelerate the encoding and rendering process and reduce model size. Additionally, QUEEN leverages inter-frame redundancy, an important feature missed by previous works, to further enhance FVV generation efficiency. The evaluation demonstrates that QUEEN can reduce training time to only 5 seconds and achieve a rendering speed of around 350 FPS with a model size of merely 0.7 MB, all while maintaining near-optimal video quality.
strength_weakness: strength_weakness: - Experimental Settings: The paper lacks details on some crucial experimental settings. For example, the number of bits used for attribute residual quantization is not specified. Similarly, for sparse position residuals, the precision is mentioned as full precision, does that mean fp32? For such case, could using bfloat16 (which reduces model size by half) solve the memory footprint issue without necessarily employing sparsity techniques?
- Evaluation Analysis: The analysis of evaluation results is insufficient. For instance, Table 2 (which is not referenced in the text), shows that introducing quantization and sparsity increases the quality (PSNR). Intuitively, these techniques should reduce quality, so an explanation is needed. Additionally, despite introducing extra processing steps (such as quantization, entropy encoding, etc.), training and rendering times do not increase. Detailed analysis/explanation on these evaluation results would be very beneficial.
- Perception of Quality: While numerical metrics like PSNR, SSIM, and LPIPS quantify the video quality, they may not fully reflect user perception of quality. A user study, though challenging, would significantly strengthen the paper.
questions: questions: - Can you provide more details on the experimental settings for attribute residual quantization and sparse position residuals? Specifically, what number of bits is used for quantization, and is fp32 used for sparse position residuals?
- Can you explain why introducing quantization and sparsity improves quality (PSNR) and why the training and rendering times do not increase despite the additional processing steps?
- Is the receiver/rendering device the same as the encoding device? If the encoded frames/videos are sent to the client side (a different device), shouldn't the rendering be evaluated on a more reasonable device instead of the A100, which is server-level and too powerful for a typical client device?
- Can you discuss/study the accuracy-memory trade-off achieved by the quantization and sparsity techniques?
limitations: limitations: The authors acknowledge the limitation of exploiting inter-frame redundancy for very dynamic videos, which is reasonable. However, the paper evaluates video quality solely using metrics like PSNR and SSIM, which may not fully capture the viewing experience. Including a user experience study would provide a more comprehensive evaluation and strengthen the paper's conclusions. Also, including the accuracy-memory trade-off study with quantization and sparsity would be beneficial.
paper_summary: paper_summary: This paper proposes a framework called QUEEN, based on Gaussian splatting, for compact free-viewpoint videos. The data size is reduced to around 0.7MB per frame while achieving fast training speeds. QUEEN encodes the Gaussian attribute changes between consecutive frames. Specifically, it uses latent codes to embed the attribute residuals and employs a learnable gating scheme to update only the Gaussians that exhibit movement. The authors also introduce methods such as gate initialization and point cloud initialization for improved performance. The results are promising.
strength_weakness: strength_weakness: This experiment was conducted on Neural 3D Videos and Immersive Videos datasets, which feature forward-facing scenes with limited viewing angles. When we talk about free-viewpoint videos, we also expect large freedom for the novel viewpoints. However, this paper does not demonstrate such scenarios, such as 360-degree rendering for dynamic objects. The authors claim that QUEEN is intended for streaming FVVs, but the experiments do not evaluate or demonstrate the "streaming" feature. 
 
There is not information that introduces how does the entropy coding work on the quantized integer latents.
questions: questions: The reviewer suggests that the authors make their claims  more accurate, as mentioned in the weaknesses section.

Some notations are confusing. \mathbf{l}_i denotes the quantized integer latents, while \mathbf{l}_{p_i} is the learnable pre-gated residual. In line 223, \alpha is learnable parameter.  what is the relationship between  \alpha and g_i ?
limitations: limitations: In the limitations section, the authors mention that per-frame training faces challenges in reconstruction capability. It would be helpful if the authors could provide a more detailed explanation.
paper_summary: paper_summary: This paper presents a novel method for free-viewpoint video learning and streaming based on 3D Gaussian splatting (3DGS). The proposed method learns the attribute residuals of the raw Gaussian points in a frame-by-frame fashion, and the learning process is incorporated with both latent quantization and sparsity regularization. In addition, dynamic contents are identified and separated through Gaussian viewspace gradient difference in order to further accelerate computation. Compared to existing works, the proposed method is able to reduce the memory cost of FVV by a large margin, while preserving high-quality representation capability.
strength_weakness: strength_weakness: * The idea of exploiting temporal redundancy via encoding attribute residuals is not new. Although it has not been explored in the context of 3DGS, similar ideas have already been studied in NeRF-based representation, such as [75]. 

* The proposed method appears to be highly sensitive to hyperparameter tuning, as indicated in Section B, Tables 9 and 10. The authors find it necessary to adjust hyperparameters for different datasets. For instance, the learning rate for position residual learning in Immersive (0.0005) is approximately twice as high as that in N3DV (0.00016). Other hyperparameters, such as gating and quantization parameters, also exhibit significant differences across these datasets.
questions: questions: None
limitations: limitations: The authors have discussed the limitation and potential social impact Sec.C and Sec.D of the supplemental document, respecitively.
paper_summary: paper_summary: This paper introduces a novel technique, Quantized Efficient Encoding (QUEEN), to achieve streamable free-viewpoint videos. Unlike methods that directly optimize per-frame 3D-GS given multi-view videos, QUEEN learns the residuals of Gaussian attributes between continuous frames by decoding learnable latent embeddings for each 3D Gaussian. Additionally, a quantized decoder technique has been adopted to significantly reduce the size of latent embeddings (except for the embedding of 3D positions), with only a slight or no decrease in rendering performance. As the authors observe that position residuals are sensitive to compression, they introduce a differentiable L0 sparsity penalty (hard concrete gate with learnable parameters) instead of quantizing the position embeddings. Finally, to improve training efficiency and reduce storage requirements, the values of the gates for each frame are initialized using the 2D viewspace Gaussian gradient difference between neighboring frames. Extensive experiments have shown that QUEEN achieves the best training time, model size, and rendering performance compared to state-of-the-art online free-viewpoint video methods.
strength_weakness: strength_weakness: 1. Section 3.2 is difficult to follow due to the absence of preliminary information. There is a lack of detail regarding the quantization process. Specifically, the exact implementation of the quantization that results in more than a 10x size reduction is not provided. A detailed ablation study of the quantization type and the length of latent embeddings would be beneficial. Additionally, since the authors do not claim they will release the code for QUEEN, it is challenging for other researchers to evaluate and validate QUEEN's performance.
2. The ablation results presented in Table 2 are confusing and lack crucial analysis. For instance, it is unclear why the baseline achieves the highest rendering FPS without attribute quantization on the Immersive Dataset, while the same quantization improves the rendering FPS on the N3DV dataset, which appears contradictory. Furthermore, there is no clear explanation for why quantization improves rendering PSNR but results in slower training speeds. Additional analysis is needed to clarify these observations.
questions: questions: 1. Please clarify and add more details about the residual quantization in Section 3.2 to make it easier to follow. It would be beneficial to include an ablation study comparing different quantization methods. Additionally, highlighting any specific technical contributions of this work compared to other quantization approaches would improve the readability and comprehension of this section, or this is just a novel application of quantization technique?
2. Please provide a more detailed analysis of the ablation study results to better understand the key design choices in QUEEN’s framework. This additional analysis will help elucidate how these design choices contribute to the performance and effectiveness of the proposed method.
limitations: limitations: The authors have addressed the limitations of QUEEN as well as the societal impacts in their supplementary material.

Comments: comment: Thanks for authors' thorough analysis of the trade-offs between rendering speed, quality, and quantization. Since my concerns have been fully addressed, I would like to raise my rating to a 7.
Thank you for initiating discussions, reviewers and AC. Thank you for the insightful comment. Rendering speed for the Gaussian splatting depends on a wide variety of factors involving total number of Gaussians, their characteristics in terms of position, rotation, scale, and opacity as the rasterization for each tile varies depending on these attributes. Hardware configurations (I/O, OS scheduling, GPU clock fluctuation) can also cause a difference. For a similar number of Gaussians, it is possible for FPS to vary drastically with different attribute distributions which is indeed what we observe for N3DV (Table E-1(b)) based on the different scaling attribute distribution (Figure R2).

**Table E-1**: Further analysis on rendering speed. We provide more comparison between the baseline and QUEEN (+Attribute Quantization) on (a) one sequence “flame” from the google immersive datasets and (b) the N3DV datasets. All time measures are in the unit of milliseconds. Storage size in MegaBytes (MB).

(a) Sequence “flame” from the google immersive datasets
| Config                    | PSNR (dB) | Storage Size (MB)| Rendering time (ms) / FPS | Decoding Time (ms) / FPS | Num. Gauss. |
| ------------------------- | --------- | ----------- | --------------------- | -------------------- | ----------- |
| Baseline                  | 28.14     | 16434       | **4.13 / 242**    | N/A             | **217K**   |
| \+ Attribute Quantization | **30.76** | **1030**    | 4.47 / 223.9      | 0.43 / 2293     | 240K       |
| Relative Increase         | +9.3%     | \-93.7%     | +8.1%             | N/A             | +10.6%     |

(b) N3DV datasets

| Config                    | PSNR (dB) | StorageSize (MB)| Rendering time (ms) / FPS | Decoding Time (ms) / FPS | Num. Gauss. |
| ------------------------- | --------- | ---------- | --------------- | ------------ | ---------- |
| Baseline                  | 31.66     | 13308      | 4.67 / 214        | N/A          | **311K**   |
| \+ Attribute Quantization | **32.04** | **1255**   | **3.51 / 285**    | 0.43 / 2337    | 311K       |
| Relative Increase         | +1.2%     | \-90.8%    | \-24.8%         | N/A          | +0%        |

Table E-1 shows a more extensive analysis on rendering efficiency and other performance metrics of the baseline and with attribute quantization, for the sequence “flames” in Immersive as well as on the N3DV dataset. On the sequence “flames” of the immersive datasets, we obtain 10.6% more Gaussians (+23k) with the densification process compared to the baseline. This leads to higher rendering time (or lower FPS) (+8.1%) which is increased further due to the additional decoding time (0.43 ms).

Our attribute quantization of the Gaussians during training affects the distribution of the attributes, leading to different cloning and splitting during densification (based on viewspace gradients and Gaussian scale attribute) compared to the baseline. This also varies with different scenes depending on the scene changes (e.g., motion and appearance/disappearance of objects).

We highlight that the main goal of our paper is to reduce the storage and training time of online free-viewpoint videos (FVVs) while maintaining the reconstruction quality and the efficient real-time rendering speeds brought by Gaussian splatting. Although it is interesting to pursue higher rendering efficiency or controllability, those are not our goal. Our approach does not explicitly control the distribution of the Gaussian attributes or the number of Gaussians, which are the main factors behind rendering speeds. We believe this can be an interesting direction for future work to pursue and we will add discussion in the revision. 

Furthermore, our method addresses a joint problem of reconstruction and quantization for online FVV, in contrast to a two-step formulation of first reconstruction and then quantization. In the latter case, it is more likely for the system to pay for additional computational cost and some loss of quality. In our case (the former case), a joint formulation shows more potential to achieve a better tradeoff between quality and efficiency, as shown in Tab. D1 as well as recent methods such as [84]. We will add discussion in the revision to better explain our contributions.

We appreciate the discussion and we will include it in the revision for future audiences. Furthermore, we will release the codebase to the community to experiment with various types of scenes and discover new insights into the effect of quantization on rendering speeds.
Dear reviewers, the authors are eagerly awaiting your response. The author-reviewer discussion closes on Aug 13 at 11:59 pm AoE. Thanks!
Dear reviewers, as we approach the final two days, please take a moment to review the author's responses and join the discussion. Thank you!
Thank you for the discussion.
Any other thoughts from other reviewers?
Thank you for the detailed reply. Most of my concerns have been addressed. However, I remain unclear about the significant decrease in FPS on the Immersive dataset, as shown in Table 2. Could the authors please clarify why a more aggressive Gaussian densification has a different impact on the baseline compared to the baseline with attribute quantization? I would appreciate further elaboration on this point.
Dear reviewers, do the authors' responses answer your questions or address your concerns? Thanks.