Sort-free Gaussian Splatting via Weighted Sum Rendering

Authors: Qiqi Hou, Randall Rauwendaal, Zifeng Li, Hoang Le, Farzad Farhadzadeh, Fatih Porikli, Alexei Bourd, Amir Said
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Sort-free, Gaussian Splatting, Weighted Sum Rendering

Abstract:
Recently, 3D Gaussian Splatting (3DGS) has emerged as a significant advancement in 3D scene reconstruction, attracting considerable attention due to its ability to recover high-fidelity details while maintaining low complexity. Despite the promising results achieved by 3DGS, its rendering performance is constrained by its dependence on costly non-commutative alpha-blending operations. These operations mandate complex view dependent sorting operations that introduce computational overhead, especially on the resource-constrained platforms such as mobile phones. In this paper, we propose Weighted Sum Rendering, which approximates alpha blending with weighted sums, thereby removing the need for sorting. This simplifies implementation, delivers superior performance, and eliminates the ``popping'' artifacts caused by sorting. Experimental results show that optimizing a generalized Gaussian splatting formulation to the new differentiable rendering yields competitive image quality. The method was implemented and tested in a mobile device GPU, achieving on average $1.23\times$ faster rendering.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1729911985112

thread_type:
official_review

<thread_object>
creator: reviewer_uk93

<content>
summary:
This work attempts to solve a key problem with current Gaussian Splatting techniques, the computational load that occurs during rendering as a result of the non-commutativity of the opacity calculation needed during the rendering process. The authors achieve this by a novel weighting of term added to each Gaussian to replace the traditional opacity term. The weighting term allows for an order independent formulation of the computation of the final colour of the a rendered ray through the field. View-dependence is obtained by the introduction of a spherical harmonic based formulation to the new weighting term similar to that currently used in prior-art Gaussian Splatting methods for view-dependent colour. Performance is evaluated against prior-art techniques.

strengths:
This work solves a crucial problem in real-time rendering of Gaussian Splatting based radiance fields. By removing the need for depth-based sorting of the Gaussians before rendering, critical memory requirements are reduced enabling the implementation of the methodology on mobile devices or other applications with limited computational capacity. So I feel that the work represents a strong contribution to the field. Other strengths include:
1. Clear and easy to read figures and tables clearly detailing the benefits of the proposed method.
2. A detailed and clear explanation of the proposed method. It is clear what has been done, how this is motivated by previous literature.
3. A strong experimental validation. Comparisons are made against well-recognized metrics and a the latest state-of-the-art methods. The experimental results support well the conclusions reached.

weaknesses:
This work is of quite a high quality. I do not see any specific weaknesses in the methodology or the experimental results. There is one specific minor weakness in the experimental section wherein I would like further information on the memory requirements of the proposed method. I detail the exact question I have in the "Questions" section below.
One final minor issue is the presence of minor grammar errors in the text as I mention in the "Questions" section below.

questions:
1. In section 4.3 the authors reveal that view-dependent opacity is obtained by the addition of a new set of spherical harmonics related to the depth weighting introduced by the authors. In the experimental section, the authors show that this doesn't impact memory usage during rendering and a rationale given in the removal of the sorting step and the potential reduction in the number of Gaussians. However, this appears to be referring to instantaneous memory requirements. How does the introduction of the new set of spherical harmonics for each Gaussian affect post-training storage requirements of the radiance field? Is a greater storage footprint needed for the radiance fields produced by the proposed methods?
2. Line 53 "maintain rendering differentiable" should be "maintain rendering differentiability".
3. Line 141 "Chris Wyman provide a good survey on these methods" should be "Chris Wyman provides a good survey on these methods"
4. Line 273, "However, DIR-WSR does not work well for the complex scenes, " should be "However, DIR-WSR does not work well for complex scenes, "
5. Line 307 ends with an additional full stop.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730108211196

thread_type:
official_review

<thread_object>
creator: reviewer_b3oc

<content>
summary:
This paper proposes a sort-free variant to the 3DGS algorithm. Because 3DGS is a semi-transparent representation, it requires sorting of all primitives which leads to higher running time, more hardware requirements, and lower compatibility on edge devices. Learning from the traditional sort-free rendering algorithm which replaces the sorting with an estimated weight from depth, this paper proposes to eliminate the sorting process of 3DGS completely. 

This paper further proposes a view-dependent opacity to better simulate the occlusion effect. By learning spherical harmonics to represent the opacity, it allows the 3D Gaussians to appear and disappear at different angles, which could further mimic the occlusion effect.

As a result, this paper achieves a slightly higher rendering speed and lower GPU memory footprint, better compatibility on edge devices, and comparable rendering quality than 3DGS.

strengths:
**Motivation**
* The motivation of this paper is very clear. It clearly illustrates the drawbacks of the sorting requirement of 3DGS and the possible benefit of removing it. Sort-free transparent object rendering is also a well-studied direction in traditional rendering. Combining the algorithms from traditional computer graphics with 3DGS is well-motivated.

**Method**
* The proposed method transformed the traditional depth-based weighted rendering to a learnable version to better suit the per-scene optimization used in 3DGS, which provides significant performance improvement. 
* The proposed view-dependent opacity is an innovative approach that mimic the occlusion effect. Because this SH parameters are optimized together, it compensates for some performance loss from incorrect occlusion estimation without sorting.

**Experiments**
* The performance on normal desktop GPU demonstrates a small improvement in rendering speed and comparable rendering speed. In fact, it is already remarkable that a sort-free rendering algorithm can match the performance of a sorting-based rendering algorithm in a scene full of semi-transparent primitives.
* The performance improvement on the edge device shows a more significant improvement because of the demanding sorting step.

weaknesses:
** Motivation **
* As shown in the paper, the sorting step is usually considered slow, but actually contributes to only 25% of the total rendering time. The real problem preventing 3DGS based method from matching the rendering speed of traditional methods is the alpha blending step. Although this paper removes the sorting step, it still needs the per-pixel alpha blending of Gaussians of calculated weights. As a result, the speed improvement is rather insignificant. 
* Rather than removing the sorting step, many works have proposed to removing the redundant Gaussians directly. These methods seem to provide much more significant speed boost. Of course these two method can be run in parallel, but it still weaken the importance of the proposed method. 

** Method **
* The proposed method is largely borrowed from the traditional sort-free methods. I think this is acceptable because of the long history of sort-free rendering research in traditional computer graphics, it does negatively affect the novelty of the proposed method slightly.

** Experiments**
* As mentioned above, the improvement on standard desktop GPU is rather insignificant. Since sort-free rendering is essentially an approximation of the sort-based rendering, it might not be a good idea to choose this method for such a small improvement while there is a possibility of very bad performance on some scenes. 
* Continuing from the last point, sort-free approximations are usually inaccurate on small and thin structures. For example, I noticed that the proposed method on the bicycle scene in MipNeRF-360 dataset is somewhat lower than the original 3DGS method. This scene contains very thin and fine structures on the bicycle which could be a better illustration of whether the proposed method suffers from similar artifacts like the traditional methods. 
* Despite the significant rendering speed improvement on the edge devices, the paper lacks a detailed explanation of how the edge device running out of resources leads to such a big decrease in rendering speed.

questions:
1. Can the authors provide some qualitative results on the Bicycle scene?
2. Can the authors provide some videos showing the removal of the popping effect?
3. Can the authors show some failure cases of the proposed approximation?

I really want to give a positive rating to this paper, because I know it is very difficult to make a sort-free rendering to achieve the same quality as the sort-based version. However, the marginal speed improvement diminishes the significance of this paper. Furthermore, I hope the authors can demonstrate some failure cases of such approximation methods, as they are quite common in traditional sort-free algorithms. Doing this could give the reader a more comprehensive understanding of the proposed method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730430938730

thread_type:
official_review

<thread_object>
creator: reviewer_man9

<content>
summary:
The authors propose a novel rasterization backend for 3D Gaussian splatting (3DGS) that addresses the computational and memory overhead associated with depth sorting. Their primary contribution is replacing traditional alpha blending with a weighted sum rendering approach. The rendering is achieved through direct weighted summation of Gaussians, where weights are determined by a parameterized truncated linear function applied to the product of color and opacity. Additionally, they introduce view-dependent optimizable opacity for individual Gaussians, modeled using spherical harmonics.
Their weighted sum rendering (WSR) technique demonstrates comparable rendering quality to conventional 3DGS. To validate the advantages of their sort-free approach, the authors implement both vanilla 3DGS and WSR on the Vulkan API and evaluate performance on mobile GPU hardware, achieving a 1.23x speedup.

strengths:
- The paper aims to address a fundamental bottleneck in 3DGS rendering by proposing a sort-free pipeline, distinguishing itself from previous approaches that primarily focused on reducing Gaussian counts.
- The investigation of various weight derivation functions (direct sum-up, exponential weighted, and linear correction weighted) demonstrates a relative comprehensive experimental analysis effort.
- Implementation and evaluation on mobile devices effectively demonstrates the practical benefits of the sort-free approach in resource-constrained environments.
- The method appears to mitigate "pop" artifacts, a persistent challenge in volumetric rendering, providing an additional benefit beyond computational efficiency.

weaknesses:
- The theoretical foundation for replacing alpha blending with an algebraic function requires stronger justification. While the proposed representation achieves good results, it lacks the physical insights inherent in traditional alpha blending, which models light obstruction during propagation. The introduction of view-dependent opacity appears necessary for acceptable rendering quality, but the paper would benefit from a more rigorous analysis of why this combination works, rather than empirical validation alone.
- The preference for linear weights over exponential weights raises questions about robustness. When considering camera movement along viewing rays, exponential weights (with β ≈ 1) should theoretically provide more consistent mixing ratios between Gaussians. Linear weights may introduce color shift artifacts due to distance-dependent mixing ratio variations. While the authors claims "some artifacts remain visible" then selected linear weight rather than exponential weight, these issues warrant more detailed examination.
- The method's generalizability is limited by the transformation of opacity into a view-dependent "black box" variable. This fundamentally alters the role of opacity, which traditionally serves as a crucial signal for pruning and densification in many 3DGS applications, potentially limiting compatibility with existing and future research in this domain.

questions:
- Does the method exhibit color shift artifacts during camera movement along the z-direction, as suggested by the concerns regarding linear weighting?
- Since opacity is not capped at 1, how did you enforce the data range of final color, simply clamp?
- How do other Gaussian attributes maintain their geometric meaning in this framework? Specifically: Does the mean of Gaussians still correspond to point cloud locations? How about depth extraction?
- While traditional 3DGS implementations employ culling to manage computational complexity, does this approach require weighted summation of ALL Gaussians? What are the implications for scalability?
- Please clarify the memory usage analysis: does it encompass only Gaussian attributes or include runtime memory requirements?
What is the memory overhead of spherical harmonics coefficients for view-dependent opacity compared to vanilla 3DGS?
- Regarding Table 2, could you elaborate on the starred data points showing significantly higher values? How does resource exhaustion due to sorting impact subsequent pipeline stages?
- Given the potential for hardware-accelerated sorting methods specifically designed for 3DGS, possibly through hardware-software co-design, how might this work adapt to or complement such future developments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730576363093

thread_type:
official_review

<thread_object>
creator: reviewer_fze3

<content>
summary:
This paper proposed a sort-free Gaussian Splatting pipeline, which reduces the computational overhead of rendering to adapt to scenarios with limited computing resources. The key insight of this paper is that the view dependent sorting in the vanilla 3DGS involves a lot of computational overhead and is difficult to port to mobile devices. Therefore, the authors proposed to use weighted sum and view-dependent opacity to approximate alpha-blending. Experiments demonstrate that proposed method achieves comparable rendering quality and faster rendering speed than the original 3DGS on mobile devices.

strengths:
- The author provides a clear analysis of the tile-based sort in the vanilla 3DGS, explains its impact on rendering speed, and explains why it is difficult to implement on other devices.
- The proposed method speeds up rendering while ensuring rendering quality, and has been verified on consumer mobile devices.
- The proposed weighted sum approximation alleviates the *pop* phenomenon caused by inconsistent sorting results under different view-directions in the vanilla 3DGS.

weaknesses:
- The proposed method gets rid of the costly sorting, but a more complex weight and view-dependent opacity are used. Although the author points out in the paper that a general scheme for learning scene representations is bound only by the mathematical constraints of the scene model. According to the experimental results in the paper, the use of simple weighted-sum and view-independent opacity cannot achieve satisfactory rendering results. Does this mean that for rendering processes that are not physically and optically, more complex models must be used, such as introducing more learnable parameters as in the paper?
- In the conclusion part of the paper, the author mentioned some recent research on compact Gaussian Splatting. According to my understanding of these works, since the methods proposed in these works obtain more compact 3D Gaussian representations, they can reduce the sorting overhead and speed up rendering to a certain extent. Therefore, I think the author should add a comparison of rendering quality, speed and memory usage between the proposed method and these works, such as Compact 3DGS[1], LightGaussian[2] and Compressed 3DGS[3] in the experimental evaluation section.

[1] Compact 3D Gaussian Representation for Radiance Field https://openaccess.thecvf.com/content/CVPR2024/papers/Lee_Compact_3D_Gaussian_Representation_for_Radiance_Field_CVPR_2024_paper.pdf

[2] LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS https://arxiv.org/pdf/2311.17245

[3]  Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis https://openaccess.thecvf.com/content/CVPR2024/papers/Niedermayr_Compressed_3D_Gaussian_Splatting_for_Accelerated_Novel_View_Synthesis_CVPR_2024_paper.pdf

questions:
In the paper, the author points out that for some complex scenes, since a large number of 3D Gaussians are needed for representation, the sorting process will exhaust the resources of mobile devices, making real-time rendering impossible, such as the bicycle scene in Mip-NeRF 360 dataset. However, I tried to visualize the bicycle scene using a gsplat-based webgl viewer on my phone (iPhone 15 Pro Max), and was surprised to find that it achieves real-time rendering, about 20-30fps. This seems to indicate that some optimizations to the 3DGS rendering pipeline (gsplat uses some tricks) can increase the rendering speed on mobile devices without more complex designs. I would like to know the author's opinion on my point of view and more analyses of the key differences that enable real-time performance on mobile devices. And they can test the performance of gsplat on Snapdragon® 8gen3 chipset and compare with their proposed method.

Here is the link of gsplat webgl-viewer https://gsplat.tech
</content>
</thread_object>

</thread>
<thread>
creation date:
1732410402040

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of reviews

comment:
We thank the reviewers for their constructive and thoughtful comments, which have significantly enhanced the quality of our submission. Revised part is showed in `teal` in the paper pdf. Please note that we moved the supplementary materials to the end of the main paper. Below, we provide a general summary of the common strengths and weaknesses identified by the reviewers, followed by specific responses to each reviewer's remarks.


### Strengths:
- The paper is well-motivated, addressing a critical bottleneck in 3D Gaussian Splatting (3DGS) rendering by introducing a sort-free pipeline.
- The proposed learnable weighted sum rendering technique, as well as view-dependent opacity, offer substantial performance improvements.
- The performance gains are particularly notable on mobile devices.
- The method effectively mitigates "pop" artifacts, a persistent challenge in volumetric rendering.

### Weaknesses:
- The paper requires a comparison with methods that reduce the number of Gaussians and with 3DGS implementations in WebGL.
- The analysis of memory usage and GPU resource exhaustion needs clarification.


As a general comment, in this paper we consider the fact that for Gaussian Splatting to be successfully used on mobile devices, it should not only be rendered in real time, but also with low power usage. While many works aim to reduce complexity, our focus is specifically on addressing the sorting problem.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel sort-free rendering pipeline for 3D Gaussian Splatting (3DGS). Inspired by order-independent transparency, the method approximates alpha blending using weighted sums, thereby eliminating the need for sorting and reducing rendering time, particularly on mobile devices. Additionally, the proposed approach mitigates pop artifacts typically caused by sorting. 

The motivation is clear, the idea is novel, and the experiments are compelling. The primary weakness of the initial submission, its lack of comparisons with compact 3DGS representations, was effectively addressed in the rebuttal. 

Overall, the paper makes a valuable contribution by presenting a novel sort-free rendering pipeline for 3DGS. The implementation and evaluation on mobile devices convincingly demonstrate the practical advantages of this approach in resource-constrained environments.