On Scaling Up 3D Gaussian Splatting Training

## Abstract

Abstract 3D Gaussian Splatting (3DGS) is increasingly popular for 3D reconstruction due to its superior visual quality and rendering speed. However, 3DGS training currently occurs on a single GPU, limiting its ability to handle high-resolution and large-scale 3D reconstruction tasks due to memory constraints. We introduce Grendel, a distributed system designed to partition 3DGS parameters and parallelize computation across multiple GPUs. As each Gaussian affects a small, dynamic subset of rendered pixels, Grendel employs sparse all-to-all communication to transfer the necessary Gaussians to pixel partitions and performs dynamic load balancing. Unlike existing 3DGS systems that train using one camera view image at a time, Grendel supports batched training with multiple views. We explore various optimization hyperparameter scaling strategies and find that a simple sqrt(batch_size) scaling rule is highly effective. Evaluations using large-scale, high-resolution scenes show that Grendel enhances rendering quality by scaling up 3DGS parameters across multiple GPUs. On the “Rubble” dataset, we achieve a test PSNR of 27.28 by distributing 40.4 million Gaussians across 16 GPUs, compared to a PSNR of 26.28 using 11.2 million Gaussians on a single GPU. Grendel is an open-source project available at: https://github.com/nyu-systems/Grendel-GS

## 1 Introduction

3D Gaussian Splatting (3DGS)[] has emerged as a popular technique for 3D novel view synthesis, primarily due to its faster training and rendering compared to previous approaches such as NeRF [].
However, most existing 3DGS pipelines are constrained to using a single GPU for training, creating memory and computation bottlenecks when applied to high-resolution or larger-scale scenes. For example, the standard Rubble dataset [] contains 1657 images, each with a 4K resolution. A single A100 40GB GPU can hold up to 11.2 million Gaussians – well below the quality saturation point for 3DGS. As we demonstrate in Section 5.2.1, increasing the number of Gaussians continues to improve reconstruction quality. Therefore, in order to scale up 3DGS training in terms of parameter count and speed, we develop the Grendel distributed training system, consisting of distributing 3DGS training across multiple GPUs and an empirical rule to automatically adapt training hyperparameters based on the batch size.

Current approaches that work around the single-device 3DGS training pipeline limitations when handling large-scale scenes typically involve compression schemes or selective rendering [, , ]. However, these methods are still confined to a single GPU and are fundamentally limited by similar memory and computational bottlenecks when scaling up further.

As large-scale distributed machine learning has become industry standard and a cornerstone for many state-of-the-art AI models such as Large Language Models [, ], it is reasonable to expect that distributing workloads across multiple GPUs leads to similar performance and scale improvements, where it allows 3DGS to handle larger, higher-resolution scenes and produce higher-quality outputs.

However, 3DGS stands out as a gradient-based but *not* a neural network-based training framework. It features a unique computation pipeline with dynamic and imbalanced workload patterns within a training iteration. Consequently, existing SysML frameworks and toolkits, such as those in [, , , ], do not apply to this domain. The frameworks that target neural networks assume a consistent and balanced workload consisting of a series of regular tensor operations (*e.g.* GEMM) that are distinct from the dynamic, non-uniform computation patterns exhibited by 3DGS training and rendering. Therefore, designing a scalable distributed system for 3DGS is uniquely challenging.

In this paper, we present several key observations on scaling up Gaussian Splatting training that inform the design of our distributed learning pipeline for 3DGS. For instance, we note that each stage of the 3DGS training pipeline in an iteration can be effectively parallelized, but the axes of parallelization differ across stages. In the *mixed parallelism* of the 3DGS training pipeline, some computations operate on individual output pixels (allowing for pixel-wise parallelism), while others operate on individual 3D Gaussians (allowing for Gaussian-wise parallelism). This enables parallelism but necessitates communication between pipeline stages. To minimize communication, we also observe that 3DGS exhibits *spatial locality*, where only a small number of Gaussians affect each image patch. Finally, the computational intensity of rendering an output pixel changes as training progresses, so we identify the *dynamic and unbalanced workloads*. As a result, any static workload partitioning strategy will be suboptimal.

In this paper, we describe Grendel, a distributed 3DGS training framework designed under these observations. Grendel uses Gaussian-wise distribution–that is, it distributes Gaussians across GPUs–for steps in a training iteration that exhibit Gaussian-wise parallelism, and pixel-wise distribution for other steps. It minimizes the communication overhead when switching between Gaussian-wise and pixel-wise distribution by assigning contiguous image areas to GPUs during pixel-wise distribution and exploiting spatial locality to minimize the number of Gaussians transferred among GPUs. Finally, Grendel employs a dynamic load balancer that uses previous training iterations to distribute pixel-wise computations to minimize workload imbalance.

Grendel additionally scales up training by batching up multiple images. This differs from conventional 3DGS training that exclusively uses a batch size of 1, which would lead to reduced GPU utilization in our distributed framework. To maintain data efficiency and reconstruction quality with larger batches, one needs to re-tune optimizer hyperparameters. To this end, we introduce an automatic hyperparameter scaling rule for batched 3DGS training based on a heuristical independent gradients hypothesis. We empirically validate the effectiveness of our proposed approach — Grendel supports distributed training with large batch sizes (we test up to 32) while maintaining reconstruction quality and data efficiency compared to batch size $=1$.

Figure: Figure 2: (a) Traditional 3DGS training pipeline using a single GPU vs. (b) Our Grendel system that distributes 3D Gaussians across multiple GPUs to alleviate the GPU memory bottleneck. We also partition the computation in the pixel and batch dimensions to for further speedup. Every square represents a $16\times 16$ block of pixels.
Refer to caption: x2.png

In summary, our work makes the following contributions:

- •
We describe the design and implementation of Grendel, a scalable, memory-efficient, adaptive
distributed training system for 3DGS. Grendel allows batched 3DGS training to be scaled up and run on up to 32 GPUs.
- •
We explore the large-batch training dynamics of 3DGS to identify a simple sqrt(batch_size) learning rate scaling strategy, which enables efficient, hyperparameter-tuning-free training for batch sizes beyond one.
- •
We show that Grendel enables high-resolution large scale scene rendering: we use 16 GPUs
and render 4K images for large-scale Rubble scene from MegaNERF []. For this scene, Grendel uses 40.4 million Gaussians to achieve a PSNR of 27.28, outperforming the current state-of-the-art.
The memory required exceeds a single GPU’s capacity, making it difficult to render this scene at this quality without Grendel’s techniques.

## 2 Gaussian Splatting: Background, Opportunities and Challenges

3D Gaussian Splatting [] (3DGS) is a rendering method that represents 3D scenes using a (potentially large) set of anistropic 3D Gaussians. Each 3D Gaussian is represented by four learnable parameters: (a) its 3D position $x_{i}\in\mathbb{R}^{3}$; (b) its shape described by a 3D covariance matrix computed using the Guassian’s scaling vector $s_{i}\in\mathbb{R}^{3}$ and rotation vector $q_{i}\in\mathbb{R}^{4}$; (c) its opacity $\alpha_{i}\in\mathbb{R}$; and (d) its spherical harmonics $sh_{i}\in\mathbb{R}^{48}$. The color contribution of each Gaussian is detemined by these parameters and by the viewing-direction.

### 2.1 Background on 3D Gaussian Training

To train 3DGS, the user provides an initial point cloud (may be random or estimated) for a scene and a set of posed images from different angles. The training process initializes Gaussians using the point cloud. Each training step selects a random camera view and uses the current Gaussian parameters to render the view. It then computes loss by comparing the rendered image to the ground truth, and uses back-propagation to update the Gaussian parameters. The training process also uses an adaptive densification mechanism to add Gaussians to under-reconstructed areas. Gaussians are added by cloning or splitting existing ones based on their position variance and scale threshold, with more details in A.1.

Concretely, the training pipeline consists of four steps: Gaussian transformation, image rendering, loss calculation, and backpropagation. Standard approaches to backpropagation are used in this setting, and we detail the remaining three steps below:

- 1.
Gaussian transformation: Given a camera view $v$ and the associated screen space, each Gaussian $i$ is transformed and projected to determine its position $x_{v,i}\in\mathbb{R}^{2}$ on screen, its distance $depth_{v,i}\in\mathbb{R}$ from the screen, and its coverage (or footprint radius) $radius_{v,i}\in\mathbb{R}$. Additionally, the color of each Gaussian $c_{v,i}$ is determined according to the viewing direction using its learnable spherical harmonics coefficients $sh_{i}\in\mathbb{R}^{48}$.
- 2.
Rendering: After Gaussian transformation, the image is rendered by computing each pixel’s color. To do so, for a given pixel $p$, 3DGS first finds all Gaussians that intersect with $p$. We say that a Gaussian $i$ intersects with $p$ if $p$ lies within $radius_{v,i}$ of the Gaussian $i$’s projected center $x_{v,i}$. Then 3DGS iterates over intersecting Gaussians in increasing depth (i.e. in increasing $depth_{v,i}$) and uses alpha-composition to combine their contributions until a threshold opacity has been reached.
- 3.
Loss calculation: Finally, the 3DGS computes the L1 and SSIM loss by comparing the rendered image to the ground truth image. The L1 loss measures the absolute difference between pixel colors, while the SSIM loss measures the similarity between pixel windows. Both metrics are computed per-pixel for both forward and backward implementations.

### 2.2 Opportunities and Challenges in distributing 3DGS

In designing Grendel for scaling up 3D Gaussian Splatting training, we exploit the following opportunities in the above-described training process and address several challenges:

##### Opportunity: mixed parallelism.

Each of the steps described above is inherently parallel but requires different kinds of work partitioning. In particular, the Gaussian transformation step operates on individual Gaussians and thus should be partitioned by Gaussians. On the other hand, the rendering and loss calculation steps operate on individual pixels(or pixels windows for SSIM loss) and thus should be partitioned by pixel.

##### Opportunity: spatial locality.

Most Gaussians intersect a small contiguous area of the rendered image due to their typically small radius. As illustrated in Figure 3, 90% of the 3D Gaussians in three scenes (Rubble, Bicycle, and Train) have a radius $<2\%\leavevmode\nobreak\ \text{of image width}$. Consequently, a pixel is affected by a small subset of the scene’s 3D Gaussians, with significant overlap among neighboring pixels’ Gaussians.

Figure: Figure 3: The footprint radius of most Guassians is small relative to the image size.
Refer to caption: x3.png

##### Challenge: dynamic and unbalanced workloads.

Different image areas intersect varying quantities of Gaussians. For instance, an image region containing the sky likely corresponds to fewer Gaussians than a region with a person. Additionally, the density, position, shape, and opacity of Gaussians change throughout training. Therefore, the number of Gaussians and their mapping to pixels evolve over time, leading to computational workload imbalances across different image regions and over the training period. Fixed partitioning schemes thus suffer from load imbalance.

##### Challenge: absence of batching.

Current 3DGS systems process images one at a time, which suffices for single GPU training. However, as shown in §5, this approach is inefficient in a distributed setting with multiple GPUs. Effective training with larger batch sizes necessitates an understanding of the unique optimization dynamics of 3DGS, which may differ from those of conventional neural networks.

## 3 System Design

Here, we describe how Grendel exploits the mixed parallelism and spatial locality of 3DGS (§3.1) to address the challenge of dynamic and unbalanced workloads (§3.2).

### 3.1 Mixed parallelism training

Figure 2(b) provides an overview of Grendel’s design.
Grendel distributes work according to 3DGS’ *mixed parallelism*: it uses Gaussian-wise distribution—where each GPU operates on a disjoint subset of Gaussians–for the Gaussian transformation step, and pixel-wise distribution–where each GPU operates on a disjoint subset of pixels— for the image rendering and loss computation step. The *spatial locality* characteristic allows Grendel to benefit from sparse all-to-all communication when transitioning between these stages.

Gaussian-wise Distribution. Grendel partitions the Gaussians, including their parameters and optimizer states, and distributes them uniformly across GPUs. Then, each GPU independently computes the Gaussian transformation for the set of 3D Gaussians assigned to it. We found that the amount of computation required does not significantly vary across Guassians, and thus evenly distributing Gaussians across GPUs allows us to fit the maximal number of Gaussians while speeding up computation linearly for this step.

Pixel-wise Distribution. We distribute contiguous image areas across GPUs for the image rendering and loss computation steps. Distributing contiguous areas allows us to exploit spatial locality and reduce the number of Gaussians transferred among GPUs in this step. In our implementation, we partition each image in a batch by dividing it into $16\times 16$-pixel blocks, serializing the blocks, and then distributing consecutive subsequences of blocks to different GPUs using an adaptive strategy (§3.2).
For batching, each GPU can be assigned blocks from different images in a batch, as shown in Figure 2(b).

Figure: Figure 4: Each GPU only considers Gaussians whose footprints intersect with its assigned pixel render area. Every square represents a 16 × 16 block of pixels.
Refer to caption: x4.png

Transferring Gaussians with sparse all-to-all communication. To render an image pixel, a GPU needs access to Gaussians that intersect the pixel, which cannot be pre-determined as they are view-dependent and change during training. Therefore, Grendel includes a communication step after the Gaussian transformation. As 3DGS exhibits spatial locality, each pixel partition only requires a small subset of all 3D Gaussians. We leverage this to reduce communication: each GPU first decides the set of intersecting Gaussians for rendering a pixel partition (Figure 4) before using a sparse all-to-all communication to retrieve Gaussians intersecting with any pixels in the partition. A reversed all-to-all communication is done during the backward pass.

Although Grendel’s design bears some resemblance to FSDP [] used for distributed deep neural network training, there are some important differences. Firstly, unlike weight sharding in FSDP, Gaussian-wise distribution in Grendel is not merely for storage but for also for computation (the Gaussian transformation). Secondly, unlike FSDP which transfers weight shards using the dense all-gather communication, Grendel transfers only relevant Guassians using sparse all-to-all communication.

### 3.2 Iterative Workload Rebalancing

##### Pixel-wise Distribution Rebalancing.

As discussed in §2.2, the computational load of rendering a pixel varies across space (different pixels) and time (different training iterations). Thus, unlike in distributed neural network training, a uniform or fixed distribution cannot guarantee balanced workloads, so an adaptive pixel distribution strategy is needed.

We record the rendering time of each pixel of each training image during every epoch after the first few. Since the scene generally changes smoothly between consecutive epochs during training, the rendering time of each pixel also changes slowly. Therefore, the rendering times from previous epochs form a good estimate of a pixel’s rendering time in the current epoch. Based on this estimate, we can adaptively assign pixels to different GPUs such that the workloads are approximately balanced.

Specifically, Grendel measures the running time (including image rendering, loss computation, and the corresponding backward computation) of each block of pixels assigned to a GPU, computes the average per-pixel computation time for the GPU, and uses this average to approximate the computation time for any pixel $p$ assigned to the GPU. For example, if a GPU is assigned pixels $p_{0}$ through $p_{n}$, and takes time $t$ for all of these pixels, then Grendel assumes that pixel $p_{i}$ where $i\in[0,n]$ requires $\frac{t}{n}$ time for computation. In subsequent iterations, the image is re-split so that the sum of the computation time for pixels assigned to all GPUs are equal. In our implementation, we use $16\times 16$ pixel blocks as the split granularity. We show the pseudocode (Algorithm 1) for calculating the Division Points to split an image into load-balanced subsequences of blocks.

Figure: Algorithm 1 Calculation of Division Points

##### Gaussian-wise Distribution Rebalancing.

When training starts, we distribute 3D Gaussians uniformly among the GPUs. As training progresses, new Gaussians are added by cloning and splitting existing ones(§2.1). Newly added Gaussians make the distribution imbalanced as different Gaussians densify at different rates that depend on the scene’s local details.
Therefore, we redistribute the 3D Gaussians after every few densification steps to restore uniformity.

## 4 Scaling Hyperparameters for Batched Training

To efficiently scale to multiple GPUs, Grendel increases the batch size beyond one so it can partition training into a batch of images in addition to pixels inside each image, as shown in Figure 2(b).

However, increasing the batch size without adjusting hyperparameters, particularly the learning rate, can result in unstable and inefficient training [, ], and hyperparameter tuning is a time-consuming and tedious process. Though there have been works that simplify learning-rate tuning for training deep neural networks, their settings are distinct from ours since they either build on SGD [] (where we use Adam) or they leverage the layer-wise structure of neural networks [, ]. We propose an automatic hyperparameter scaling rule for the Adam optimizer to enable hyperparameter-tuning-free batched 3DGS training. Our result is driven by the *Independent Gradients Hypothesis* unique to 3DGS. Interestingly, our proposed square-root scaling rule coincides with recent works that study Adam learning rate scaling for neural network training even though they make distinct assumptions [, ].

We propose to scale Adam’s learning rate and momentum based on batch size as follows:

$$ $\displaystyle\lambda^{\prime}=\lambda\times\sqrt{\text{batch_size}}$ (1) $\displaystyle\beta_{1}^{\prime},\beta_{2}^{\prime}=\beta_{1}^{\text{batch_ size}},\beta_{2}^{\text{batch_size}}$ (2) $$

where $\lambda$ is the original learning rate, and $\beta_{1},\beta_{2}$ are the original first and second moments in Adam. $\lambda^{\prime},\beta_{1}^{\prime},\beta_{2}^{\prime}$ are the adjusted hyperparameters to work with a greater batch size. We refer to these as the square-root learning rate scaling and the exponential momentum scaling rules.

Independent Gradients Hypothesis. To arrive at these scaling rules, it is helpful to consider 3D GS training in a simplified setting. Suppose that the gradients received by 3D Gaussian parameters from each camera view are independent of gradients induced by other views.
Consequently, if we are given a batch of $b$ camera views, taking $b$ sequential gradient descent steps for each view in the batch is equivalent to taking one bigger step where the gradients are summed together. If we were using the vanilla gradient descent algorithm and averaging the gradients in a batch, setting the learning rate to scale linearly with the batch size achieves this equivalence. However, 3D GS uses Adam, an adaptive learning rate optimizer that (1) divides the gradients by the square root of the per-parameter second moment estimate, and (2) uses momentum to combine current gradients and past gradients in an exponential-moving-average fashion, making a bigger update different from simply summing up smaller batch-size-one updates. Under the independent gradients hypothesis, we derive the following corrections to Adam hyperparameters to approximate batch-size-one training with a larger batch:

Let us denote $g_{k}$ as the gradient of some parameter evaluated at view $k$, and $g=\frac{\sum_{j\in V}g_{j}}{|V|}$ as the full-batch gradient (mean of gradients across views), where $V$ is the set of all views. Let us further assume $\mathbb{E}[g_{k}]=0$ for all $k$. By the independence assumption: $\operatorname{Cov}(g_{k},g_{j})=\mathbb{E}[(g_{k}-0)(g_{j}-0)]=0\quad\text{
when}\quad k\neq j\quad\text{and}\quad\mathbb{E}[(g_{k})^{2}]\quad\text{when}
\quad k=j.$

Then, parameter update from a batch-size-1 Adam step (without momentum) on view $k$ is:

$$ $\Delta^{\{k\}}=\frac{g_{k}}{\sqrt{\mathbb{E}\left[\mathbb{E}_{j\in V}\left[g_{ j}^{2}\right]\right]}}=\frac{g_{k}}{\sqrt{\mathbb{E}\left[|V|g^{2}\right]}}= \frac{g_{k}}{\sqrt{|V|}\sqrt{\mathbb{E}\left[g^{2}\right]}}.$ $$

However, the parameter update from one Adam step (without momentum) on a batch of views $B\subseteq V$ of size $b$ is:

$$ $\Delta^{\{B\}}=\frac{\sum_{k\in B}g_{k}/b}{\sqrt{\mathbb{E}\left[\mathbb{E}_{B ^{\prime}\subseteq V}\left[\left(\sum_{j\in B^{\prime}}g_{j}/b\right)^{2} \right]\right]}}=\frac{\sum_{k\in B}g_{k}/b}{\sqrt{\mathbb{E}\left[\frac{|V|}{ b}g^{2}\right]}}=\frac{\sum_{k\in B}g_{k}/b}{\sqrt{\frac{|V|}{b}}\sqrt{\mathbb {E}\left[g^{2}\right]}}=\frac{1}{\sqrt{b}}\frac{\sum_{k\in B}g_{k}}{\sqrt{|V|} \sqrt{\mathbb{E}\left[g^{2}\right]}}.$ $$

Thus, setting the learning rate $\lambda^{\prime}=\lambda\times\sqrt{b}$ allows the batch update $\Delta^{\{B\}}$ to match with the total individual updates $\sum_{k\in B}\Delta^{\{k\}}$.

Alongside the square-root learning rate scaling (Eq 1), we also propose an exponential momentum scaling to accommodate larger batches (Eq 2). Initially used by [], this rule scales the momentum parameters with $\beta^{\prime}=\beta^{\text{batch_size}}$, which exponentially decreases the influence of past gradients when the batch size increases. We wish to stress that in the real world, even though some cameras share similar poses, a set of random cameras generally observe different parts of a scene, hence the gradients in a batch are mostly sparse and can be thought of as roughly independent. We empirically study the independent gradient hypothesis and evaluate our proposed scaling rules.

### 4.1 Empirical Evidence of Independent Gradients

To see if the *Independent Gradients Hypothesis* holds in practice, we analyze the average per-parameter variance of the gradients in real-world settings. We plot the sparsity and variance of the gradients of the diffuse color parameters starting at pre-trained checkpoints on the “Rubble” dataset [] against the batch size in Figure 5. We find that the inverse of the variance increases roughly linearly, then transitions into a plateau. We find this behavior in all three checkpoint iterations, representing early, middle, and late training stages. The initial linear increase of the precision suggests that gradients are roughly uncorrelated at batch sizes used in this work (up to 32) and supports the independent gradients hypothesis. However, it is worth noting that even though a single image has sparse gradients, when there are many images in a batch, the gradients overlap and become less sparse. They also become more correlated because we expect images with similar poses to offer similar gradients.

Figure: Figure 5: Gradients are roughly uncorrelated in practice. On the “Rubble” dataset [], the inverse of the average parameter variance increases linearly, then rises to a plateau, suggesting that the gradients are roughly uncorrelated initially but become less so as the batch size becomes large. Averaged over 32 random trials.
Refer to caption: x5.png

Figure: (a) Learning rate scaling rules vs. BS invariance.
Refer to caption: x6.png

### 4.2 Empirical Testing of Proposed Scaling Rules

To empirically test whether the proposed learning rate and momentum scaling rules work well, we train the “Rubble” scene to iteration 15,000 with a batch size of 1. Then, we reset the Adam optimizer states and continue training with different batch sizes. We compare how well different learning-rate and momentum scaling rules maintain a similar training trajectory when switching to larger batch sizes in Figure 6. Since different parameter groups of 3D GS have vastly different magnitudes, we focus on one specific group, namely the diffuse color, to make the comparisons meaningful. Figure 6(a) compares three different learning rate scaling rules $\in$ [constant, sqrt, linear] where only our proposed “sqrt” holds a high update cosine similarity and a similar update magnitude across different training batch sizes. Similarly, 6(b) shows our proposed exponential momentum scaling rule keeps update cosine similarity higher than the alternative which leaves the momentum coefficients unchanged.

## 5 Evaluation

Our evaluation aims to demonstrate Grendel’s scalability, showing both that it can render high-resolution images from large scenes, and that its performance scales with additional hardware resources. And we include results from abalation study to demonstrate the importance of Grendel’s dynamic load balancing. The ablation study for the learning rate scaling strategies have already been discussed in 4.2, along with our analysis.

### 5.1 Setting and Datasets

##### Experimental Setup.

We conducted our evaluation in the Perlmutter GPU cluster []. Each node we used was equipped with 4 A100 GPUs with 40GB of GPU memory, and interconnected with each other using 25GB/s NVLink per direction. Servers were connected to each other using a 200Gbps Slingshot network.

##### Datasets.

We evaluate Grendel using the datasets and corresponding resolution settings shown in Table 1. Of these, Rubble and MatrixCity Block_All represent the large scale datasets that are out of reach for most existing 3DGS systems, while other datasets are commonly used in 3DGS papers. These datasets vary in area size and resolution to comprehensively test our system.

**Table 1: Scenes used in our evaluation: We cover scenes of varying sizes and resolutions.**
| Dataset | # Scenes | Resolutions | # Images | Test Set Setting |
| --- | --- | --- | --- | --- |
| Tanks & Temple [] | 2 | $\sim$ 1K | 251 to 301 | 1/8 of all images |
| DeepBlending [] | 2 | $\sim$ 1K | 225 to 263 | 1/8 of all images |
| Mip-NeRF 360 [] | 9 | 1080P | 100 to 330 | 1/8 of all images |
| Rubble [] | 1 | $4591\times 3436$ | 1657 | official test set |
| MatrixCity Block_All [] | 1 | 1080P | 5620 | official test set |

##### Evaluation Metrics.

We report image quality using SSIM, PSNR and LPIPS values, and throughput in training images per second. We take both forward and backward time into consideration of throughput. And note that throughput in images per second may differ from throughput in iterations per second, as one iteration includes the batch size number of images.

### 5.2 Performance and Memory Scaling

We start by evaluating Grendel’s scaling, and how additional GPUs impact computation performance and memory.

Figure: Figure 7: To avoid OOM, 4 GPUs are needed to train the large 4K “Rubble” scene. We further improve throughput by distributing across even more GPUs and increasing the batch size.
Refer to caption: x8.png

Figure: Figure 9: Training Speedup and PSNR on Mip-NeRF360 and Tanks&Temples+Deep Blending.
Refer to caption: x10.png

##### Computation.

We evaluated how additional GPUs impact Grendel’s performance using both large-scale (rubble and train) and small-scale (Mip-Nerf360) datasets.

We used the Rubble scene to evaluate the training throughput. For this experiment we used 35 million Gaussians which have been trained to convergence. Because of the time required to render 4K images for this scene, we measured throughput for training over another 10,000 images times, and in Figure 8 we report throughput (in images per second) as we vary the number of GPUs (x-axis) and batch size (y-axis). We observe that we cannot render this scene with a single GPU (regardless of batch size) because of its memory requirements. Furthermore, both increasing the number of GPUs and increasing batch size yield performance improvements: performance increases from 5.55 images per second (4 GPUs, batch size 1) to 38.03 images per-second (32 GPUs, batch size 64) in our experiments. We provide additional details about this experiment in Appendix C.2.

Next, we use the $980\times 545$ resolution Train scene to evaluate image quality during scaling (this image is small enough to allow us to render the image fully). Our results in Figure 8 show that additional GPUs improve throughput while maintaining image quality when trained with the same total number of images. Notably, our 16-GPU setup with a batch size of 32 completes training on 30K images in just 2 minutes and 42.97 seconds, representing the state-of-the-art training speed to the best of our knowledge.

We observed in Figure 10 the same improvement in performance without degradation for other scenes taken from the Mip-Nerf360 dataset (first half) and the Tanks & Temple and Deep Blending datasets (second half), and using default hyperparameters from the 3DGS repository []. We train both configurations on the same number of images: 50k for Mip-NeRF 360 and 30k for the slightly smaller TT & DB datasets, to ensure convergence and a fair comparison. Additional details on this experiment are in Appendix C.2.

##### Memory Scaling.

Scaling the number of GPUs also increases available memory, and thus the amount of Gaussians that can be used to represent a scene. We evaluated this effect by using densification to add additional Gaussians until we ran out of memory. In Figure 12 we show number of Gaussians (in millions) when using batch size 1, 4 and 16, while varying the number of GPUs, and observe linear scaling.
In §5.2.1 we show the utility of using additional Gaussians, and we provide additional details about this evaluation in Appendix C.2.

Figure: Figure 11: More GPUs provide additional memory to support more Gaussians before encountering OOM.
Refer to caption: x12.png

#### 5.2.1 Gaussian Quantity vs. Reconstruction Quality

Figure: Figure 13: Scalability Statistics: Gaussian Quantity vs. Reconstruction Quality Using more Gaussians results in better test metrics for reconstruction. The red line indicates the number of Gaussians a single GPU can handle, which is insufficient for achieving high-quality results.
Refer to caption: x14.png

Figure: Figure 14: Visualization: Gaussian Quantity vs. Reconstruction Quality
Refer to caption: extracted/5684424/visualization/output.png

Scaling to multiple GPUs allows Grendel to use a larger number of Gaussians to represent scenes. A larger number of Gaussians can capture fine grained scene details, and should thus be able to better reconstruct large-scale, high-resolution scenes. We evaluated this effect using three scenes: Rubble, MatrixCity Block_All, and Bicycle, and varied the number of Gaussians used by changing densification settings: we lowered the gradient norm threshold to initiate densification and reduced the threshold for splitting Gaussians instead of cloning until the densification mechanism produced the target number of Gaussians without manual interference.

We rendered Rubble and Matrix City Block_All using 16 GPUs and a batch size of 16, while we used 4 GPUs and a batch size of 4 for Bicycle. The difference in number of GPUs and batch sizes is due to differences in scene sizes: bicycle is much smaller than the other two datasets. In Figure 13 we show that image quality metrics (PSNR, SSIM and LPIPS) improve as we add more Gaussians. The red line in the Rubble and Matrix City Block_All graphs shows the number of Gaussians that can fit on a single GPU (Bicycle, being smaller, can be rendered on a single GPU). Figure 14 shows the rendered images as we scale Gaussians quantity, and demonstrates the quality improvements are human visible. These results demonstrate benefits of using more Gaussians, and demonstrate the necessity of multi-GPU 3DGS training systems like Grendel.

### 5.3 Ablation Study

Figure 10 and 12 illustrate that our load balancing techniques and increased batch size significantly improve training throughput on both 1080p Mip-NeRF360 dataset and 4K Rubble datasets, compared to the one GPU baseline and our straightforward distributed system with a conventional batch size of one and no load balancing. Although good speed can be achieved without load balancing, load balancing allows us to consistently achieve even higher throughput across various types and scales of scenes.

## 6 Related Works

Large-scale scene reconstruction.
Prior work has proposed systems that use the divide-and-conquer approach to scale 3DGS to work with large scenes. VastGaussian [], CityGaussisan [], and Hierarchical Gaussian [] divide large scenes into small regions, train on these smaller regions on a single GPU, and then merge the resulting images. Hierarchical Gaussian [], Octree-GS [] and Scaffold-GS [] describe level-of-detail based approaches to adaptively reduce the number of Gaussians considered for particular scenes. None of these systems can consider a full large-scale scene directly and achieve the same quality as ours. Similar approaches [, , ] have also been used to scale NeRF, but they cannot directly apply to 3DGS which exhibits a different computation pattern as we discussed in Section 1.

Distributed training for neural networks.
Existing work have exploited various types of parallelism to train neural networks across GPUs. These include data parallelism [], tensor parallelism [, ], pipeline parallelism [, ] and FSDP [, ]. Several systems also support multiple types parallelism and/or aim to automatically partition the workload to optimize training speed [, , ].
However, as we discussed earlier (§1), neural network training and 3DGS have very different computation patterns. The former performs repeated layer-wise computation dominated by dense matrix multiply operations while the latter’s 3 stages training process is irregular and sparse. As a result, although Grendel’s distribution strategy may resemble those seen in existing work (e.g., FSDP []), the details are quite different.

Large Batch Size Training.
Large batch training has bee widely adopted to improve the ML training performance and efficiency, but it has also been recognized by prior work [] that increasing batch size can adversely impact model performance. This has led to the development of empirical rules, including linear scaling and learning rate warmum [], and layer-wised adaptive rate scaling [, ]. Theoretical work [, , ] have also predicted and validated scaling laws for increasing batch sizes. Our work is inspired by these, but focuses on batch size scaling for 3DGS training.

## 7 Conclusion

We have described Grendel, a distributed 3D Gaussian training system. Our evaluation shows that Grendel can render high-resolution high-quality images from large scenes that are beyond the capabilities of existing approaches, and that it shows nearly perfect performance and memory scalability. We plan to open source Grendel so that the community can more easily experiment with 3DGS based reconstruction techniques for large scenes.

## 8 Acknowledgements

We thank Xichen Pan and Youming Deng for their help on paper writing. We thank Matthias Niessner for his insightful and constructive feedback on our manuscript. We thank Yixuan Li and Lihan Jiang from the MatrixCity team for their assistance in providing initial data points of their dataset. We thank Kaifeng Lyu for discussions on Adam training dynamics analysis. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231.

## Appendix A Additional Preliminaries & Observations Details

This appendix provides additional information about 3DGS, beyond what was covered in §2.

### A.1 Densification Process

Densification is the process by which 3DGS adds more Gaussians to improve details in a particular region. A Gaussian that shows significant position variance across training steps, might either be clones or split. The decision on whether to clone or split depends on whether their scale exceeds a threshold. Hyperparameter determine the start and stop iteration for densification, its frequency, the gradient threshold for initiating densification, and the scale threshold that determines whether to split or clone. To create more Gaussians, we need to increase the stop iteration and frequency, and decrease the gradient threshold for densification. If we aim to capture more details using smaller Gaussians, we should lower the scale threshold to split more Gaussians. The training process also includes pruning strategies such as eliminating Gaussians with low opacity and using opacity reset techniques to remove redundant Gaussians.

### A.2 Z-buffer

The indices of intersecting gaussians for each pixel are stored in a Z-buffer, used in both forward and backward. This Z-buffer is the switch between View-dependent Gaussian Transformation and Pixel Render. Since a single gaussian can project onto multiple pixels within its footprint, the total size of all pixels’ Z-buffers exceeds both the count of 3DGS and pixels. The Z-buffer itself, along with auxiliary buffers needed for sorting it, etc, consumes significant activation memory. This can also lead to out-of-memory (OOM) errors if the resolution, scene size, or batch size is increased.

### A.3 Mixed Parallelism

In the main text, some steps of 3DGS are not mentioned, but these steps can also be parallelized. The Gaussian transformation backward and gradient updates by the optimizer are also Gaussian-wise computations and will be distributed the same way as the Gaussian transformation forward. Similarly, the Render Backward and Loss Backward computations are pixel-wise and will be distributed just like the Render Forward.

Regarding the memory aspect, each Gaussian has independent transformed states, gradients, optimizer states, and parameters for each camera view. Therefore, we save these states together on the corresponding GPU that contains their parameters. And activation states like significant Z-buffers, auxiliary buffers for sorting and other functions, loss intermediate activations are managed pixel-wise along with the image distribution.

Regarding densification mechanism, since we clone, split or prune Gaussians independently based on their variance, we perform this process locally on the GPU that stores them.

### A.4 Dynamic Unbalanced Workloads

Physical scenes are naturally sparse on a global scale. Different area has different densities of 3D gaussians (i.e sky and a tree). Thus, the intensity of rendering not only varies from pixel to pixel within an image but also differs between various images, leading to workloads unbalance.

Besides, during the training, gaussians parameters are continuously changing. More precisely, the change of 3D position parameters and co-variance parameters affect each gaussian’s coverage of pixels on the screen. The change of opacity parameters affect the number of gaussians that contribute to each pixel. Both of them lead to render intensity change. The densification process targets areas under construction. During training, simpler scene elements are completed first, allowing more complex parts to be progressively densified. This means Gaussians from different regions densify at varying rates. The dynamic nature of the workloads is more pronounced at the beginning of training, as it initially focuses on constructing the global structure before filling in local details.

The different computational steps have distinct characteristics in terms of workload dynamicity. Even though, the rendering computation is dynamic and unbalanced; computation intensity for loss calculation remains consistent across pixels, and the view-dependent transformation maintains a uniform computational intensity across gaussians. Actually, render forward and backward have different pattern of unbalance and dynamicity. The computational complexity for the forward process scales with the number of 3DGS intersecting the ray. In contrast, the complexity of the backward process depends on Gaussians that contributed to color and loss before reaching opacity saturation, typically those on the first surface. Then, running time for render forward and backward, loss forward and backward have different dominating influence factors, and every step takes a significant amount of time.

## Appendix B Additional Design Details

### B.1 Scheduling Granularity: Pixel Block Size

In our design, we organize these pixels from all the images in a batch into a single row. Then, we divide this row into parts, and each GPU takes care of one part. However, if there are a lot of pixels, the strategy scheduler computation overhead will be very large. So we group the pixels into blocks of 16 by 16 pixels, put these blocks in a row and allocate these blocks instead. The size of block is essentially the scheduling granularity, which is a trade-off between scheduler overhead and uneven workloads due to additional blocks. After scheduling, we will have a 2D boolean array, compute_locally[i][j], indicating whether the pixel block at i-th row and j-th column should be computed by the local GPU. We will then render only the pixels within the blocks where compute_locally is true.

### B.2 Gaussian Distribution Rebalance

An important observation is that distributing pixels to balance runtime doesn’t necessarily balance the number of Gaussians each GPU touches in rendering; So, to minimize total communication volume, GPUs may need to store varying quantity of Gaussians based on the formula above. Specifically, only the forward runtime correlates directly with the number of touched 3DGS; however, the time it takes for pixel-wise loss calculations and rendering backward depends on the quantity of pixels and the count of gaussians that are indeed contributed to the rendered pixel color, respectively. In our experiments, random redistribution leads to fastest training here, even if its overall communication volume is not the minimum solution. Because in our experiment setting, we use NCCL all2all as the underlying communication primitive, which prefers the uniform send and receive volume among different GPU. If we change to use communication primitive that only cares about the total communication volume, then we may need to change to other redistribution strategy.

## Appendix C Additional Experiments Setting and Statistics

### C.1 Statistics for Mip-NeRF 360, Tank&Temples dataset and DeepBlending

**Table 2: Performance Comparison Between Non-Distribution and 4 GPU Distribution**
| Dataset | Scene | 1 GPU (bsz=1) | 4 GPU (bsz=4) |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | PSNR | Throughput | PSNR | Throughput |
| Mip-NeRF360 | counter | 29.16 | 16.25 | 29.19 | 56.24 |
|  | kitchen | 31.49 | 14.24 | 31.40 | 49.16 |
|  | room | 31.51 | 15.82 | 31.18 | 53.36 |
|  | stump | 26.19 | 14.95 | 26.19 | 54.53 |
|  | bicycle | 24.63 | 12.01 | 24.69 | 44.44 |
|  | garden | 26.82 | 12.10 | 26.86 | 45.83 |
|  | bonsai | 32.34 | 17.87 | 32.23 | 61.88 |
|  | flowers | 21.11 | 14.47 | 21.10 | 53.94 |
|  | treehill | 22.38 | 14.78 | 22.43 | 55.31 |
| Tank&Temples | train | 21.84 | 34.72 | 21.75 | 101.69 |
|  | truck | 25.44 | 27.55 | 25.42 | 95.85 |
| DeepBlending | playroom | 30.11 | 21.98 | 30.22 | 75.38 |
|  | drjohnson | 29.15 | 17.74 | 29.19 | 62.11 |

### C.2 Scalability

Table 3, 4 and 5 show the increased reconstruction quality with more gaussians. While many hyperparameters influence the number of Gaussians created by densification, we focused on adjusting three key parameters: (1) the stop iteration for densification, (2) the threshold for initiating densification, and (3) the threshold for deciding whether to split or clone a Gaussian. Initially, we gradually increased the densification stop iteration to 5,000 iterations. However, due to the pruning mechanism, this adjustment alone proved insufficient. Consequently, we also lowered the two thresholds to generate more Gaussians. For a fair comparison, all other densification parameters—such as the interval, start iteration, and opacity reset interval—were kept constant. For the Rubble scene, each experiment run for the same 125 epochs, exposing models to 200,000 images, ensuring consistency. Although training larger models for longer durations and lowering the positional learning rate improved results in my observations, we maintained consistent training steps and learning rates across all experiments to ensure fairness.

Table 6, 7 show the Throughput Scalability by Increasing batch size and leveraging more GPUs, for Rubble and Train scene, respectively. Essentially, more GPUs and larger batch size give higher throughput.

Table 8 demonstrates that additional GPUs increase available memory for more Gaussians, evaluated on the Rubble scene with various batch sizes reflecting different levels of activation memory usage. Essentially, more GPUs provide additional memory to store Gaussians, while a larger batch size increases activation memory usage, leaving less memory available for Gaussians."

**Table 3: Scalablity on Rubble: Gaussian Quantity, Results and Hyperparameter Settings**
|  | Results | Densification Settings |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Experiment | n3dgs | PSNR | SSIM | LPIPS | Stop Iter | Thresholds |
| EXP 1 | 2114045 | 24.84 | 0.70 | 0.48 | 5000 | (0.0002, 0.01) |
| EXP 2 | 5793396 | 25.85 | 0.75 | 0.42 | 15000 | (0.0002, 0.01) |
| EXP 3 | 9173931 | 26.14 | 0.77 | 0.38 | 50000 | (0.0002, 0.01) |
| EXP 4 | 11168630 | 26.28 | 0.78 | 0.37 | 50000 | (0.00018, 0.008) |
| EXP 5 | 15754744 | 26.61 | 0.79 | 0.35 | 50000 | (0.00015, 0.005) |
| EXP 6 | 21177774 | 26.91 | 0.80 | 0.33 | 50000 | (0.00013, 0.003) |
| EXP 7 | 30474202 | 27.06 | 0.82 | 0.31 | 50000 | (0.0001, 0.002) |
| EXP 8 | 40397406 | 27.28 | 0.82 | 0.29 | 50000 | (0.00008, 0.0016) |

**Table 4: MatrixCity Block_All Statistics: Gaussian Quantity, Results and Hyperparameter Settings**
|  | Results | Densification Settings |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Experiment | n3dgs | PSNR | SSIM | LPIPS | # Start Points | # Densify Iter |
| EXP 1 | 1545568 | 24.41 | 0.73 | 0.41 | 1545568 | 0 |
| EXP 2 | 3867136 | 25.36 | 0.77 | 0.36 | 3867136 | 0 |
| EXP 3 | 9485755 | 26.6 | 0.82 | 0.27 | 7743616 | 5000 |
| EXP 4 | 14165332 | 26.78 | 0.83 | 0.25 | 15540941 | 5000 |
| EXP 5 | 24355726 | 27.0 | 0.84 | 0.23 | 15540941 | 30000 |
| EXP 6 | 30074630 | 26.96 | 0.84 | 0.22 | 15540941 | 40000 |

**Table 5: Bicycle Statistics: Gaussian Quantity, Results and Hyperparameter settings**
|  | Results | Densification Settings |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Experiment | n3dgs | PSNR | SSIM | LPIPS | Stop Iter | Thresholds |
| EXP 1 | 2185112 | 24.09 | 0.66 | 0.35 | 5000 | (0.0002, 0.01) |
| EXP 2 | 3035508 | 24.28 | 0.68 | 0.32 | 7000 | (0.0002, 0.01) |
| EXP 3 | 4154806 | 24.59 | 0.70 | 0.29 | 10000 | (0.0002, 0.01) |
| EXP 4 | 5272686 | 24.71 | 0.71 | 0.28 | 15000 | (0.0002, 0.01) |
| EXP 5 | 6579244 | 24.76 | 0.72 | 0.27 | 15000 | (0.00018, 0.008) |
| EXP 6 | 9636072 | 24.85 | 0.73 | 0.25 | 15000 | (0.00015, 0.005) |

**Table 6: Scalability on Rubble: Speed up from More GPU and Larger Batch Size**
| GPU Count | bsz=1 | bsz=2 | bsz=4 | bsz=8 | bsz=16 | bsz=32 | bsz=64 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 GPU | OOM |  |  |  |  |  |  |
| 2 GPU | OOM |  |  |  |  |  |  |
| 4 GPU | 5.55 | 6.52 | 7.28 | OOM |  |  |  |
| 8 GPU |  | 12.56 | 12.55 | 13.74 | OOM |  |  |
| 16 GPU |  |  |  | 22.45 | 24.75 | 25.18 | OOM |
| 32 GPU |  |  |  |  | 36.10 | 39.12 | 38.03 |

**Table 7: Scalability on Train: Speed up from More GPU and Larger Batch Size**
| Experiment | # GPU | Batch Size | Throughput | PSNR |
| --- | --- | --- | --- | --- |
| EXP 1 | 1 | 1 | 34.72 | 21.84 |
| EXP 2 | 4 | 16 | 112.78 | 22.01 |
| EXP 3 | 8 | 16 | 151.52 | 22.09 |
| EXP 4 | 8 | 32 | 159.57 | 21.73 |
| EXP 5 | 16 | 16 | 167.60 | 22.06 |
| EXP 6 | 16 | 32 | 185.19 | 21.76 |

**Table 8: Scalability on Rubble: More Available memory with more GPU**
| GPU Count | bsz=1 | bsz=4 | bsz=16 |
| --- | --- | --- | --- |
| 1 GPU | 12.71 M | 7.10 M | OOM |
| 2 GPU | 31.40 M | 21.80 M | 3.91 M |
| 4 GPU | 63.44 M | 43.48 M | 19.55 M |
| 8 GPU | 116.85 M | 82.31 M | 36.44 M |
| 16 GPU | 230.41 M | 169.37 M | 74.98 M |
| 32 GPU | 354.46 M | 313.10 M | 150.21 M |

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: (1) At line 206, the method splits an image into 16x16 pixel blocks. I'm curious whether the patch size of image blocks can influence the performance of image rendering quality.

(2) Since this method enables training 3DGS from multiple views, it can gather gradients from multiple views and then better constrain the training of 3DGS (as has been shown in the paper *MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis*). However, I saw no performance gains using the distributed training. In lines 312-323, the paper claimed that "even though some cameras share similar poses, a set of random cameras generally observe different parts of a scene, hence the gradients in a batch are mostly sparse and can be thought of as roughly independent.". I think that might be the reason that the distributed training does not improve the rendering quality. Can the authors try to explain/discuss this further? For example, are there any non-random batch-splitting methods that can be applied to improve the performance?
soundness: 3
strengths: (1) This work enables the distributed training of 3DGS, which can be very useful to accelerate the training of 3DGS.

(2) The paper is well-written and easy to understand.

(2) This work did exhaustive experiments on very large-scale datasets (e.g., the MatrixCity dataset)
summary: This paper, named Grendel, presents a system to enable the distributed training of 3DGS. By leveraging the spatial locality and discovering the Gaussian intersection on image pixels, Grendel enables transferring Gaussians with sparse all-to-all communication while balancing the communication overloads. The load balancing can be simply adjusted according to the time estimation of rendering a pixel in previous iterations. While increasing the batch size needs adjusting the hyperparameters for optimizers, Grendel proposed an automatic hyperparameter scaling rule to enable the hyperparameter-tuning-free batched 3DGS training. Experiments conducted on the small-scale Mip-NeRF360, Tansk-and-Temples, and DeepBlending dataset, the large-scale Mega-NeRF dataset, and the large-scale MatrixCity data validated the effectiveness of the proposed method.
weaknesses: (1) The related works are put into the last section of the paper, which is weird to me. It lacks discussion of some existing distributed methods, such as **DOGS** (Chen and Lee, NeurIPS 2024), and RetinaGS(Li, et, al, arXiv 2024).

(2) In Sec.6 Related Works, The discussion of VastGaussian, CityGaussian, and Hierarchical Gaussian is wrong. They do not merge the resulting images. Instead, they merge the sub-models. And the authors also claimed that "None of these systems can consider a full large-scale scene directly and achieve the same quality as ours". This is ***over-claimed and does not respect existing works since they did not compare to any of these methods*** but only tried different configurations of their Grendel methods.

(3) Baselines are missing in the current version. It is a good system work to me. However, over-claiming something in the paper won't make it a better work. I would like to see a fair comparison of the method to more baselines (original 3DGS, VastGaussian, CityGaussian, DOGS, etc.) and raise my score if they do so.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Can the proposed method be applied to 3DGS with LOD design?
2. How does the training across different machines instead of different GPUs affect the performance of the proposed paper?
soundness: 3
strengths: **Motivation**
* This paper has a strong motivation of expanding the training of 3DGS into multiple GPUs/machines. The vanilla 3DGS has a limited number of primitives supported due to the GPU memory limit, but the exceptional performance of 3DGS indicates its great usefulness in large-scale reconstruction. This paper is monumental in providing support in terms of parallel training.

**Method**
* This paper correctly identifies the per-Gaussian and per-pixel stages of the training pipeline of 3DGS. Since it split the set of Gaussians onto each GPU, the first per-Gaussian stage is relatively simple. For the per-pixel stage, the paper provides intuitive and empirical support for the locality property of 3DGS for each pixel. The paper cleverly leverages this property to design a sparse communication to transport the 3DGS across GPU at a low speed and bandwidth sacrifice. This enables the distributed training of 3DGS.
* This paper rigorously analyzed the scaling property of the 3DGS training process, and carefully designed a hyper-parameter setting paradigm based on the independent gradient assumption and empirical support. 
* To further improve the utilization of all GPUs, the load balancing algorithm is introduced to ensure each GPU can adaptively adjust their workload during training, because the number of Gaussians required for each region of the image cannot be predetermined.

**Experiments**
* This paper conducts extensive experiments across various scales of scenes under different experiment settings. The experiment result are well presented to demonstrate the improvement in primitive count, training speed, and rendering quality.
summary: This paper proposes a parallel training pipeline for 3D Gaussian Splatting to be trained on up to 32 GPUs, instead of 1 GPU only. This significantly improves the limit of 3DGS in terms of primitive count, scene scale, scene detail, and training speed. This paper splits the 3DGS training process into per-Gaussian stage and per-pixel stage. In per-Gaussian stage, each GPU has its own Gaussians to calculate the Gaussian transformations, mostly 3D to 2D transformation and color evaluation. In per-pixel stage, each GPU is in charge of a set of pixels for rendering, and performs a sparse communication to fetch the required Gaussians from other GPUs. Lastly, a backward propagation is carried out in a similar manner to pass back the gradients. As a result, the proposed method demonstrates a significantly higher number of Gaussians supported, much faster training speed, and a moderate increase in rendering quality.
weaknesses: **Method**
* The paper follows the vanilla 3DGS design and does not consider the Level of Detail support. Since this paper focuses on the large-scale reconstruction, LOD is essential for downstream applications. It is not clear how the proposed method can support LOD.

**Experiment**
* The quality improvement is not as significant as the number of primitives or the training speed. However, I would like to argue that this is because of the lack of an appropriate dataset to evaluate a very large-scale reconstruction, which limit the performance of this paper. 
* Although the training speed improvement is very significant with this parallel training, large scenes require very time-consuming pose estimation to be done first. The overall training time including the pose estimation might not be improving as much as the reported improvement. This is not the problem of this paper, but does weaken the impact of this paper.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How much does the rendering speed change?

2. By how much does the actual GPU memory requirement per GPU reduce when using batch processing?

3. What is the storage requirement based on the number of Gaussians?

$~$

Minor comments:

The figure reference is inconsistent in line 173.
soundness: 3
strengths: 1. Proposal of parallelization with respect to different stages of the rendering pipeline of 3DGS is interesting.

2. With the proposed method, training 3DGS on high-resolution images is possible.

3. Batch processing can significantly increase the training speed.
summary: This paper proposes Grendel, a parallelized computation system for 3D Gaussian Splatting. Grendel introduces two parallelization strategies designed for different stages of the 3DGS rendering pipeline: partition by Gaussians and partition by pixels. The partition by Gaussians method distributes Gaussians within the point cloud across multiple GPUs, while partition by pixels assigns pixels across GPUs. Communication between these parallelization schemes is conducted through NCCL all2all. To enhance batch optimization, a learning rate scaling rule is proposed. Experiments demonstrate performance improvements through the utilization of more Gaussians compared to 3DGS approaches in Large scene datasets.
weaknesses: 1. This paper lacks a conclusion and limitations, making it difficult to understand its overall contribution contribution and appear incomplete.

2. Similar to the proposed learning rate scaling, RAdam [1] also suggests a method for variance rectification. It also rectifies the learning rate using variance to enable adaptive learning and, I believe, takes batch size into account. However, the advantage of the proposed method remains unclear.

3. The rendering speed is expected to be slow due to the time required for Gaussian transfers between GPUs, however, the paper does not provide details regarding rendering speed.

4. Based on Fig. 14, it appears that increasing batch size does not decrease the maximum number of Gaussians significantly.

5. As shown in Fig. 11, for datasets that are not high-resolution like Rubble, the performance gain becomes marginal even if the number of Gaussians exceeds what can be handled by a single GPU.

[1] Liu et al., *On the Variance of the Adaptive Learning Rate and Beyond*, ICLR 2020

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 2
questions: 1. The legends in Figure 1 are put in inversed order. A modification seems to be required.
2. In Algorithm 1, the meaning of ET_j is confusing and I didn't find a clear definition. An annotation in the graph can optimize the reading experience.
3. In the "Independent Gradients Hypothesis" section, why can we assume E(g_k)=0 over all views? This hypothesis is not that obvious.
soundness: 4
strengths: 1. Originality.  A reasonable parallel training strategy with novel modification for 3DGS based on valuable insights.
2. Quality. Sufficient experimental validation on effectiveness of the method.
3. Significance. Pioneering and well-optimized parallel training implementation for 3DGS, resolving critical challenges in synchronized parallel 3DGS training. On the one hand, GrendelGS avoids tedious model pertaining or data and primitive partitioning in prior works like VastGaussian or CityGaussian. On the other hand, this work enables efficient training when multiple GPUs are available, especially under instant requirements or large-scale scenes.
summary: This paper introduces GrendelGS, which eliminates the bottleneck of 3DGS in parallel training and scaling up. GrendelGS mixes Gaussian-wise and pixel-wise parallelism, so as to fit computation requirements of different stages. It also makes an effort to minimize communication among GPUs and workload imbalance. And it reveals hyperparameter scaling law for 3DGS parallel training.
weaknesses: 1. While the superior efficiency over the original 3DGS is clearly and sufficiently illustrated through experiments, it would be beneficial to show if this synchronized parallel training mechanism shows higher efficiency over existing open-sourced asynchronized paradigms, such as HierarchyGS, OctreeGS, or CityGS. Note that for a fair comparison, the batch size and overall iterations should be aligned. Considering the limited time for rebuttal, a comparison with one existing method on 2-3 representative large-scale scenes is enough.
2. The Independent Gradients Hypothesis is introduced to support the hyperparameter scaling law. It's reasonable that the hypothesis is true on Rubble or even other large-scale scenes, since the views are extremely sparse and approximately independent. However, for condensed views over small scenes or single objects, the hypothesis may not well suit the reality, especially when the batch size grows.  It seems more reasonable to limit the hypothesis to cases with significant view sparsity, or check if the scaling law brings worse results on small scenes.

Meta-review: This paper presents a parallel training method for 3D Gaussian splatting, which significantly improves the training speed and working scene scale. The paper is strongly motivated for scaling the 3DGS. Although the quality improvement is rather limited, the method has a strong merit in rigorous analysis of the scaling property of 3DGS across GPUs, and the effectiveness of the proposed method is significant. The four expert reviewers are consistently happy about the paper, resulting in all positive ratings. The AC agreed with the reviewers' opinions and recommended the acceptance of the paper.

Decision: Accept (Oral)