GSGAN: Adversarial Learning for Hierarchical Generation of 3D Gaussian Splats
Authors: Sangeek Hyun
Jae-Pil Heo
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D GANs
Generative Adversarial Networks
3D Gaussian Splatting
3D Generative Models
Primary area: generative_models
TL;DR: For the first time, we leverage 3D Gaussian representation in 3D GANs for efficient rendering with explicit 3D representation.

Abstract: Most advances in 3D Generative Adversarial Networks (3D GANs) largely depend on ray casting-based volume rendering, which incurs demanding rendering costs. One promising alternative is rasterization-based 3D Gaussian Splatting (3D-GS), providing a much faster rendering speed and explicit 3D representation. In this paper, we exploit Gaussian as a 3D representation for 3D GANs by leveraging its efficient and explicit characteristics. However, in an adversarial framework, we observe that a na\"ive generator architecture suffers from training instability and lacks the capability to adjust the scale of Gaussians. This leads to model divergence and visual artifacts due to the absence of proper guidance for initialized positions of Gaussians and densification to manage their scales adaptively. To address these issues, we introduce GSGAN, a generator architecture with a hierarchical multi-scale Gaussian representation that effectively regularizes the position and scale of generated Gaussians. Specifically, we design a hierarchy of Gaussians where finer-level Gaussians are parameterized by their coarser-level counterparts; the position of finer-level Gaussians would be located near their coarser-level counterparts, and the scale would monotonically decrease as the level becomes finer, modeling both coarse and fine details of the 3D scene. Experimental results demonstrate that ours achieves a significantly faster rendering speed (×100) compared to state-of-the-art 3D consistent GANs with comparable 3D generation capability.

GSGAN: Adversarial Learning for Hierarchical Generation of 3D Gaussian Splats
Sangeek Hyun
Jae-Pil Heo" Sungkyunkwan University
Sungkyunkwan University
Abstract
Most advances in 3D Generative Adversarial Networks (3D GANs) largely depend ray casting-based volume rendering, which incurs demanding rendering costs. One promising alternative rasterization-base 3D Gaussian Splatting (3D-GS), providing much faster rendering speed and explicit 3D representation. In this paper, we exploit Gaussian as a 3D representation for 3D GANs by leveraging its efficient and explicit characteristics. However, in an adversarial framework, we observe that a naive generator architecture suffers from training instability and lacks the capability to adjust the scale of Gaussians. This leads to model divergence and visual artifacts due to the absence of proper guidance for initialized positions of Gaussians and densification to manage their scales adaptively. To address these issues, we introduce GSGAN, a generator architecture with hierarchical multi- scale Gaussian representation that effectively regularizes the position and scale of generated Gaussians. Specifically, we design hierarchy of Gaussians where finer- level Gaussians parameterized by their coarser-level counterparts; the position of finer-level Gaussians would be located near their coarser-level counterparts, and the scale would monotonically decrease as the level becomes finer, modeling both coarse and fine details of the 3D scene. Experimental results demonstrate that ours achieves significantly faster rendering speed (x100) compared to state-of-the-art 3D consistent GANs with comparable 3D generation capability. Project page: https://hse1032. 10tp: wtt.../t....oooogggg......
Introduction
Figure 1: Generated examples from the proposed method (FFHQ-512, AFHQ-Cat-512). Ours synthesize multi-view consistent images with significantly faster rendering speed by leveraging 3D Gaussian representation. We represent a 3D scene as a composite of hierarchical Gaussians, where each level of Gaussian depicts coarse and fine details corresponding its level. To visualize the effects of individual Gaussian, the right-most images are rendered by reducing the scale of Gaussians.
The research field of 3D generative models has recently emerged and shows impressive generation capability in various domains such text-to-3D [1-7] and image-to-3h [8-10]. Among them,
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

3D Generative Adversarial Networks [11-16] are an adversarial learning framework between 3D generator and a 2D discriminator, capable of synthesizing 3D models solely by training with collections of 2D images or with the additional use of their corresponding camera poses. Specifically, the generator synthesizes a 3D model and renders it as a 2D image using given camera parameters, and then the 21) discriminator determines its realness and its match to the given camera pose.
The generally used volume rendering 3D GANs is ray casting [17], which is widely adopted in NeRF literature [18-21]. Ray casting- based rendering demonstrates its prominent capability to model a 3D scene in various domains of research; however, it is also known for its excessive computational costs. Specifically, it requires (H x W x D) point sampling for rendering an image from a single camera view, where H, W, D denotes height, width, the number of sampled points on a ray (depth). This demanding computational cost hinders previous methods in 3D GANs from performing the rendering process at higher resolutions [12] or forces them to use architectures optimized for efficient 3D representation, leading to degraded generation quality [13, 15].
Recently, 3D Gaussian Splatting (3D-GS) [22] has been introduced for the 3D reconstruction task. This method represents a 3D scene as a composition of 3D Gaussians, projecting and a-blending these Gaussians render images. This rasterization-bast
method notably enhances rendering speed. Despite this innovative progress in the 3D reconstruction task, its application in 3D generative models has not yet been studied. One important challenge is the training algorithm of 3D-GS. Unlike the scene-independent and fully differentiable training of NeRFs, 3D-GS requires additional constraints such as the proper initialization of Gaussian positions for given scene by SfM [23] and densification that heuristically manages the scale and number of Gaussians. Since 3D GANs deploy a single generator trained by gradient descent to model the distribution of 3D data, these non-differentiable and scene-dependent characteristics of 3D-GS restrict its application in 3D GANs.
In this paper, we extend the application of 3D Gaussian representation with rasterization 3D GANs, leveraging its efficient rendering speed for high-resolution data. In an adversarial learning framework, we observe that a naïve generator architecture, which simply synthesizes set of Gaussians without any constraints, suffers from training instability and imprecise adjustment of the scale of Gaussians. For example, at the early stage of training, all the Gaussians disappear in the rendered images. or there are visual artifacts created by long-shaped Gaussians despite its convergence.
To address these training difficulties, we devise method regularize the Gaussian representation for 3D GANs, focusing particularly on the position and scale parameters. To this end, we propose a generator architecture with hierarchical Gaussian representation. This hierarchical representation models the Gaussians of adjacent levels to be dependent, encouraging the generator to synthesize the 3D space in coarse-to-fine manner. Specifically, we first introduce a locality constraint whereby the positions of fine-level Gaussians are located near their coarse-level counterpart Gaussians and are parameterized by them. thus reducing the possible positions of newly added Gaussians. Then, we design the scale of Gaussians to monotonically decrease as the level of Gaussians becomes finer, facilitating the generator's 's ability to model the scene in both coarse and fine details. Based on this representation, we propose a generator architecture that effectively implements the hierarchy of Gaussians. With additional architectural details, we validate that the proposed generator successfully synthesizes realistic 3D models (Fig. 1) with stabilized training and enhanced generation quality.
Our contributions are summarized follows:
We firstly exploit a rasterization-basel 3D Gaussian representation for efficient 3D GANs.
We introduce GSGAN, a hierarchical 3D Gaussian representation that regularizes the pa- rameters of Gaussians by building dependencies between Gaussians of adjacent hierarchies, stabilizing the training of Gaussian splatting- 3D GANs.
The proposed method achieves significantly faster rendering speed compared to state-of- the-art 3D GANs while maintaining comparable generation quality and 3D consistency.
2 Related works
2.1 3D Generative Adversarial Networks
3D GANs [11, 16] learn the 3D representation from collection of 2D images with corresponding camera parameters. Recent progress mostly focuses on generator architecture, especially for enhanc-

---

ing efficiency. For example, EG3D [12] introduces the tri tri-plane representation with an additional upsampling 2D image after volume rendering. Differently, there another stream that directly renders high-resolution image without a 2D upsampler. For instance, GRAM [13] learns the iso-surfaces for efficient sampling of points on a ray, based on an implicit-network generator [11]. Additionally, Vox- graf [24] exploits the sparse voxel grid instead of feature fields to represent the 3D scene. Epigraf [15] introduces a patch-wise training scheme to reduce the computational cost of volume rendering at high resolution (e.g.. 512x512) during the training phase. Differently, Mimic3D [25] proposes method to learn 3D representation at high resolution by distilling the 2D upsampling result of EG3D into the nigh-resolution tri- tri-plane. Most recently. WYSIWIG [26] proposes SDF representation- based GANs for reducing the number of sampling points on rays in a high-resolution 3D representation.
While previous methods use ray casting as volume rendering method with demanding rendering costs, we exploit efficient rasterization by adopting 3D Gaussians as 3D representation.
2.2 Gaussian splatting for 3D representation
3D Gaussian Splatting (3D-GS) [22] represents a 3D scene composition of 3D Gaussians. This approach is known for its fast rendering speed, which rasterization- based rendering leads to, as well as its prominent reconstruction quality and faster convergence. Recently, various domains of research utilize 3D-GS, such as human avatar modeling(27-299 facial editing [30], text-to-3D synthesis [1-7], and image-to- -D synthesis [8-10]. These approaches typically substitute the neural radiance field with a Gaussian representation to leverage its efficient characteristics.
The research efforts also try to apply 3D Gaussian representation to an adversarial learning framework, especially with structural prior such as the human body and facial template. For example, Gaussian Shell Maps [31] focus on the task of 3D human generation based on SMPL template [32] with an adversarial learning framework. Given a human body template, the generator learns to synthesize multiple shell maps containing Gaussian parameters, achieving 3D consistent generation with faster rendering speed. Similarly, GGHead [33] focuses on 3D face generation using FLAME template [34]. Similar to Gaussian Shell Maps, they synthesize texture maps containing Gaussian parameters and position offsets by 2D generator based on StyleGAN2 [35]. However, these methods largely depend on the pre-defined template as structural prior.
Differently, in this paper, we focus on extending the application of 3D-GS to Generative Adversarial Networks without any geometric prior such as SfM that 3D-GS traditionally requires.
3 Proposed method
3.1 Preliminaries
3D Generative Adversarial Networks 3D GANs [16, 11, 12] an adversarial learning framework between the 31) generator g(2,0) 0) and 2D) discriminator d(T). Specifically, for a given randomly sampled latent code E IRdx Ps and camera pose 0 E IR ~ Pg. the generator g(z, 0) synthesizes a 3D scene and renders it to a 2D fake image I E #x x3 based on the given camera pose. Then, the discriminator learns to discriminate the real image I, E " and the fake image I,, while the generator learns to deceive the discriminator. In addition, the discriminator often uses the camera pose to encourage to be aware of 3D information by making estimate the camera parameter [13, 36] performing conditional adversarial loss [12].
3D Gaussian splatting 3D Gaussian representation for 3D modeling is recently introduced by 3D Gaussian Splatting [22]. It represents a scene as composition of anisotropic 3D Gaussians estimated by multi-view images and their corresponding camera poses. This Gaussian representation contains parameters such position μ E R a scale E R3, quaternion g E R., an opacity E RR. and a color e R-, In world space, Gaussian is defined as follows:
G(x) = - 1====)
== RSS R".
where R E R1x4 and 1 are rotation matrix and scaling matrix obtained from quaternion q and scale and x is world coordinate.
3

---

denstfy(G" .6')
Parameterization of hierarchical Gaussians
Levd 0
Level -1
Lead
Leva
G
6'= (fl (5
+ 3 As
+
Levd #1
Level =5
Full
G1-1 (em sid gg e a
a²
Coarser-leeel Gaussian g
Finer-level Gaussian 6
Worl coortinate system
Coordinate systemn of coarser-level Gaussian d+11
(b) Example of generated Gaussians of various hierarchy levels
(a) Illustration of hierarchical representation of Gaussians
Figure 2: Illustration and examples of hierarchical Gaussian representation. (a) We parameterize the finer-level Gaussians by the parameters of coarser-level counterparts for regularizing the scale and position of synthesized Gaussians. (b) Example of synthesized Gaussians across multiple hierarchy levels. Gaussians represent coarse or fine details according to its hierarchy level.
To render the image, the color of each pixel C' is determined by blending the contributions of all N 3D Gaussians that overlap with the pixel as follows:
C= Σ c II ag))
al,)
(2)
j=1
where o; is the color of each Gaussian, and a' is blending weight of 2D projection of Gaussian multiplied by a per-point opacity a. The order of Gaussians is sorted by their depth.
3.2 Hierarchical 3D Gaussian representation
Our focus is on utilizing Gaussians as the 3D representation of the generator in 3D GANs. We begin with a simple generator that takes a randomly sampled latent code as input and outputs N Gaussians, without any restrictions. However, in this scenario, we observe that this naïve application suffers from training instability and fails to properly manage the position μ and scale s, as shown in examples in Fig. 8. Therefore, we concentrate on guiding the position and scales that the generator synthesizes.
To this end, we propose a hierarchical structure of Gaussians to effectively regularize the position and scale of Gaussians, illustrated in Fig. 2. Firstly, we define the hierarchy level E {O, L 1}. from coarse to fine levels, where each level contains a set of Gaussian parameters. In detail, we establish dependency between the Gaussian parameters of adjacent levels. For simplification, we explain the dependency between two hierarchically adjacent Gaussians, G' and G where G' originates from -1 We aim model the 3D) representation in coarse-to-fine manner, by assigning coarser- and finer-level Gaussians Gi-1 and G¹ to be responsible for coarser and finer details of the 3D scene, respectively.
For the position parameter, we impose a locality constraint that bounds the position pe of the finer- level Gaussian G with its corresponding Gl-1, Specifically, we introduce the local position parameter ju' defined in the local coordinate system, which is centered, rotated, and scaled by the position ja and scale s' and quaternion q' of the coarser-level Gaussian G'-1, Then, the position p in world space is formulated by transforming the local position ğL as follows:
p R -' ///
where R1 and 5 are rotation and scaling matrix obtained from g' and s'-1, This operation ensures the position of Gaussians at finer levels depends on coarser-level Gaussians, while residing near the location of coarser-level counterparts.
For the scale parameter, we enforce the scale parameter to monotonically decrease to a certain degree as its hierarchy level increases. In detail, we define the scale of the finer-level s' using the relative scale difference against the coarser-level scale s Additionally, we restrict this scale difference s' to always be a vector of negative values. Furthermore, we introduce the constant As which further lowers the scale of the finer-level. This process is defined as follows:
s + + As, where s'As < 0.
(4)

---

Generator architecture
Hierarchical 3D Gausslans
Anchors A r-ot
Gaussions G
AdaIN
Latent code
Const
z ()..J)
Generated Gaussians (N rW +
Allertion Scale (1-
blocky
Gaussi ans [69
NocuL
x
IoGau1
Style w
FC
block,
FC
loGaus
+ [G'
Cunza
NdalN
block,
MLP
Scate (0-12I)
Style w
x²
toGaus
1G"
densify (egn. (3, 5))
(a) Illustration generator architecture with hierarchical 3D Gaussians
(b) black architecture
Figure 3: The architecture of the generator and its block. (a) Generator synthesizes the multiple-level
of anchors and Gaussians, which contains the residual parameters A' and G¹. Anchors are utilized to regularize the finer-level Gaussians, while Gaussians are used for actual rendering. After generating these parameters, we combine them with anchors from previous level A by densify operation, as defined in eqn. 3,4, 4, 5 (green arrow). (b) The generator consists of stacks of blocks, each of which is sequence of attention and MLP layers. The latent code is conditioned on the generator through AdaIN and layerscale, where the modulation and scaling parameters are derived from style code w.
For the other parameters, we additionally define the residual Gaussian parameters a', ,', at level 4, which are added the Gaussian parameters of the previous level as follows:
d'=a''++ g' d'= d' d'd-1+2
We call this hierarchical relationship between G and Gaussians with residual parameters and it enables the generator to model the 3D space
{āl, s², à', "', } G as densify(G'-I,C) in coarse-to-fine manner, where the fine-level Gaussians depict the detailed part of the coarse-level counterparts. Importantly, it stabilizes the training of GANs by significantly reducing the possible positions of Gaussians and encourages the generator to use various scales of Gaussians, thereby boosting the generation capability to model both coarse and fine details.
3.3 GSGAN; the generator architecture with hierarchical 3D Gaussians
In this section, propose generator architecture for leveraging the aforementioned hierarchical structure of Gaussians (Fig. 3). Basically, we adopt a transformer based architecture, composed of stacks of attention and MLP layers, which is generally used architecture handle unstructured 3D point cloud data [37-39].
First of all, we define generator g(2, 0) sequence of generator blocks, where block, denotes the generator block at specific level 1. At the coarsest level 0, block, takes input as N learnable positions const R. Nx3 and latent code z, where N is the number initial Gaussians, then outputs the high-dimensional features zd. Then, for a feature of 7th Gaussian z., we process the feature z! by the output layer toGaus, to obtain Gaussian parameter {uy, s9, q a. c.) G.. For arbitrary level block; takes input as feature x from the previous block and latent code z, then outputs the feature z/y- Importantly, the output block toGaus, does not directly synthesize the Gaussian parameters GJ. Instead, the intermediate output G) contains the local position iil and relative scale difference $), as well as the other residual parameters e/ á/, 9/. This intermediate output G is combined with the corresponding Gaussians G from the previous level to finally synthesize the Gaussian G², at level I, following the operations in Sec. 3.2. This process, which establishes the hierarchical dependency between Gaussians of adjacent level G and G/,, defined as follows:
blocky ;),, G/. toGaus (z1), G densify(G!-11 (.),
where densify operation denotes the combining process of parameters in hierarchically adjacent G!- and Gr mentioned in eqn. 3, 5.
5

---

As the fine-level Gaussians have smaller scales compared to the coarse-level ones, the number of Gaussians should be increased as the hierarchy increases to successfully synthesize the fine details. Thus, we expand the number of Gaussian parameters G to have total of r' N vectors for each parameter, where r is an upsampling ratio. In other words, we define the Gaussian Gj2 to be dependent on G% where j ri + k and & E {0, {0, 1}.
After synthesizing the Gaussian parameters of every level, we use all of them for generating the image (i.e. total (N rN + ... N)) Gaussians are used for rendering). For rendering, use a tile-based rasterizer following 3D-GS [22].
Anchor Gaussians decompose Gaussians for regularization and rendering In the aforemen- tioned architecture, Gaussians are used not only to represent a 3D scene but also to regularize their coarser-level Gaussian counterparts. This means Gaussians must be trained to precisely guide the parameters of finer-level Gaussians while simultaneously depicting the sharp details in real-world images. However, achieving both of these objectives can be challenging. For instance, we observe that the scale of Gaussians can become nearly zero along specific axis, leading to excessively strong regularization on the position of finer-level Gaussians. To handle this issue, we introduce an auxiliary set of Gaussians that only contributes to regularization, instead of actual rendering.
Specifically, we introduce an anchor Gaussian A' for a specific level which has identical parameter- ization to G However, this type of Gaussian is only used for regularization by deploying it as the input of densify. especially for the coarser-level Gaussian input. Therefore, an anchor Gaussian A!-1 only learns to guide the parameters of their finer-level counterpart G), so its usage relieves the effects of strong regularization caused by zero variance. To generate it, we simply make toGausy synthesize two sets of Gaussians, GO and Al, for a given feature zrj. This process is achieved by re-defining the eqn. 6 as follows:
[G), A/} = (4)),
GO density(4"
A = deussit(44"",A.) (7)
As the Gaussians of coarsest level / = 0 does not have their coarser-level counterparts, we define A AQ and G. = (A G.).
Architectural details Following the conditioning convention of previous GANs [35, 40], we utilize the mapping network to modify the latent code into the style code Then, the style code affects the synthesis process by AdaiN [41]. As noted, the generator block
stack of attention and MLP layers, then we replace the layer norm in attention and MLP by AdaIN, following generally used approach for transformer- bassed based GANs [42, 433. For blocks of coarser levels, we utilize the general self-attention without positional encoding the attention mechanism, whereas use local attention [44] for finer levels, as the interaction between -' N points is computationally demanding. For expanding the features in generator blocks after the coarsest level, we simply use the subpixel operation [45] with skip connection and repeat Gaussians G in r times.
One important architectural design is the usage of layerscale [46], which is a learnable vector that adjusts the effects of the residual block by multiplying it by the output of the residual block. Typically, it is initialized by zero-valued vector, removing the effects of layers in the early stage of training. We observe it is essential for stabilizing the position of Gaussians in early iterations. addition, we use the adaptive version of layerscale [47] conditioned by latent code on every attention and MLP layer in the generator.
Also, we use the camera direction as condition for the color layer in toGaus to model the view- dependent characteristics and employ the background generator, which resembles the generator architecture but with reduced capacity and results in the Gaussians located within sphere of a radius of 3, while the foreground resides in the [-1, I] cube. For further details, we elaborate on them the Appendix A.I.
3.4 Training objectives
Similar to previous 3D GANs [12], we adopt the non-saturating adversarial loss [48] with RI regularization [49]. Formally, these objective functions are defined follows:
Lav = Ev-P, P. [f((d((2, 0)){ + E,, - P,, (f(-d(L,) + xl(vd(1.)
(8)

---

Table 1: Quantitative comparison on FFHQ and AFHQ- Cat datasets terms of FID-50K-full and rendering time. We mainly compare ours with the 3D consistent models, except EG3D utilizing the upsampler. FID scores are taken from previous work [25]. Rendering time is measured on single RTX A6000 GPU. "The rendering time of EG3D at 512 resolution consists of the time for volume rendering at 128 resolution and 2D upsampling operations.
3D consistency
FFHQ
AFHQ-Cat
Rendering time (ms) 256x256
Methods
256
512x512
256x256
512x512
512x512
EG3D [12]
4.80
4.70
3.41
2.72
15.5
GRAM [13]
13.8
13.4
GMPI [54]
11.4
8.29
7.67
EpiGRAF [15]
9.71
9.92
6.93
Voxgraf [24]
V
9.60
9.60
GRAM-HD [55]
10.4
7.67
173.0
197.9
Mimic3D [25]
5.14
5.37
4.14
4.29
106.8
402.1
GSGAN (Ours)
6.59
5.60
3.43
3.79
2.7
3.0
where f(t) log((1 exp(-t)) is a softplus function and À is R1 regularization strength.
We additionally guide the 3D information to the discriminator and generator by introducing contrastive learning between pose embedding obtained from the images and camera parameters. Specifically, the discriminator has a pose branch dp that estimates the pose embedding Pz from an input image. Then, we introduce pose encoder that consists of MLP layers and encodes the camera parameter 0 into the pose embedding Pa- Similar to previous work [36]. we utilize the contrastive objective which enhances similarity between corresponding pr and po- Formally, this objective is defined follows:
Cpose = llo exp(sim(p1.P5) P3 P²)/T) 3/T)
Σ (exp(sim p+)/T))
where sim(.,.) is cosine similarity and B is a batch size and P is positive sample corresponding to a pose embedding Py from the image, and T temperature scaling parameter. For the discriminator, we calculate Lpose using real data, while using fake data for training the generator.
Furthermore, we introduce two losses regularize the position of anchor Gaussians in the coarsest level, n2). We first regularize the averaged position of µ4 to be zero for encouraging the center of Gaussians residing near the origin of the world space. Secondly, we reduce the distance between the positions of the K nearest anchor Gaussians to prevent anchor Gaussians from falling apart from the others. These two regularization losses are defined follows:
N
N
K
Lecenter
Σ i=1
Le NK
( KNN( k))1
(10)
where KNN(HA) A. k) the position of 1tth nearest neighbor of jth anchor Gaussian of 14 To sum up, the final objective function f. is as follows:
L - Latv + Aposeppout + Ceenter Auma Crans
(11)
where Apose, Accuters and Aknn are strengths of the corresponding objective function.
4 Experiments
4.1 Experimental settings
Following the experimental settings of previous 3D GANs [12, 25], we use FFHQ [35] and AFHQ- Cat [50] datasets with 256x256 and 512x512 resolutions. For example, we augment the datasets with the horizontal flip and additionally use adaptive data augmentation [51] for AFHQ-Cat dataset, which has limited size. Camera pose labels are obtained from the official repository of EG3D [12], which are predicted by off-the-shelf pose estimators [52, 53]. We train the model from scratch on each dataset. For further implementation details, please refer to Appendix A.

---

Figure 4: Qualitative results of the proposed method with truncation psi (1b 0.7).
GRAM-HD
Mimic3D
Ours
FEHQ
AFHQ
Figure 5: Qualitative comparison with 3D consistent methods with truncation psi ( = 0.7).
4.2 Experimental results
Quantitative results We mainly compare the proposed method with previous 3D consistent GANs, in terms of FID. These methods have strict 3D consistency, which directly renders the high-resolution images from 3D representation without any 2D upsampling operations. As reported in Tab. we validate that the proposed method surpasses most of the previous methods and also achieves comparable generation capability compared to the state-of-the-art, Mimic3D, Especially for AFHQ- Cat dataset, we achieve a much lower FID, even comparable to the non-3D consistent baseline, EG3D. Next, we evaluate the rendering speed enhancement of the proposed method compared to baseline methods. To compute it, we first synthesize 3D representation of each model and measure the processing time of rendering. the baseline methods, achieving more than 100 times faster rendering than Mimic3D in 512x512 resolution. Moreover, it even faster than non-3D consistent baseline FG3D, which the model that exploits 2D) upsampling operation for reducing the efficient rendering. In addition, one important point is that rendering time is almost identical regardless of the image resolution, suggesting that the proposed method can be more effective at higher resolutions. Additionally, the training time of the proposed methods is 28 RTX A6000 days, while the state-of-the-art Mimic3D requires A100 days on the FFHQ-512 dataset. This notable gap in training cost implies that ours can achieve comparable generation capability efficiently in both rendering and training speed.
reported, ours shows significantly faster speeds compared to
Qualitative results We present examples of generated images from the proposed method and the most recent 3D consistent GANs [25, 55], in Fig. 5. In both datasets, we observe that the proposed method successfully synthesizes the multi-view consistent images, validating its capability for synthesizing the realistic 3D scene. Also, we validate ours can generate both coarse and fine details such as coarse details of skin in human facial images and fine details of fur in cat images.
Level-by-level visualization To further understand how the synthesized Gaussians work, we visualize the Gaussians from an individual level. As depicted Fig. 6, we observe that Gaussians capture the image components from overall structure fine details the level increases.

---

1=0 1=1 l=2
1= 4
1= 5
Composited.
Figure 6: Level-by-level visualization of synthesized Gaussians.
Table 2: Comparison of 3D consistency. We compare ours with the most recent 3D consistent GANs [55, 25]. Faces are segmented by off-the-shelf model [56] to remove effects of background.
Fake logit
-10
-15
No constraints
FFHQ-256
FFHQ-512
Clip
PSNR
SSIM
PSNR SSIM
Ours
-25
GRAM-HD
34.45
0.9648
32.15
0.9244
10
15
25
Ticks (4K images tick)
Mimic3D
40.36
0.9927
35.95
0.9822
Figure 7: Fake logits of ablated models the early stage of training.
GSGAN
41.69
0.9883
37.85
0.9695
Comparison with 3D consistency As a 3D generative model, it is important to maintain 3D consistency across different views. To validate the 3D consistency of the synthesized 3D model, we measure how well the generated 3D scene is reconstructed by surface estimation model, following previous works [25, 55]. Specifically, we the surface estimation model to generated multi-view image and compute the reconstruction error of the multi-view view images used for training. For surface estimation, we use Neus2 [57], applying a facial segmentation mask estimated from an off-the-shelf network [56] to eliminate the effects of the background. Furthermore, generator for our method. As shown in Tab. 2, the proposed method significantly outperforms GRAM-HD while achieving performance comparable to Mimic3D. This experiment suggests that our approach generates 3D-consistent results by leveraging the explicit 3D representation provided by 3D Gaussians.
utilize only the foreground
Training stability at the early stage of training We conduct experiment assess the effect of the proposed method on training stability, particularly at the early stage training, by observing the fake logit of the discriminator. As shown in Fig. 7, we observe that the model without any constraints exhibits rapid divergence, accompanied by markedly low fake logit, indicating that the discriminator already distinguishes between real and fake data perfectly. When applying minimal constraint that limits the scale to its predefined maximum, the model does not diverge but still suffers from instability, evidenced by large standard deviation of the logit. In contrast, demonstrates stable training compared to other models. Note that, the standard deviation visualized by 1-a.
Ablaioon study We perform an ablation study on the proposed components, particularly focusing on the proposed hierarchical architecture. When ablating the regularizations, we keep the residual representation other parameters, (q, a, c). In the absence of any constraints, the model exhibited an FID score exceeding 300, signifying a failure to converge, in Tab. 3. With a minimal constraint that clips scales, the model does not diverge but significantly suffers from its low generation capability. observe enhancements in FID, showing the
As we gradually attach the proposed components, effects of the regularization of position and scale and the introduction of anchor Gaussians.
In Fig. 8, we provide visualizations of synthesized Gaussian positions. Initially, simple clipping of the scale allow the model to synthesize images, but leads to visual artifacts due elongated Gaussians with large scales. Additionally, many Gaussians remain outside the scene, likely due to their tendency not to overlap, particularly when large-scale Gaussians exist. After applying position regularization, Gaussians become more densely located, although visual artifacts and unused

---

+ Clipping scale
+ Position reg.
Table 3: Ablation study on FFHQ-256. "Clipping scale" means the model with the maximum limit of scale.
FID
+A nchor Gaussian (Our full modal)
No constraints
300~
Scale reg.
Clipping scale
95.97
Position reg. (eqn. 3)
17.65
Scale reg. (eqn. 4)
13.80
+ Background generator
12.61
Anchor Gaussian
6.59
Figure 8:
Visual comparison with ablated models. Points are visualized by the position of Gaussians.
Gaussians persist. Upon introducing scale regularization, the synthesized images no longer exhibit such artifacts, but the model struggles to capture precise geometries, as evidenced by the point cloud visualizations. Finally, the model incorporating anchor Gaussians successfully synthesizes realistic images while accurately estimating Gaussian positions. Note that, we clip the Gaussian positions to exist within the range of [-1, 1] in the cube.
Additional visualizations We additionally provide 1) additional generated examples, 2) examples with densely changed camera positions, 3) latent interpolation and w+ inversion, 4) visualization of anchor Gaussians, and 5) effects of background generator in Appendix A, and code implementation with additional videos in supplementary zip file, so please refer to them.
Broader Impact and Limitations
The proposed method follows the negative social impact of previous 3D generative models. For example, it may be used for synthesizing fake news or deepfake. Furthermore, since boost the rendering speed of 3D GANs, 100 faster than previous methods, it encourage the generation of them by reducing computation cost for rendering the image.
Broader Impact
Limitations Different from the 3D-GS, which adaptively removes and introduces Gaussians by densification, the proposed method synthesizes a fixed number of Gaussians. This lack of adaptivity in the number of Gaussians remains limitation, as the number of Gaussians the scene it makes. Also, the scale in hierarchical Gaussian representation is somewhat dependent on the hyperparameter such as As. These factors require adjustment of hyperparameters and can affect the performance of the generator.
differ depending
Conclusion
In this paper, we exploit the 3D Gaussian representation in the domain 3D GANs, leveraging fast rendering speed with explicit 3D representation. To relieve the absence of proper initialization and densification process of 3D-GS, we propose hierarchical Gaussian representation which effectively regularizes the position and scale of generated Gaussians. We validate the proposed technique to stabilize the training the generator with 3D Gaussians and encourage the model to learn the precise geometry of 3D scene, achieving almost x100 faster rendering speed with comparable 3D generation capability.
7 Acknowledgements
This work was supported in part by MSITT&KNP//KIPOT (Police Lab 2.0, No. 210121M06), MSIT/ITTP (No. 2022-0-00680, 2020-0-01821, 2019-0-00421, RS-2024-00459618 RS-2024- 00360227, RS-2024-00437100, -S42024-00437633), and MSIT/NRF (No. RS R8-2024-003)7722),
10

---

References
[1] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. sero-shot text-guided object generation with dream fields. Proceedings of the IEEF/CVF conference on computer vision and pattern recogmition, pages 867-876, 2022.
[2] Ben Poole, Ajay Juin, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[3] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. 16653, 2023.
[4] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: figh-fidelity and diverse text-to-3d generation with variational score distillation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[5] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magicšd: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[6] Haochen Wang, Xiaodan Du Jiahao Li. Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12619-12629, 2023.
[7] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d; Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22255,, 2023.
[8] Amit Raj, Srinivas Kaza, Ben Poole. Michael Niemeyer. Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Klir Aberman, Michael Rubinstein. Jonathan Barron, al. Dreambooth3d: Subject- text-to-3d generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2349-2359 2023.
[9] Junshu Tang, Tengfei Wang, Bo Zhang. Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: fidelity 3d creation from single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22819 2211992, 2023.
[10] Luke Melas-Kyriazi, Iro Laina. Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruc- tion of any object from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8446 8455, 2023.
[11] Eric R Chan, Marco Monteiro, Petr Kellnhofer. Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799-5809, 2021.
[12] Fric R Chan, Connor 7 Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry-aware 3d generative adversarial networks. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[13] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10673-10683, 2022.
[14] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint 1 2021.
[15] Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka. Epigraf: Rethinking training of 3d gans. Advances in Neural Information Processing Systems, 35:24487-24501, 2022.
[16] Thu Nguyen-Phuoc Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsuper- vised learning of 3d representations from natural images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7588-7597, 2019.
[17] James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 18(3):155-174, 1984.
[18] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65 (1):99 106, 2021.
11

---

[19] Jonathan T Barron, Ben Mildenhall. Matthew Tancik, Peter Hedman, Ricardo Martin-Brualle and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[20] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedmmn. Mip-nerf' 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[21] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4::1-15, 2022.
[22] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trunsactions on Graphicx, 42(4):1-14, 2023.
[23] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers. pages 535-846. 2006.
[24] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. Advances Neural Information Processing Systems, 35:33999- 34011, 2022.
[25] Xingyu Chen, Yu Deng, and Baoyuan Wang. Mimic3d: Thriving 3d-aware gans via 3d-to-2d 2d imitation. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2288-234,,11 IEEE Computer Society, 2023.
[26] Alex Trevithick, Matthew Chan, Towaki Takikawa, Umar Iqbal. Shalini De Mello, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. What you see is what you gan: Rendering every pixel for high- fidelity geometry in 3d gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22765 22775,, 2024.
[27] Shenhan Qian. Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Simon Giebenhain, and Matthias NieBner. Gaussianavatars: Photorealistic head avatars with rigged 3d gaussians. arXiv preprint arXiv:2312.02069, 2023.
[28] Zhe Li, Zerong Zheng, Lizhen Wang. and Yebin Liu. Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling, arXiv preprint arXiv:231 2023.
[29] Yuelang Xu, Benwang Chen, Zhe Li, Hongwen Zhang, Lizhen Wang, Zerong Zheng, and Yebin Liu. Gaus- sian head avatar: Ultra high-fidelity head avatar via dynamic gaussians. arXiv preprint 23X.v2231..03029, 2023.
[30] Zhenglin Zhou, Fan Ma. Hehe Fan, and Yi Yang. Headstudio: Text animatable head avatars with 3d gaussian splatting. arXiv preprint arXiv:2402.066 49, 2024.
[31] Rameen Abdal, Wang Yifan, Zifan Shi, Yinghao Xu, Ryan Po, Zhengfei Kuang, Qifeng Chen, Rin Yeung, and Gordon Wetzstein. Gaussian shell maps for efficient 3d human generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9441-9451, 2024.
[32] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Pupers: Pushing the Boundaries, Volume 2, pages 851-866. 2023.
[33] Tobias Kirschstein, Simon Giebenhain. Jiapeng Tang, Markos Georgopoulos. and Matthias Nießner. Gghead: Fast and generalizable 3d gaussian heads. arXiv preprint arXiv:2406.09377. 2024.
[34] Tianye Li, Timo Bolkart, Michael. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans, ACM Transactions on Graphics. (Proc. SIGGRAPH Asia), 36(6): 194:1-194:17, 2017. URL https: / //wii.//ooo//oo org/ 10. 1145/3130800 3130813.
[35] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[36] Kyungmin Jo, Wonjoon Jin, Jaegul Choo, Hyunjoon Lee, and Sunghyun Cho. 3d-aware generative model for improved side-view image synthesis. In P Pruceedings of the IEEE/CVF International Conference on Computer Vision, pages 22862-22877 2023.
[37] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
12

---

[38] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
[39] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv: mt.02222..87,,,, 2022.
[40] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 81 5110-8119 19, 2020.
[41] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510 2017.
[42] Kwonjoon Lee, Huiwen Chang, Lu Jiang. Han Zhang, Zhuowen Tu, and Liu. Vitgan: Training gans with vision transformers. arXiv preprint arXiv:2107.04588 2021.
[43] Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. Styleswin: Transformer-based gan for high-resolution image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11304-110 11314, 2022.
[44] Yue Wang, Yonghin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1-12, 2019.
[45] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1874-1883, 2016.
[46] Hugo Touvron, Matthiuu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. Proceedings of the IEFF/CVF international conference on computer vision, pages 32-42, 2021.
[47] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023.
[48] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Farley, Sherjil Ozair, Aaron
[49] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 14940. PMLR, 2018.
[50] Yunjey Choi, Youngjung Uh. Jaejun Yoo. and Jung- Woo Ha. Stargan v2: multiple domains. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 8188 0888881972220 2020.
Diverse image synthesis for
[51] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33: 12104- 12114,114444 2020.
[52] Yu Deng, Jiaolong Yang. Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. Accurate 3d face reconstruc- tion with veakly-supervive learning: From single image image set. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[53] T. B. Lee. Cat hipsterizer, https://ggtthbb.coom http://hhhii.ccceaaaaaaaattttapp
2018.
[54] Xiaoming Zhao, Fangchang Ma. David Gtlrr,, Zhile Ren, Alexander Schwing, and Alex Colburn. Generative multiplane images: Making 2d gan 3d-aware. In European Conference on Computer Vision, pages 18-35. Springer, 2022.
[55] Jianfeng Xiang, Jiaolong Yang. Yu Deng, and Xin Tong. Gram-hd: 3d-consistent image generation high resolution with generative radiance manifolds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2195-2205, 2023.
[56] Changqian Yu, Jingbo Wang, Chao Peng. Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 241, 2018
13

---

[57] Yiming Wang, Qin Han, Mare Habermann, Kostas Danilidis, Christian Theobalt, and Lingjee Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 3295-3306, 2023.
[58] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 1rX1v:1710.10196, 2017.
[59] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? Proceedings of IEEE/CVF international conference on computer vision, pages 4432 4441, 2019.
[60] Daniel Roich, Ron Mokady, Amit Bermano, and Daniel Cohen-Or. Pivotal tuning latent-based editing of real images. ACM Trunsactions on graphics (200), 42(1)::113, 2022.
14

---

A Appendix
A.1
Implementation details
First of all, please refer to the attached code in supplementary for a more detailed implementation of the proposed method. For the coefficients of objective functions, we use X = 1, Apose and Akm = 10. We train the model until the discriminator sees 10-15M images. Differently, we use Acemer I, 10 each for FFHQ and AFHQ-Cat datasets. Following prior works [12, 25J, we also
generator pose conditioning [12] for modeling pose-specific attributes. Other hyperparameters not mentioned are identical to EG3D [12].
For the architectural details, we use an upsampling ratio r = 4 and the initial number of Gaussians N = 256. For the number of hierarchical levels, adopt L for the dataset with 256x256 resolution, and = for 512x51 512 resolution. We use the stack of 6 blocks at the lowest level and 1 block for the other levels, that is, 87K and 349K Gaussians are used for 256 and 512 resolutions. The number of channels is [512, 512, 256, 128, 128, 128] for the block of levels [0. 1, 2, 3, 4, 5], respectively. We use equalized learning rate [58] for every layer. Also, we only apply AdaIN and adaptive layerscale on the layers lower than level 3, as we observe the modulation on higher levels does not affect the rendering results much. For discriminator architecture, we use StyleGAN2 discriminator following EG3D [12], but with an additional pose embedding layer composed of the MLP layer.
For the representation of Gaussian parameters, we follow the 3D-GS [22] that uses the exponential activation function for the scale parameters and normalizes the quaternion to make it valid. As these operations are performed in the rendering process, we omit them for simplicity.
Attention layer We deploy the attention layer for modeling the interaction between the Gaussians. For the lowest levels / O, we utilize the general self-attention. Since the self-attention on a large number of Gaussians requires too much computation cost, we use local attention similar to EdgeConv [44], which computes attention scores against KNN points (k = 16), for the levels / >> 0. As calculating the KNN points also becomes demanding where the number of points to be enlarged, we model the points to interact with the points from the identical parent point, for levels > 3, instead of computing the actual KNN points.
Output layer (toGaus) toGaus layer is basically set of linear layers that synthesize the Gaussian parameter. That is, it consists of 5 linear layers which are parallelly computed to output ju, 8, â, q, c, respectively. They have an input as the intermediate feature at, and an additional camera parameter 0 only for the color é. In addition, the layer for the relative position jú has tanh function as the activation to restrict the range of output position, and the layer for the scale difference & has a softplus function multiplied by -1 as the activation to make it have negative values. Thus, toGaus layer at specific level is defined as follows:
p' = Linp(())), x' tanh(Ling
-ooolllus((inn(()))) e' Lina('', 0), (12)
where E {á, /} and Lin()) is linear layer for the Gaussian parameter. Also, we initialize the bias of linear layers set the quaternion to be an identity rotation matrix and the opacity be 0.1 and the scale to be v
The constant As reducing the scale along with hierarchy is adjusted by the resolution of the given data as below:
As log(V/ x W W/(L VN)), where H, W are the height and width of the image. L is the total level of hicrarchy, and N is the initial number of Gaussians at level = 0.
(13)
Background generator The architecture of the background generator is similar the proposed generator, but the architecture of block is different. In detail, we do not utilize the attention and MLP layers, instead using simple linear layers with demod- and modulation [35] to reduce the computational cost. Also, the position parameters synthesized by the background generator are normalized to make the generated Gaussians located on the sphere. We set the radius of the sphere as 3.
15

---

FFHQ-S12
AFHO-Cat-512
Figure 9: Generated examples FFHQ-512 and AFHQ-Cat-512
For the hyperparameters, we set the number of initial Gaussians = 2000, and the number of hierarchy L 2 with upsampling factor = 4 and the channel of every layer 128. Thus, the background composed of 10,000 Gaussians total.
Training trick for FFHQ dataset We observe that training the FFHQ dataset lacks the capability to accurately model the precise structure of 3D scenes, particularly for structures viewed from different vertical angles. To address this issue, augment the pose distribution during training. Specifically, with a probability of 0.5, we sample poses from a pre-defined distribution characterized by the standard deviation of yaw and pitch (Tyaw, pitch) of the FFHQ dataset. We define the yaw distribution as NO, ryaw) and the pitch distribution as U(-30piich 30chh)), where represents a uniform distribution. This oversampling strategy ensures better coverage of images with varied vertical perspectives.
A.2 Additional examples
present additional examples generated by our method the FFHQ and AFHQ-Cat datasets in Fig. 9, 10. Furthermore, provide multi-view images synthesized from densely sampled camera positions in Fig. 11. Specifically, for each 3D scene, we sample yaw angles in the range of [-0.4, 0.4] radians and pitch angles in the range of [-0.4, 0.1] radians. These multi-view images demonstrate the 3D consistency of our method in qualitative manner. For every visualization, we use truncation psi 0.7 and fix camera condition as frontal view.
A.3 Latent interpolation
One of the notable characteristics of GANs is semantic latent space. To validate that the proposed method also exhibits this characteristic, we synthesize images using linearly interpolated latent codes. As shown in Fig. 12, the interpolated images gradually transition from the given source to the target image, demonstrating smooth and coherent transformations. This indicates that our method maintains mcaningful and semantically rich latent space.
16

---

FFHQ-256
AFHQ-Cal-258
Figure 10: Generated examples on FFHQ-256 and AFHQ-Cat-256
Figure 11: Examples of multí-view generation. We sample the camera pose of [-0.4, 0.4] radian for
yaw, and [-0.4, +0.1] for pitch.
17

---

Figure 12: Linear interpolation between latent codes (w space)
A.4 W+ Inversion
We perform the w+ inversion [59] from given single view in-the-wild image and predict novel view synthesis using the estimated 3D scene. As shown in Fig. 13, the proposed method successfully inverts the in-the-wild image and synthesizes novel views from it, demonstrating the method's ability to generalize and manipulate real-world images. We expect that recent inversion techniques, such as pivotal tuning [60], could further enhance the quality of the inversion.
A.5 Visualization of anchor Gaussians
We visualize the Gaussians and anchor Gaussians level-by-level in Fig. 14. Note that, we manually set the opacity of the anchors as sigmoid(1), as they do not have actual opacity for rendering. As visualized, we observe that the anchors typically have a larger scale compared to the Gaussians. We also verify it by calculating statistics of the scale of anchors and Gaussians. In detail, the average scale of anchors 55.88 across 100 randomly sampled scenes, while the average seale Gaussians e 7.12 We hypothesize that the Gaussians adjust their scales represent sharp details in real-world images, whereas the anchor Gaussians are mainly used to constrain and regularize the parameters of the coarser-level Gaussians.
A.6 Effect of background generator
We visualize Gaussians generated from the background generator Fig. 15. the left side, we synthesize the image with and without the background Gaussians. As visualized, the background Gaussians are responsible for the backgrounds, especially for the blurred area. For more visualization, provide an image rendered by camera far from the world-coordinate origin the right side.
18

---

Target
Inverted result:
Novel view synthesis
Figure 13: Novel view synthesis using w+ inversion.
Anchors A
Result
Gaussians G
Anchors A
Result
Gaussians
Anchors A
Result
Gaussians G
Figure 14: Visualization Gaussians and anchor Gaussians for each level. We set opacity a of anchors sigmoid(1) for visualization. From left-to-right, levels of Gaussians and anchors increase except right-top image, which is composite of Gaussians from all levels.
19

---

w/ background
Visualization of background Gaussians
wo background
Figure 15: Visualization of effects of background generator
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We elaborate on our claims in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We refer to the limitation on sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
This paper does not contain any theory assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report the implementation details sec. A. and reference of datasets in 4.1, and also attach the implementation code in supplementary.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
We attach the implementation code in supplementary and will release it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
wwwiiisssioolooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmie / //nips. wttpccccpppccccceeececccccccccct http:///ggiiieeeeeeeuuuuiiiiii ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We report the detailed training and test details in appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Multiple runs of training are computationally too demanding in 3D GANs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the training resource in Sec. 4, which is in charge of most of the computational cost for the experiments in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /tt.r//eerrpp.eeeee cc
Answer: [Yes]
Justification:
We review the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We elaborate on the broader impacts sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper does not have such risks, as we do not use models and data concerned about any privacy issues.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We check datasets has license of CC BY-NC-SA 4.0 and CC-BY-NC
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not provide assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "yvwMQoaagF", "reviewer": "Reviewer_Abis", "summary": "The paper proposes a method to train 3D GANs using 3D Gaussians. A limitation to training 3D Gaussians in an adversarial training setup is the instability of the scale optimization, which leads to the scale explosion. To address this, the paper proposes a method of hierarchical generation of Gaussians using the Stylegan architecture. As a result, this approach can generate consistent 3D objects at higher rendering speeds. The results are comparable to the volume rendering approaches with the approach surpassing the generation speeds. The authors evaluate their method on two datasets – FFHQ and AFHQ CATs.", "questions": "1) What specific challenges were encountered when attempting to train or generate images at the original resolution of 1024?\n2) Are there plans to demonstrate results at 1024 resolution in future work, and what optimizations might be needed to achieve this?\n3) Can the architecture be adapted or extended to handle high-frequency details better at higher resolutions?\n4) How does the current framework compare to models like GSM[1] and GNARF[2] in terms of articulation and control?\n5) Can the integration of FLAME/SMPL models enhance the current framework's capabilities, and if so, how might this be implemented?\n6) Have you attempted to extract geometry from your models, and if so, what were the results?\n7) How does your approach compare with Eg3d regarding geometry extraction and handling artefacts?\n8) What does running COLMAP on your outputs reveal about the quality and presence of artefacts compared to volume rendering methods?\n9) What are the primary causes of floating artefacts observed in your results?\n10) Are there specific scenarios or conditions where floating artefacts are more prominent?", "limitations": "The Authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper addresses the problem of scale explosion when 3D Gaussians are trained under adversarial losses. The paper addresses this problem by a hierarchical approach to the generation of these Gaussians. This would help in scaling future 3D models to higher texture resolutions.\n\n2) The paper comprehensively compares the existing methods based on volume rendering and shows that the quality and consistency of results are comparable. The rendering speed exceeds the volume rendering methods.\n\n3) The paper shows inversion results that have applications in various single image to 3D generation tasks.", "weaknesses": "1) Why not the original resolution of 1024 for the faces? The authors claim that the generator is 100x faster. The paper does not demonstrate results with this resolution since it fixes the scale. GSMs[1] also mention the scale problem that makes it difficult to train high-frequency details. If the current method addresses the problem, does it scale well to higher resolution?\n\n2) There are many works now employing underlying FLAME/SMPL models like GSM[1], GNARF[2] etc to model faces/bodies. This helps in natural articulation and control. This framework only supports generation with limited articulation and editing. A discussion of these frameworks with limitations and advantages would help.\n\n3) Although the authors show the scaled-down Gaussians for visualization, there is no evidence of extraction of geometry from the models. Some works in the Gaussian domain[3] try to extract the geometry. How does it compare with Eg3d? How does running the COLMAP on the outputs look like? Are there more artefacts than the volume rendering methods?\n\n4) There still seem to be some floating artefacts as seen in the supplementary videos.\n\n[1] Abdal, Rameen, et al. \"Gaussian shell maps for efficient 3d human generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Bergman, Alexander, et al. \"Generative neural articulated radiance fields.\" Advances in Neural Information Processing Systems 35 (2022): 19900-19916.\n[3] https://github.com/MrNeRF/awesome-3D-gaussian-splatting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review.\n\n---\n### “Reason for not using original 1024 resolution / Can this method be extended to high resolution such as 1024x1024?”\n\nFirst of all, please let us clarify that we just follow the most general benchmarks used in EG3D, which uses a maximum resolution of 512x512. To validate the suitability of the proposed method on more high-resolution data, we perform an additional experiment on FFHQ-1024 dataset. Due to the limited rebuttal period and computing resources, we train the model until the discriminator sees 5M images, which is only a ⅓ of the total iteration.\n\nAs a result, we observe that the proposed method achieves FID20K 10.99, already surpassing our baseline GRAM-HD’s 12.0, indicating successful training on high-resolution data.\n\nFurthermore, we provide qualitative examples of FFHQ-1024 dataset in PDF (Fig. 1). As visualized, we observe that the proposed method can handle the high-frequency details with the descent quality of 3D consistency. \n\n---\n### “Articulation and control ability of proposed method compared to template-based 3D GANs / Adaptatiblity of the proposed method with templates such as FLAME/SMPL”\n\nWe first highlight that the proposed method is more fundamental research for extending 3D Gaussian representation on 3D GANs without additional supervision such as templates. This line of research would be beneficial to various research areas. For example, GNARF is also largely based on EG3D, which is a pure 3D GANs without any usage of templates. \n\nIn this context, we believe template-based methods are hard to compare directly with the proposed method. Instead, ours has controllability that leverages various editing techniques based on GANs such as drag-based editing [R#Abis-1], text-based editing [R#Abis-2], and other editing using its semantic latent space [R#Abis-3, 4].\n\nFurthermore, we argue that the proposed method can be easily integrated with templates, as the reviewer mentioned. For example, similar to GNARF, we synthesize templates by the proposed method and warp them by the FLAME/SMPL’s motion distribution pre-computed from training dataset before giving it to the discriminator. This framework would encourage the generator to synthesize the 3D template scene that can be manipulated by motions in FLAME/SMPL.\n\nTo support our claim, we train ours with FLAME in FFHQ-256. For implementation, we subsample 1K points in the fixed basic FLAME template and substitute it with anchors in the coarsest level. Since it increases the number of initial points, we reduce the number of levels by removing a generator block. As a result, we validate it successfully trained by achieving FID 7.32 in a single trial without any hyperparameter tunings, suggesting the promising possibility for integration with template-based methods.\n\n[R#Abis-1] Pan, Xingang, et al. Drag your gan: Interactive point-based manipulation on the generative image manifold. SIGGRAPH 2023.\n\n[R#Abis-2] Lei, Biwen, et al. DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaptation by Combining 3D GANs and Diffusion Priors. CVPR 2024.\n\n[R#Abis-3] Wang, Tengfei, et al. High-fidelity gan inversion for image attribute editing. CVPR 2022.\n\n[R#Abis-4] Ling, Huan, et al. Editgan: High-precision semantic image editing. NeurIPS 2021.\n\n---\n### “Extract geometry from the proposed method / COLMAP on the output of ours”\n\nIn the case of original 3D Gaussian splatting, there is no proper way to extract the geometry such as mesh from trained Gaussians, and it is known as the generated mesh by marching cube shows noisy surface [R#Abis-5]. We have tried to extract geometry using marching cube algorithm following previous work [R#Abis-6], building an opacity field from Gaussians and applying marching cube on it. As shown in Fig. 2 of PDF, we observe the mesh extracted by marching cube seems noisy too.\n\nHowever, there are recent approaches for improving Gaussian splatting to be mesh-extractable, and we believe these methods can be easily integrated into our framework by incorporating regularization terms [R#Abis-5] or altering the output representation to 2D Gaussians [R#Abis-6].\n\nMeanwhile, as ours can synthesize multi-view consistent rendering results, COLMAP on outputs can reconstruct the reliable geometry. We report some examples of COLMAP dense reconstruction of ours in Fig. 3 of PDF. Moreover, we quantitatively compare ours with EG3D in terms of COLMAP (reprojection error, # matched point from sparse recon., # matched points from dense recon.) with 10 random samples. As a result, ours and EG3D achieve (0.440, 3301, 176K) and (0.733, 2360, 156K), respectively, validating the 3D consistency of the proposed method by low errors and large # of matched points.\n\nFloating artifacts are also reconstructed well, as they are the outputs of the generator and represented by 3D Gaussians which is 3D consistent itself.\n\n[R#Abis-5] Guédon et al. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. CVPR 2024\n\n[R#Abis-6] Huang et al. \"2d gaussian splatting for geometrically accurate radiance fields.\" SIGGRAPH. 2024.\n\n\n---\n### “Primary causes of floating artefacts and scenarios for these artefacts.’\n\nWe believe that some elements in the generated results, such as floaters, are not actually artifacts. Instead, they may represent high-frequency details that the background generator should synthesize. We expect these artifacts can be easily solved by proper tuning of the background or introducing other background architecture such as 2D generator with alpha masks.\nWe elaborate on it more in the last question of R#Wn5i.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "868vbEq07Q", "reviewer": "Reviewer_RZRC", "summary": "The paper proposes a hierarchical Gaussian Splatting for 3D GAN. The authors claim that such structure lead to stable training and fast rendering.", "questions": "Please refer to \"weaknesses\".", "limitations": "No negative social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The hierarchical GS structure is interesting.", "weaknesses": "* The motivation to propose such a hierarchical GS structure is not clear. As stated in line 119-122, the design is used to mitigate the instability of position/scale in GS training. It is not an intuitive solution and it is not well interpreted in the paper.  The examples in Fig. 6 are too simple and confusing, please refine the figures to make it easy to understand why the hierarchical design is reasonable.\n\n* The hierarchical structure of Gaussian Splatting is not a new topic.  The literature review about GS and its variants is not sufficient to support the claimed novelty.\n\n* The author claims the rendering speed as their contribution. In my opinion, the fast rendering is the characteristic of GS itself. If the authors claim the proposed hierarchical GS makes it faster,  ablation and discussion are needed as evidence.  As shown in Tab. 1, the selected methods for rendering time comparison are all papers published before GS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive reviews.\n\n---\n### “The motivation of hierarchical GS is not clear and Fig. 6 is confusing to understand”\n\nGANs with 3D Gaussian solve more complex problems compared to previous GANs for 2D image synthesis, as they need to predict various parameters such as 3D position and shape of Gaussians and color at once while previous methods need to estimate the color of fixed positions.\nThis high complexity of the problem makes the solution space more sparse (i.e. all Gaussian parameters such as position and shape should be well-matched with each other), and it degenerates the training stability of GANs since it makes the generator hard to converge to such narrow desirable solution.\nFor example, high-resolution data which has a sparse feasible solution space is hard to train compared to low-resolution one [R#RZRC-1]. From this point of view, we design the generator to synthesize in a coarse-to-fine manner with regularization that reduces the sparsity of solutions by synthesizing Gaussians from coarse-level (easier to solve) to fine-level, similar to the approach of previous work [R#RZRC-1]. \n\nTherefore, we devise a method of regularizing the position and scale while generating Gaussians coarse-to-fine. Detailed rationale of ours is as follows:\n\n1) With the hierarchical architecture, the positions of fine-level Gaussians are restricted by its coarser-level ones, thus possible positions of Gaussians are extremely reduced, making the model to be optimized more easily. Furthermore, as we design the local coordinate system by parameters of coarser Gaussians, finer ones should reside in their coarser Gaussian counterparts. Thus, if the scale of the coarsest Gaussians does not diverge or explode, finer Gaussians also would be located in the scene within a valid range (i.e. they are used for rendering and located in viewing frustum). \n2) Along with the hierarchy, we enforce the scale of finer-level gaussian to be smaller than its coarser-level counterparts. Thus, the generator learns to use various scales of Gaussians for building 3D scenes, leading the generator to depict both coarse and fine details. With this regularization, we can prevent the generator from converging to short-cut such as synthesizing the Gaussians of large scales.\n\nIn Fig. 6, we demonstrate that the naive generator architecture produces images with artifacts from elongated Gaussians of large scales (left in Fig. 6). Additionally, it generates disorganized, invalid positions outside the cube (right in Fig. 6), instead of being well-aligned with the desired 3D scene. Conversely, our proposed method (bottom-right examples) synthesizes sharp textures without such artifacts and maintains structured positions.\nMoreover, models with minimal or no constraints exhibit training instability (Fig. 5). Notably, the diverged model fails to synthesize valid Gaussians, resulting in the disappearance of all Gaussians in the rendered images, as noted in line 50.\n\nWe will carefully revise the figures and explanations. (Fig. 4 in PDF)\n\n[R#RZRC-1] Karras et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation. ICLR 2018\n\n[R#RZRC-2] Karnewar et al. Msg-gan: Multi-scale gradients for generative adversarial networks. CVPR 2020\n\n[R#RZRC-3] Zhang et al. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. CVPR 2017\n\n---\n### “Hierarchical structure of GS is not a new topic and insufficient to show novelty”\n\nFirst of all, we emphasize that the most important contribution of this paper is the first work that extends the application of GS on 3D GANs without any structural priors. To achieve this, we introduce several novel components such as the hierarchical architecture of GS and anchor Gaussians and transformer-based generator architecture.\n\nThanks for noticing us availability of the prior works about the hierarchical structure of GS. \nWith careful searches on GS-related work, we found two works as below:\n\n- Chen et al. GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting. CVPR 2024\n- Kerbl et al. A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets. SIGGRAPH 2024\n\nGaussianEditor is a text-based 3D scene editing method by diffusion-based guidance. With diffusion-based guidance, they introduce the additional objective that reduces L2 distance between the position of the parent node and their child nodes after densification.\n\nThere are several major differences between ours and their method. For example, 1) they aim to solve an editing task, and 2) they require a well-trained 3D-GS and densification process, and 3) their method only adjusts the objective function, and 4) they do not consider the scale of Gaussians. \n\nThe second one solves 3D reconstruction of a very large-scale scene by building a hierarchy between 3D scenes modeled by GSs. So, they pursue a different goal from ours and also they only consider hierarchy between 3D scenes, not the individual Gaussians. Also, it is released on arxiv (17 June 2024)  after NeurIPS submission.\n\nTherefore, we believe ours has its own snovelty. \n\n---\n### “Advantages such as fast rendering is not the contribution of this work / Compared methods are out-dated”\n\nFirst of all, we do not claim that the proposed hierarchical architecture of GS enhances the rendering speed. We argue that the application of GS is hard to extend in 3D GANs, as GS requires some heuristics and structural priors for training and the training can be unstable in an adversarial framework. \n\nTherefore, the major contribution of our work is the extension of GS on 3D GANs, so the advantages of GS compared to NeRFs such as fast rendering and strict 3D consistency are achieved naturally.\nWe also clarify that the proposed method is the first 3D GANs using GS, so there are no papers about GS-based 3D GANs we can compare with, and we compare ours with Mimic3D (Oct 2023) published after GS (Aug 2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OE4btSlB3P", "reviewer": "Reviewer_Wn5i", "summary": "This paper proposes a new 3DGAN based on the 3DGS representation. To maintain stable training and good generation quality, this paper introduces a hierarchical Gaussian to generate the results in a coarse-to-fine manner. Meanwhile, a reasonable transformer-based architecture is proposed to implement the hierarchical Gaussian. Moreover, the proposed Anchor Gaussian is also reasonable. The generation results in both the cat and human domains are good.", "questions": "see the weakness.", "limitations": "see the weakness.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The whole pipeline is reasonable and interesting. To stable 3DGAN training when using 3DGS, it is smart to utilize a hierarchical architecture to predict the parameters of 3DGS in a coarse-to-fine manner.\n2. The Anchor Gaussian can disentangle the rendering and regulation well, which helps the method generate high-quality results.\n3. The transformer-based architecture is suitable to implement the Hierarchical Gaussians.\n4. The results are good and the FID is also comparable. Meanwhile, since the inherent advantage of 3DGS, the proposed method can generate high-resolution images with low time-consuming.", "weaknesses": "1. The L_pose is not used in the EG3D or other baseline methods, and I want to know how this loss affects the training of the proposed method. Meanwhile, the pose encoder of the camera parameter Is trained simultaneously with GAN? \n\n2. Is there a branch that generates the background regions similar to the Panohead? I find some artifacts in the generation results (i.e., some floaters).\n\nOverall, this is a good paper that introduces a reasonable 3DGS-based GAN. I tend to accept the current version. I run the code, the results are good!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive reviews.\n\n---\n### “Effect of $L_\\text{pose}$ and details about the pose encoder”\n\n$L_{pose}$ is used for guiding the pose information to the generator. In detail, for real data, the discriminator learns to estimate the pose embedding corresponding to the given image by contrastive loss. Similarly, for fake data, the generator learns to make the pose embedding of the synthesized image would be estimated to its camera parameter by the discriminator. Thus, we can guide the generator to synthesize the proper 3D scene without conditional adversarial losses EG3D uses.\n\nThe pose encoder is jointly trained with GANs, and it is just a shallow MLP composed of 2 FC layers that simply transform the camera parameters into the high-dimensional feature vector.\n\nBesides, these kinds of pose losses are often used for 3D GANs [R#RZRC-1, 2, 3].\n\n[R#RZRC-1] Deng, Yu, et al. Gram: Generative radiance manifolds for 3d-aware image generation. CVPR 2022.\n\n[R#RZRC-2] Xiang, Jianfeng, et al. Gram-hd: 3d-consistent image generation at high resolution with generative radiance manifolds. ICCV 2023.\n\n[R#RZRC-3] Jo, Kyungmin, et al. 3d-aware generative model for improved side-view image synthesis. ICCV 2023.\n\n\n\n---\n### “Background generator and some artifacts such as floaters”\n\nYes. We deploy the background generator that resembles the architecture of the proposed method, as we want to model the entire 3D scene by gaussian splats. There are some architectural differences as follows: we use modulated FC layers instead of transformer-like blocks, and the positions of background Gaussians are normalized to be located on the sphere with radius=3, while foreground Gaussians would reside in [-1, 1] cube.\n\nAlso, we reduce the channels and the number of upsampling. Other details are available in the Appendix.\n\nWe believe that some artifacts in the generation results like floaters are actually not artifacts, instead, some high-frequency details that the background generator should synthesize.\n\nHowever, since we have focused on building the generator architecture optimized for the foreground, we did not tune the background generator architecture much. For example, we simply use only a single upsampling block with a much smaller number of Gausisans (i.e. total of 10K Gaussians for background).\n\nAs a result, we observe some tendency that the generated Gaussians from the background generator typically have high-scale and blurred shapes and do not model the high-frequency details. \n\nWe believe that this artifact can be easily resolved by introducing a 2D background generator with an alpha mask prediction like Panohead [R#RZRC-4], or sophisticated tuning with the current background generator architecture, or just using a foreground mask to decompose the background.\n\n[R#RZRC-4] An, Sizhe, et al. Panohead: Geometry-aware 3d full-head synthesis in 360deg. CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ULIzRq9GkX", "reviewer": "Reviewer_LD2e", "summary": "The paper presents a 3D-aware GAN (3DGAN) framework using a 3D Gaussian Splatting model.\nGiven a collection of single view 2D images (e.g., FFHQ, AFHQ), the proposed method can train 3D aware images of comparable quality to SOTA NeRF-based method, but with a much faster rendering speed (up to 100x) during inference, thanks to the efficient 3DGS representation. The paper achieves strict 3D consistency as all the pixels are rendered with 3DGS without relying on post 2D super resolution (EG3D). This is among the first works (e.g., Gaussian Shell Maps in CVPR2024) that introduce the 3DGS in 3DGAN framework. \n\nIn details, the paper uses hierarchical gaussian representations, and the 3D representations are learned in a coarse-to-fine manner. \nTo regularize the generation of 3D Gaussians the paper introduces anchor gaussians which will create the next level gaussians, but the anchor gaussians itself were only used for regularization. Additional regularization losses are applied to the coarsest level gaussian anchors to regularize their positions. \n\nIn terms of the generator architecture, the paper borrows similar structures from StyleGANs, similar to other 3D GANs but introduces some attention-based MLP architectures. \n\nThe paper also introduces the layerscale layer, which is initialized to be zero to stabilize the positions of the gaussians in the early stage of the training. Coincidentally, the same strategy was also used in GGHead.", "questions": "-On the most important technical contribution, I did not fully understand why the paper needs to introduce anchor gaussians. Couldn't the same regularizations (if they exist at all) applied to the rendering gaussians themselves? \n-In early 3D GAN research (e.g., pi-GAN), they used fully MLP-based architectures to learn unconstrained 3D space, but later EG3D showed triplanes are a more efficient 3D representations. Did the author consider the use of triplane? \n-How many Gaussians are used in total with all layers and how sensitive the method is for the number of gaussians? I think this is an interesting question many readers would be interested in.\n-it seems that the position regularizations only applied to the coarsest level in Eq (10). What about scale regularizations? What about regularizations to other layers? Please clarify.\n-In Eq (4), why there is an additional delta s, which seems to be doing the same thing as s^{hat_l}? How is delta s < 0 enforced in the loss? This detail seems missing.", "limitations": "The limitations and potential negative impacts are explained. Additional limitation could be the generated image quality is still not on par with post 2D super resolution 3DGAN baseline (EG3D).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-The paper presents a 3D GAN framework using a 3D GS representation. The method achieves strict 3D consistency and a significantly improved speed while achieving comparable FID scores to SOTA NeRF methods (e.g., Mimic3D)\n-learning 3D representations from a single view in the wild image collection is an important research area and this area has improved significantly due to the improved computational efficiency of 3D representation (e.g., triplane in EG3D). The use of 3DGS in 3DGAN will likely boost the overall quality of 3D learning from in the wild 2D images, and the similar method could be applied in the future to learn other domains (e.g., objects, full body etc).\n-The paper seems to have sufficient numerical and qualitative comparisons.\n-The paper is easy to follow.", "weaknesses": "-The related work section is extremely short and is lacking a lot of relevant citations. Please look at recent papers in the similar area \n(e.g., **GGHead: https://arxiv.org/abs/2406.09377**, \n**WYSIWYG: https://research.nvidia.com/labs/nxp/wysiwyg/**), \nand properly cite all the related works. This is a major issue.  \n-Gaussian Shell Maps is also a highly relevant work (CVPR 2024), the code is available here: https://github.com/computational-imaging/GSM\n-Most technical details are provided but some important details seem missing (see below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review. \n\n---\n### “Short length of related work and lack of a lot of relevant citations”\n\nWe apologize for the lack of relevant citations, and will carefully revise the related works section including the papers below:\n\nGaussian Shell Maps (CVPR 2024)\n\nThey focus on the task of 3D human generation based on SMPL template with an adversarial learning framework. With the given human body template, the generator learns to synthesize multiple shell maps containing Gaussian parameters. \nWe believe that the proposed method is more fundamental research about extending the 3D Gaussian representation in the domain of GANs without any structural priors.\nMoreover, we also perform some experiments about the extension of the proposed method using a template such as FLAME face model, so please refer to it.\n\nGGHead (SIGGRAPH 2024)\n\nFirst of all, please note that GGHead is released on arxiv at (Jun 13, 2024) after Neurips submission (May 22, 2024).\n\nGGHead focuses on 3D face generation using FLAME template. Similar to Gaussian Shell Maps, they synthesize texture maps containing Gaussian parameters and position offsets. They also suggest initializing position offsets to 0, akin to our proposed method utilizing layer-scale in the generator architecture.\nTheir framework also requires the use of templates with a foreground mask. As mentioned earlier, we believe our work represents valuable and fundamental research of GS on 3D GANs.\n\nWYSIWYG (CVPR 2024)\n\nWYSIWYG is 3D GANs based on SDF representation. They first learn to volume rendering (VR) at low-resolution (LR) with uniform sampling. Then, they distill the LR weights for VR  to a high-resolution (HR) volume to reduce the number of sampling points on rays, achieving the feasible speed of rendering directly at HR.\nThey show prominent FID on FFHQ-512, but ours surpasses them in terms of FID on AFHQ-Cat-512. Furthermore, ours still achieves a much faster rendering speed almost 100 times compared to WYSIWYG which requires ~222ms for rendering a single image at 512 resolution with a A100 GPU.\n\n\n---\n### “Motivation and importance of anchor Gausisans”\n\nFirst of all, we clarify that regularizations by loss functions (except adversarial and pose losses) apply only to the positions of the coarsest anchor Gaussians.\n\nWe observe that Gaussians used for rendering tend to become overly sharp and elongated, similar to findings in Gaussian Splatting [R#LD2e-1], with scales toward a specific axis approaching zero. This occurs because real-world textures are generally sharp, necessitating sharp Gaussians.\n\nWhile this is not an issue in typical Gaussian splatting for 3D reconstruction, it poses problems for hierarchical Gaussians by excessively narrowing the possible positions of coarser Gaussians. For example, if a scale for a specific axis becomes zero, coarse Gaussians are limited to a plane rather than a volume. As visualized in Fig. 6 of manuscript, we empirically validate it degrades the quality of positions without the anchor Gaussians. An alternative approach is to prevent Gaussians from becoming too small, but it might cause images to be blurred and degrade visual quality.\n\nThus, we separate Gaussians for rendering from those used for regularization. Anchors, not involved in rendering, do not become overly sharp and can guide coarser Gaussian positions effectively. As shown in Appendix (lines 557-558), anchors have a higher scale than rendering Gaussians.\n\nFurthermore, this method requires minimal additional computation since we only split the output layers ($\\text{toGaus}$), and anchors not being used for rendering incur no extra rendering cost.\n\n[R#LD2e-1] Feng, Yutao, et al. \"Gaussian splashing: Dynamic fluid synthesis with gaussian splatting.\" CVPR 2024.\n\n---\n### “Consideration to use triplane representation”\nInitially, we considered using triplane representation. Following EG3D, we synthesized triplanes with a StyleGAN2 generator and estimated warping offsets and Gaussian parameters from voxel features. While this approach worked, it underperformed compared to a transformer-based architecture.\n\nWe hypothesize that MLPs are more efficient for modeling scenes with Gaussian representations compared to NeRF-based methods, which require modeling density and color for every coordinate, including empty space. In contrast, our approach focuses only on the positions of Gaussians.\n\nMoreover, a key architectural difference is that MLP-based methods such as Pi-GAN cannot use attention, performing attention between every sampled point on rays is computationally infeasible. In our method, however, Gaussians interact through attention operations.\n\n---\n### “The number of Gaussians and the sensitivity of method about the number of Gaussians”\n\nWe use a total of 87296 and 349440 Gaussians for 256 and 512 resolutions, respectively (i.e. 5 and 6 blocks for each 256 and 512 resolution).\n\nWe experimented by halving the number of Gaussians in FFHQ-256, by reducing the initial Gaussians from 256 to 128 (i.e. total 43648 Gaussians). As a result, the model with the reduced Gaussians achieves FID 7.99, a degraded FID compared to the original FID 6.59, since the representation power is reduced. However, we observe that the synthesized results are still visually plausible with multi-view consistency.\n\n---\n### “Details about the regularizations including the position and scale regularizations / reason for $\\Delta$ s”\n\nWe apply regularization losses only on the coarsest-level Gaussians. For the other parameters (scale) or layers, we only use adversarial and pose losses. That is, scale parameters are regularized by architectural design.\n\nFor $\\Delta s$, we aim to ensure the scale of fine Gaussian would be smaller than its coarser counterpart. To this end, we introduce $\\Delta s$ that minimizes the scale within a certain degree. We parameterize it by the image resolution and the number of coarsest Gaussians as noted in eqn. 13 in Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to all reviewers for their constructive reviews.

We made a rebuttal for each reviewer, so please refer to them.

Additionally, we provide a single PDF containing the below contents.

1) Examples from FFHQ-1024. 

2) Predicted meshes from the generated 3D Gaussians.

3) COLMAP dense reconstruction examples of the proposed method.

4) Detailed caption of Fig. 6 of the manuscript.


For a detailed explanation about each figure, please refer to a rebuttal for the below reviewers.

(1, 2, 3) - Reviewer Abis
(4) - Reviewer RZRC

Meta-review: The paper presents a 3D-aware GAN that learns to generate 3D Gaussian Splats  from a collection of single view 2D images. The method inherits the fast rendering speed from 3DGS and achieves comparable image quality to SOTA NeRF-based methods.

The reviewers are positive about this work. They consider the approach interesting, the presentation easy to follow and the results convincing. One concern was the related work section, which the authors addressed in the rebuttal and promised to update for the final version.

I recommend acceptance.
Decision: Accept (poster)