Gaussian Graph Network: Learning Efficient and Generalizable Gaussian Representations from Multi-view Images
Authors: Shengjun Zhang
Xin Fei
Fangfu Liu
Haixu Song
Yueqi Duan
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
novel view synthesis
Primary area: machine_vision

Abstract: 3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis performance. While conventional methods require per-scene optimization, more recently several feed-forward methods have been proposed to generate pixel-aligned Gaussian representations with a learnable network, which are generalizable to different scenes. However, these methods simply combine pixel-aligned Gaussians from multiple views as scene representations, thereby leading to artifacts and extra memory cost without fully capturing the relations of Gaussians from different images. In this paper, we propose Gaussian Graph Network (GGN) to generate efficient and generalizable Gaussian representations. Specifically, we construct Gaussian Graphs to model the relations of Gaussian groups from different views. To support message passing at Gaussian level, we reformulate the basic graph operations over Gaussian representations, enabling each Gaussian to benefit from its connected Gaussian groups with Gaussian feature fusion. Furthermore, we design a Gaussian pooling layer to aggregate various Gaussian groups for efficient representations. We conduct experiments on the large-scale RealEstate10K and ACID datasets to demonstrate the efficiency and generalization of our method. Compared to the state-of-the-art methods, our model uses fewer Gaussians and achieves better image quality with higher rendering speed.

Gaussian Graph Network: Learning Efficient and Generalizable Gaussian Representations from Multi-view Images
Shengjun Zhang, Xin Fei, Fangfu Liu, Haixu Song, Yueqi Duan"
Tsinghua University
(zhangsj23, feix21) @mails. tsinghua. edu. cn, duanyueqi@tsinghua edu. cn
Abstract
3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis performance. While conventional methods require per-scene optimization, recently several feed-forward methods have been proposed to generate pixel-aligned Gaussian representations with learnable network, which generalizable different scenes. However, these methods simply combine pixel-aligned Gaussians from multiple views scene representations, thereby leading artifacts and extra memory cost without fully capturing the relations of Gaussians from different images. In this paper, we propose Gaussian Graph Network (GGN) generate efficient and generalizable Gaussian representations. Specifically, we construct Gaussian Graphs model the relations Gaussian groups from different views. To support message passing at Gaussian level, we reformulate the basic graph operations over Gaussian representations, enabling each Gaussian to benefit from its connected Gaussian groups with Gaussian feature fusion. Furthermore, we design Gaussian pooling layer to aggregate various Gaussian groups for efficient representations. We conduct experiments on the large-scale RealEstate and ACID datasets to demonstrate the efficiency and generalization of our method. Compared to the state-of-the-ar methods, our model uses fewer Gaussians and achieves better image quality with higher rendering speed.
Introduction
Novel view synthesis is fundamental problem in computer vision due to its widespread applications, such virtual reality, augmented reality, robotics and
Remarkable progress has been made using neural implicit representations [30 33 40), but these methods suffer from expensive time
consumption in training and rendering 26 12
60 Io [5] 3T. Recently. 3D Gaussian Splatting (3DGS) 19 has drawn increasing attention for explicit Gaussian representations and real-time rendering performance. Benefiting from rasterization-base rendering. 3DGS avoids dense points querying in scene space, so that it can maintain high efficiency and quality.
37
Since 3DGS relies per-subject [19 per-frame 28 parameter optimization, several generaliz- able methods [63] 6 41 are proposed to directly regress Gaussian parameters with feed-forward networks. Typically, these methods 6 generate pixel-aligned Gaussians with U-Net architectures, epipolar transformers or cost volume representations for depth estimation and parameter predictions, and directly combine Gaussian groups obtained from different views scene representations. How- ever, such combination of Gaussians leads to superfiuous representations, where the overlapped regions covered by similar Gaussians predicted separately from multiple images. While simple solution is to delete redundant Gaussians, it ignores the connection among Gaussian groups. As
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ground Truth
PixelSplatt (7N6k)
MVSplat (262K)
MVSplat (81K)
Clurs (81K)
Mitheds
PSNR
Vinoe
22
1000
2500
Pixel-wise Gaussian
Gaussian Graph
Gaussians (K)
(b)
(c)
Figure 1: Comparison of previous methods and ours. (a) We visualize the rendering results of various methods and report the number of Gaussians in parentheses. (b) Previous pixel-wise methods can be considered degraded case of Gaussian Graphs without edges. (c) We report PSNR well the number Gaussians for pixelSplat [4, MVSplat
[6] and GNN under different input settings.
illustrated in Figure Tal pixelSplat 4 and MVSplat (6 suffer from artifacts with several times as many Gaussians as ours, while the deletion of similar Gaussians hurts rendering quality.
To tackle the challenge, we propose Gaussian Graphs to model the relations of Gaussian groups from multiple views. Based on this structure, we present Gaussian Graph Network (GGN), extending conventional graph operations to Gaussian domain, that Gaussians from different views not independent but can learn from their neighbor groups. Precisely, we reformulate the scalar weight of edge weight matrix to depict the interactions between two Gaussian groups, and introduce Gaussian pooling strategy to aggregate Gaussians. Under this definition, previous methods 4 6 can be considered degraded case of Gaussian Graphs without edges. As shown in Figure lb GGN allows message passing and aggregation across Gaussians for efficiency representations.
We conduct extensive experiments on both indoor and outdoor datasets, including RealEstate [64 and ACID 224] While the performance of previous methods declines as the number of input views increases, our method can benefit from more input views. As shown in Figure I our model outperforms previous methods under different input settings with higher rendering quality and fewer Gaussian representations. Our main contributions can be summarized as follows:
We propose Gaussian Graphs to construct the relations different Gaussian groups, where each node is a set of pixel-aligned Gaussians from input view.
We introduce Gaussian Graph Network to process Gaussian Graphs by extending the graph operations Gaussian domain, bridging the interaction and aggregation across Gaussian groups.
Experimental results illustrate that our method can generate efficient and generalizable Gaussian representations. Our model requires fewer Gaussians and achieves better rendering quality.
Related Works
2.1 Neural Implicit Representations
Early researches focus on capturing dense views reconstruct scenes, while neural implicit represen- tations have significantly advanced neural processing for 3D data and multi-view images, leading
to high reconstruction and rendering quality (29) 35 59 40). particular, Neural Radiance Fields
(NeRF) [30] has garnered considerable attention with fully connected neural network represent

---

complex 3D scenes. Subsequently, following works have emerged to address NeRF's limitations and enhance its performance. Some studies aim to solve the long-standing problem of novel view synthesis by improving the speed and efficiency of training and inference [5 12
31
37
26
60
I0]
Other research focuses on modeling complex geometry and riew-dependent effects to reconstruct
dynamic scenes 7 20 36
44
51
23
48
43 21 47 Additionally, some studies 146 42 56
52
have worked on reconstructing large urban scenes to avoid blurred renderings without fine details, which is challenge for NeRF-based methods due to their limited model capacity. Furthermore, other works [33] 45 50 55 apply NeRF to novel view synthesis with sparse input views by incorporating
additional training regularizations, such depth, correspondence, and diffusion models.
2.2 3D Gaussian Splatting
More recently, 3D Gaussian Splatting (3DGS) [9] has drawn significant attention in the field of computer graphics, especially in the realm of novel view synthesis. Different from the expensive volume sampling strategy in NeRF, 3DGS utilizes a much more efficient rasterization based splatting approach render novel views from a of 3D Gaussian primitives. However, 3DGS may suffer from artifacts, which occur during the splatting process. Thus, several works [57
11
17 22
have
been proposed to enhance the quality and realness of rendered novel views. Since 3DGS requires millions of parameters to represent a single scene, resulting in significant storage demands, some researches (27 13 9 [8]
focus on reducing the memory usage to ensure real-time rendering while maintaining the rendering quality. Other works 665 [54 are proposed to reduce the amount of required images to reconstruct scene. Besides, to extend the capability of 3D Gaussians from representing
static scenes to 4D scenarios, several works 44 8 58 53 have been proposed to incorporate faithful
dynamics which are aligned with real-world physics.
2.3 Generalizable Novel View Synthesis
Optimization-biit based methods train a separate model for each individual scene and require dozens of images to synthesis high-quality novel views. In contrast, feed-forwand models learn powerful priors from large-scale datasets, so that 3D reconstruction and view synthesis can be achieved via a single feed-forwar inference. A pioneering work, pixeINeRF [61], proposes feature- -bsed method to employ the encoded features to render novel views. MVSNeRF [5] combines plane-swept cost volumes which is widely used in multi-view stereo with physically based volume rendering to further improve the quality of neural radiance field reconstruction. Subsequently, many researches 2 3 14
16 34 38 25 have developed novel view synthesis methods with generalization and decomposition abilities. Several generalizable 3D Gaussian Splatting methods have also been proposed to leverage sparse view images to reconstruct novel scenes. While Splatter Image [1] and GPS-Gaussian [ regress pixel-aligned Gaussian parameters to reconstruct single objects or humans instead of complex scenes, pixelSplat [4] takes sparse views input to predict Gaussian parameters by leveraging epipolar geometry and depth estimation. MVSplat 6 constructs a cost volume structure to directly predict depth from cross-view features, further improving the geometric quality.
However, these methods follow the paradigm of regressing pixel-aligned Gaussians and combine Gaussians from different views directly, resulting in an excessive number of Gaussians when the model processes multi-view inputs. comparison, we construct Gaussian Graphs to model the relations of Gaussian groups. Furthermore, we introduce Gaussian Graph Network with specifically designed graph operations to ensure the interaction and aggregation across Gaussian groups. this manner, can obtain efficient and generalizable Gaussian representations from multi-view images.
3
Method
After predicting positions and features of Gaussian groups from input views with extracted feature maps, we build a Gaussian Graph to model the relations. Then, we process the graph with Gaussian Graph Network via our designed graph operations on Gaussian domain for information exchange and aggregation across Gaussian groups We leverage the fused Gaussians features to predict other Gaussian parameters, including opacity a, covariance matrix E and colors c.
The overall framework is illustrated in Figure 2
3

---

Input Images
Maps
Goussian Node
Position
image Encoder
Feature Predictor
(..
Predictor
Graph
Figure 2: Overview of Gaussian Graph Network. Given multiple input images, extract image
features and predict the means and features of pixel-aligned Gaussians. Then, we construct Gaussian Graph to model the relations between different Gaussian nodes. We introduce Gaussian Graph Network to process our Gaussian Graph. The parameter predictor generates Gaussians parameters from the output Gaussian features.
3.1 Building Gaussian Graphs
Given N input images I {I+} E RNNHxxX33 and their corresponding camera parameters
C = {c,, we follow the instructions of pixelSplat [ and MVSplat [6 to extract image features:
F= Dimge (I),
[F. Nx
where o
image a 2D backbone. We predict the means and features of pixel-aligned Gaussians:
J = Humproj depth (Fi), E "wwx3 Ji == t feat
F.) E R "rWD,
where T Pdepth and T feat stand for neural networks to predict depth maps and Gaussian features, and Vunproj is the unprojection operation.
We build Gaussian Graph G' with nodes V = {u:} (us, fi)) To model the relations between nodes, we define its adjacency matrix [a:j]NNN as:
i=
0ij dovertap (U; 1y), i x
where Voverlap Ue, U,) computes the overlap ratio between view and view j. To limit further computational complexity, we prune the graph by preserving edges with top weights and ignore other possible edges. The degree matrix D [diy] NXN satisfies d, E, a.j. Thus, the scaled adjacency can be formulated as:
Ā= or D=
3.2 Gaussian Graph Network
Linear Layers. Assuming that a conventional graph has N nodes with features {or} E RNxC and the scaled adjacency matrix A [āg]NNNN the basic linear operation can be formulated
0i Σ W E R
(5)
where W E RCxD is learnable weight. Different from the vector nodes in conventional graphs, each node our Gaussian Graph contains set of pixel-aligned Gaussians es {14, fe). Therefore. extend the scalar weight ask of an edge matrix Es-* [e H HWW. which depicts the detailed relations at Gaussian-leve between U, and 24- For the sake of simplicity, we define
11, (proosMesss Co) Uproy
=
0, 10
Upros (Ho,is co) + Upros (urrs)

---

Algorithm Gaussian Graph Network
Input: Multi-view images I = {Ti), camera parameters C = {ci). the number of graph layers h
Output: Gaussian parameters (/4, 0,,
F <- 4 smage (I,
μ4 - Pumproj (#depth(F)).)) (F)
fo + seac(FF) i= 1,2, .. ,N
A + D
for < 1 to h do
4: - (Σ aig E (W) i= 1,1,,,..,,
end for (H, t U' 1=1 Spooting (G*)
R. S, a, (f) Us Cs.
Σ RSS R
where sss the center of the i-th Gaussian in Gaussian group Us. and prry is the projection function which returns coordinates of the occupied pixel. In this manner, the linear operation Gaussian Graph can be defined as:
N
f
Das - LWF
Pooling layers. we have a series of connected Gaussian Graphs {G" JG")is where G* C G, then operate the specific pooling operation each G and combine them together:
pooting (G) = U Opooting(G
s=1
If Go only contains one node v², Spooting (G*) v² Otherwise, we start with random selected node uf = (nf f()) G Since G is connected graph, we can randomly pick up its neighbor oz with corresponding camera parameters ez. We define a function d'siiitarity.
max || po:
M. ll max e 1
siimitarity (v). uf)
(9)
j-ti 0 max n
where of is the m-th Gaussian in node un-
We define the merge function thmerge:
Ø
Vaimilority
(vf,m- v7) <
Umerge (Uj ei²)
(10)
j.mm
otherwise
Then, merge nodes together get a new node
FW
U
Umerge (uj.m u?) U
(11)
new
m=1
In this manner, aggregate connected graph G step by step to one node. Specifically, previous methods can be considered as degraded Gaussian Graph without edges:
N
N
Opooling (G) = U
Spooting (G)) = U v..
(12)
s=1
where Opooting degenerates simple combination of nodes.
Parameter prediction. After aggregating the Gaussian Graph (μ, f) d'polting(G), we predict the rotation matrix R, scale matrix S, opacity a and color c:
R = @n(f), = Φs(f), a a= a= c= Φc(f)
(13)
where dn. øs. oh and dh are prediction heads. The covariance matrix can be formulated as:
Σ= RSS RT.
(14)
Our Gaussian Graph Network illustrated in Algorithm where a stands for non-linear operations, such ReLU and GeLU.
5

---

Experiments
We validate proposed Gaussian Graph Network for novel view synthesis. In Section 4.1 compare various methods on RealEstate 10K [64] and ACID [24) under different input settings. In Section 4.2 analyze the efficiency of representations. In Section 4.3 further validate the generalization of our Gaussian representations under cross dataset evaluation. In Section 4.4 we ablate our designed operations of Gaussian Graph Network.
Table Quantitative comparison on RealEstate 10K benchmarks. We evaluate all models with 8, 16 input views. Models accept multi-view inputs, and only preserve Gaussians from two input views for rendering.
Views
Methods
PSNR1
SSIM1
LPIPS.
Gaussians (K)
FPS1
pixelSplat
20.19
0.742
0.224
786
110
pixelSplat
20.84
0.765
0,2217
393
175
4 views
MVSplat
20.86
0.763
0.217
262
197
MVSplat
21.48
0.768
0.213
131
218
Ours
24.76
0.784
0.172
102
227
pixelSplat
18.78
0.690
0.304
1572
64
pixelSplat
20.79
0.754
0.243
393
175
views
MVSplat
19.69
0.768
0,238
524
133
MVSplat
21.39
0.766
0.215
131
218
Ours
25.15
0.793
0.168
126
208
pixelSplat
17.80
0.647
0.320
3175
pixelSplat
20.75
0.754
0.245
393
175
views
MVSplat
19.18
0.753
0.250
1049
83
Splatt
21.34
0.765
0.215
131
218
Ours
26.18
0.825
0.154
150
190
Table 2: Quantitative comparison on ACID [24] benchmarks. We evaluate all models with 4, 8, 16 input views. Models accept multi-view inputs, and only preserve Gaussians from two input views for rendering.
Views
Methods
PSNR
SSIM1
LPIPS
Gaussians (K)
FPS
pixelSplat
20.15
0.704
0.278
786
110
pixelSplat
23.12
0.742
0.219
393
175
views
MVSplat
20.30
0.739
0.246
262
197
MVSplat
23.78
0.742
0.221
131
218
Ours
26.46
0.785
0.175
102
227
pixelSplat
18.84
0.692
0.304
1572
64
pixelSplat
23.07
0.738
0.232
393
175
8 views
MVSplat
19.02
0.705
0.280
524
133
MVSplat
23.72
0.744
0.223
131
218
Ours
26.94
0.793
0.170
126
208
pixelSplat
17.32
0.665
0.313
3175
37
pixelSplat'
23.04
0.694
0.279
393
175
16 views
SVSplat
17.64
0.672
0.313
1049
MVSplat
23.70
0.709
0.278
131
218
Ours
27.69
0.814
0.162
150
190
4.1 Multi-view Scene Reconstruction and Synthesis
Datasets. We conduct experiments on two large-scale datasets, including RealEstate 10K and ACID 22.. RealEstate 10K dataset comprises video frames of real estate scenes, which are
(64

---

Input
Ground Truth
PixelSplat
MVSplat
Ours
Figure 3: Visualization results on RealEstate [64] and ACID 24 benchmarks. We evaluate all
models with 4, 16 views as input and subsequently test three target novel views.
split into 67,477 scenes for training and 7,289 scenes for testing. ACID dataset consists of natural landscape scenes, with 1 11,075 training scenes and 1,972 testing scenes. For two-view inputs, model is trained with two views as input, and subsequently tested on three target novel views for each scene. For multi-view inputs, we construct challenging subsets for RealEstate 10K and ACID across a broader range of each scenario to evaluate model performance. We select 4, 8 and 16 views reference views, and evaluate pixelSplat [4) MVSplat (6) and ours the same target novel views.
Implementation details. Our model is trained with two input views for each scene on a single A6000 GPU, utilizing the Adam optimizer. Following the instruction of previous methods 4 6) all experiments are conducted on 256 256 resolutions for fair comparison. The image backbone initialized by the feature extractor and cost volume representations in MVSplat [6]. The number of

---

Table Inference time comparison across different views. We train our model on input views and report the inference time for 4 views, 8 views, and 16 views, respectively.
2 views
views
views
views
pixelSplat [4]
125.4
137.3
298.8
846.5
2938.9
MV Ours
12.0
60.6
126.4
363.2
1239.8
12.5
75.6
148.1
388.8
1267.5
Ours
MVSplat
pixelSplat
Gaussians (M)
200
26
24
3
FPS
150
PSNR
2
100
20
50
18
0
4
12
16
20
24
4
8
12
16
20
24
8
12
16
20
24
Input Views
Input Views
Input Views
Figure 4: Efficiency analysis. We report the number of Gaussians (M), rendering frames per second (FPS) and reconstruction PSNR pixelSplat [4). MVSplat [6] and our GGN.
graph layer is set to 2. The training loss is linear combination of MSE and LPIPS 62 losses, with loss weights and 0.05.
Results. As shown in Table and Table 2, our Gaussian Graph benefits from increasing input views, whereas both pixelSplat 44] and MVSplat [6] exhibit declines in performance. This distinction highlights the superior efficiency of our approach, which achieves more effective 3D representation with significantly fewer Gaussians compared pixel-wise methodologies. For view inputs, our method outperforms MVSplat 6 by about 4dB on PSNR with more than 2x fewer Gaussians. For more input views, the number of Gaussians in previous methods increases linearly, while GGN only requires small increase. Considering that previous methods suffer from the redundancy of Gaussians, we adopt these models with multi-view inputs and preserve Gaussians from two input views for rendering. Under this circumstance, pixelSplat 4 and MVSplat [6] achieve better performance than before, because the redundancy of Gaussians alleviated some degree. However, the lack of interaction at Gaussian level limits the quality of previous methods. In contrast, our Gaussian Graph can significantly enhance performance by leveraging additional information from extra views. Visualization results in Figure 3 also indicate that pixelSplat 4 and MVSplat (6) tend to suffer from artifacts due duplicated and unnecessary Gaussians in local areas, which increasingly affects image quality more input views are added.
train: re10k
test: ACID
train: ACID
test: rel0k
Ground Truth
PixeISplat
MVSplat
Ours
Figure 5: Visualization of model performance for cross-dataset generalization on RealEstate OK
64
and ACID 22 benchmarks.

---

Table 4: Cross-dataset performance and efficiency comparisons on RealEstate 10K 664 and ACID 24 benchmarks. We assign eight views as reference and test on three target views for each scene.
Train Data
Test Data
Method
PSNR
SSIM
LPIPS_
Gaussians (K)
FPS
pixelSplat
18.65
0.715
0.276
1572
ACID
re10k
MVSplat
19.17
0.731
0.270
524
133
Ours
24.75
0.759
0.252
126
208
pixelSplat
19.75
0.724
0.264
1572
64
re10k
ACID
MVSplat
20.58
0.735
0.239
524
133
Ours
25.21
0.762
0.234
126
208
Table 5: Ablation study results of GGN RealEstate 664 reference views and renders three novel views.
benchmarks. Each scene takes eight
Models
PSNR
SSIM
LPIPS
Gaussians (K)
Gaussian Graph Network
25.15
0.786
0.232
126
w/o Gaussian Graph linear layer
24.72
0.778
0,246
126
w/o Gaussian Graph pooling layer
20.25
0.725
0.272
524
Vanilla
19.74
0.721
0.279
524
4.2 Efficiency Analysis.
In addition to delivering superior rendering quality, our Gaussian Graph network also enhances efficiency in 3D representations. As illustrated in Figure when using input images, our model outperforms MVSplat by 8.6dB on PSNR with approximately one-tenth the number of 3D Gaussians and more than three times faster rendering speed. Additionally, we compare the average inference time of our model with pixel-wiss methods in Table 3 Our GGN is able to efficiently remove duplicated Gaussians and enhance Gaussian-lever interactions, which allows it to achieve superior reconstruction performance with comparable inference speed to MVSplat.
4.3 Cross Dataset Generalization.
To further demonstrate the generalization of Gaussian Graph Network, we conduct cross-dataset experiments. Specificallyy, all models are trained on RealEstate 10K [64 or ACID 24] datasets, and are tested on the other dataset without any fine-tuning. Our method constructs the Gaussian Graph according to the relations of input views. As shown Table our GGN consistently outperforms pixelSplat [4] and MVSplat [ on both benchmarks.
4.4 Ablations
To investigate the architecture design of our Gaussian Graph Network, we conduct ablation studies RealEstate10K [64] benchmark. We first introduce vanilla model without Gaussian Graph Network. Then, we simply adopt Gaussian Graph linear layer to model relations of Gaussian groups from multiple views. Furthermore, we simply introduce Gaussian Graph pooling layer to aggregate Gaussian groups to obtain efficient representations. Finally, we add the full Gaussian Graph Network model both remove duplicated Gaussians and enhance Gaussian- level interactions.
Gaussian Graph linear layer. The Gaussian Graph linear layer serves as a pivotal feature fusion block, enabling Gaussian Graph nodes to learn from their neighbor nodes. The absence of linear layers leads to performance drop of 0.43 dB on PSNR.
Gaussian Graph pooling layer. The Gaussian Graph pooling layer is important to avoid duplicate and unnecessary Gaussians, which essential for preventing artifacts and floaters in reconstructions and speeding the view rendering process. As shown in Table 5 the introduction of Gaussian Graph pooling layer improves the rendering quality by 4.9dB on PSNR and reduces the number of Gaussians nearly one-fourth.

---

Conclusion and Discussion
this paper, we propose Gaussian Graph to model the relations of Gaussians from multiple views. To process this graph, we introduce Gaussian Graph Network by extending the conventional graph operations to Gaussian representations. Our designed layers bridge the interaction and aggregation between Gaussian groups to obtain efficient and generalizable Gaussian representations. Experiments demonstrate that our method achieves better rendering quality with fewer Gaussians and higher FPS.
Limitations and future works. Although GGN produces compelling results and outperforms prior works, it has limitations. Because predict pixel-aligned Gaussians for each view, the representations are sensitive to the resolution of input images. For high resolution inputs, 1024 1024, we generate over million Gaussians for each view, which will significantly increase the inference and rendering time. GGN does not address generative modeling of unseen parts of the scene, where generative methods, such as diffusion models can introduced to the framework for extensive generalization. Furthermore, GGN focuses on the color field, which does not fully capture the geometry structures of scenes. Thus, a few directions would be focused in future works.
This work was supported by the National Natural Science Foundation of China under Grant 62206147. We thank David Charatan for his help on experiments in pixelSplat.
Acknowledgemmnts.
10

---

References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, pages 5855-5864, 2021.
[2] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool. Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation. In CVPR, pages 3981-39999 2022.
[3] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, pages 5799-5809, 2021.
[4] David Charatan, Sizhe Li, Andrea Tagliasacchi. and Vincent Sitzmann. pixclsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:2312. 12333 2023.
[5] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pages 14124-14133, 2021.
[6] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint ar0:::2403. 14627, 2024.
[7] Yilun Du, Yinan Zhang. Hlong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and vidco processing. In ICCV, pages 14304-14314 IEEE Computer Society, 2021.
[8] Bardienus Duisterhof, Zhao Mandi, Yunchao Yao, fa- Liu, Mike Zheng Shou, Shuran Song, and Jeffrey Ichnowski. Md-splatting: Learning metric deformation from 4d gaussians in highly deformable scenes. arXiv preprint arXiv:2312.000 2023.
[9] Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv preprint a7X2: 2023.
[10] Sara Fridovich-Keil Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjao Kanazawa. Plenoxels: Radiance fields without neural networks. CVPR, pages 5501-5510, 2022.
[11] Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao. Zhang, and Yao Yao. Relightable gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing, arXiv preprint 2rXiv:2311.1644 2023.
[12] Stephan J Garbin, Marek Kowalski, Matthew Johnson. Jamie Shotton, and Julien Valentin. Fastnerf: High-tidelity neural rendering 200íps. In ICCV, pages 14346-14355. 2021.
[13] Sharath Girish, Kamal Gupta, and Abhinav Shrivastava. Eagles: Efficient accelerated 3d gaussians with lightweight encodings. arXiv preprint arXiv:2312.04564, 2023.
[14] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint 1 2021.
[15] Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientmerf efficient neural radiance fields. In CVPR, pages 12902-12911, 2022.
[16] Wonhong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In ICCV, pages 12949-12958, 2021.
[17] Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. arXiv preprint drXiv:2311.1797..00 2023.
[18] Kai Katsumata, Due Minh Vo. and Hideki Nakayama. An efficient 3d gaussian representation for monocular/multi-vin dynamic scenes. arXiv preprins arXiv:2311.12897, 2023.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ToG, 1-14, 2023.
[20] Zhengqi Li. Simon Niklaus. Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, pages 6498-6508, 2021.

---

[21] Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In CVPR. pages 4273-4284, 2023.
[22] Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. arXiv preprint arXiv:33 164473,,, 16473, 2023.
[23] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPHI Asia, pages 1-9, 2022.
[24] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from single image. In ICCV, pages 14458- 14467, 2021.
[25] Fangfu Liu, Chubin Zhang, Yu Zheng, and Yuegi Duan. Semantic ray: Learning a generalizable semantic field with cross-reprojcttio attention. In CVPR, pages 17386-17396, 2023.
[26] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NewrIPS, 33:15651 13:15651-15663, 2020.
[27] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs. Structured 3d gaussians for view-adaptive rendering. arXiv preprint arXiv:2312.00109, 2023.
[28] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprini arXiv:2308. 09713, 2023.
[29] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In CVPR, pages 4460-4470, 2019.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, Communications of the ACM, 65(1):99-106, 2021.
[31] Thomas Müller, Alex Evans, Christoph Sehied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ToG, -1() 2022.
[32] KI. Navaneet, Kossar Pourahmadi Meibodi, Soroush Abbasi Koohpayegani, and Hamed Pirsiavash. Compact3d: Compressing gaussian splat radiance field models with vector quantization. arXiv preprint arXiv:2311.18152 2023.
[33] Michacl Niemeyer, Jonathan Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, pages 5480-5490, 2022.
[34] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR, pages 1453-11464, 2021.
[35] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, pages 165-174, 2019.
[36] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, pages 10318-10327, 2021.
[37] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In ICCV, pages 14335-14345, 2021.
[38] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. NeurlPS, 333:00154-2016 2020.
[39] Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Frédo Durand. Light field networks: Neural scene representations with singlo-evaluatinn rendering. In NewrIPS, 2021.
[40] Vincent Sitzmann, Michael Zollhofer, and Gordon Wetzstein. Scene representation networks: Continuous
3d structure-awar aware neural scene representations. NeurlPS, 32, 2019.
[41] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. arXiv preprint urXiv:2312.13150 2023.
[42] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, pages 8248 8258, 2022.
12

---

[43] Fengrui Tian, Shaoyi Du, and Yueqi Duan. Mononerf: Learning generalizable dynamic radiance field from monocular videos. In ICCV, pages 17903-17913, 2023.
[44] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. ICCV, pages 12959-12970. 2021.
[45] Prune Truong, Murie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. ieee. CVPR, volume 1, 2023.
[46] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fl-throughs. In CVPR. pages 12922-12931, 2022.
[47] Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. CVPR, pages 12375-12385, 2023.
[48] Liao Wang, Jiakai Zhang, Xinhang Liu, Fugiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time, In CVPR, pages 13524-13534, 2022.
[49] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 2023.
[50] Jamie Wynn and Daniyar Turmukhambetov. Diffusionerf: Regularizing neural radiance fields with denoising diffusion models. In CVPR, pages 4180-4189, 2023. 2023.
[51] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, pages 9421-9431, 2021.
[52] Yuanho Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV, pages 106-122. Springer, 2022.
[53] Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang. and Chenfanfu Jiang. Physgaussiam: Physics-imtegratee 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198. 2023.
[54] Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Sparsegs: Real-time 360 {(deg] sparse view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00206 2023.
[55] Haofei Xu, Ampei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang. Mare Pollefeys, Andreas Geiger, and Fisher Yu. Murf: Multi-baseline radiance fields. arXiv preprint arXiv:23/2.04565, 2023.
[56] Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai. and Dahua Lin. Grid-guided neural radiance fields for large urban scenes. In CVPR, pages 8296-8306, 2023.
[57] Zhiwen Yan, Weng Fei Low, Yu Chen, and Gim Hee Lec. Multi-scale 3d gaussian splatting for anti-aliased rendering, arXiv preprint Xiv: 17089, 2023.
[58] Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:23 10642, 2023.
[59] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurlPS, 33:2492- 2502, 2020.
[60] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In ICCV, pages 5752-5761, 2021.
[61] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixclnerf: Neural radiance fields from one or few images. In CVPR, pages 4578 4587, 2021.
[62] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, pages 066-595,70 2018.
[63] Shunyuan Zheng. Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. Gps- gas-gaussin:: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. arXiv preprint arXiv:2312.02/555. 2023.
13

---

[64] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arX1v:1805.09817, 2018.
[65] Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00451 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide assumptions and proofs both in Section 3 and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We propose a new network architecture and describe clearly and fully in Section 3 Section 4 and appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We will release our codes when we prepare it well.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com htte://allllliiiiee///oeeeuaaass swiniioiscyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide implementation details both Section 4 and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the experimental setting in previous studies, which do not include error bars their experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report information on the computer resources in Section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our paper does not use existing asserts.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new asserts.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "bCxGDmVNmL", "reviewer": "Reviewer_sL12", "summary": "The paper presents an incremental design built upon existing generalizable GS reconstruction frameworks (e.g., PixelSplat, MVSplat) to fusion overlapped pixel-aligned gaussian points from multiple images with a graph-based operator and pooling. This additional design presents a faster and better rendering effect than PixelSplat and MVSplat.", "questions": "Already discussed in weaknesses regard to the training efficiency and scalability.", "limitations": "The authors also mentioned the limitation in model efficiency, especially when increasing the number and resolution of images. Yet this part is actually more important and worth some number to measure this issue. Detailed discussion in weaknesses.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Overall, I agree that the proposed framework is a good choice for fusing pixel-aligned gaussian points from frameworks like PixelSplat or MVSplat. Grouping and fusing these points in 3D space with a graph-based strategy sounds reasonable and effective in solving the artifact issue caused by PixelSplat.\n\n2. The visualization and comparison do present a better novel view synthesis effect with fewer gaussian points when compared with previous efforts like PixelSplat and MVSplat, which results in faster render speed compared with these two works.", "weaknesses": "***[Efficiency and Scalability]*** \n\nAlthough the paper presents a better rendering speed (a natural outcome of reducing gaussian points), it lacks a discussion regarding learning efficiency: the training latency of the proposed design. We know the point-aligned gaussian points are actually quite a large number, especially when increasing the number and resolution of images. However, classical graph-based methods are usually not efficient in handling a large number of points. Further, the limitation on training efficiency might further restrict the stability of the proposed design. Consequently, I hope the author can consider adding a discussion and benchmarking on a breakdown of training and inference latency of the additional graph-based network. It is fine if the current proposed design is not so efficient, but it is definitely worth a discussion.\n\nOn the other hand, the proposed graph network is fundamentally seeking a way to process and fuse points in 3D space. The point cloud processing community already has years of accumulation on handling this kind of data efficiently and effectively. One suggestion is to further enhance performance with the latest efforts in point cloud processing, such as Point Transformer V3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below we address specific questions.\n\n**1. Training and inference latency**\n\nWe agree with you that the analysis of training and inference latency of our proposed method is important.  We discuss this topic in our general rebuttal. We will add this discussion on training and inference latency in Section 4.2 (line 190) as you suggested.\n\n**2. Inspiration from point cloud processing community**\n\nYour suggestion does inspire us a lot. We further add a point cloud processing part from Point Transformer V3 to extract Gaussian features (Model A). As shown in Table 1, such design has improvement on performance. Due to the limitation of rebuttal time, we adopt a simple design, but this design still boost the performance. We argue that this is a promising direction, and we will consider this suggestion as one of our future works to further benefit from the progress in point cloud processing area. \n\nTable 1: Results of PSNR on RealEstate10K benchmark.\n\n|          |  Ours | Model A |\n|:--------:|:-----:|:-------:|\n|  4 views | 24.76 |  24.83  |\n|  8 views | 25.15 |  25.26  |\n| 16 views | 26.18 |  26.30  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uWD1O5LHOG", "reviewer": "Reviewer_LnYN", "summary": "This paper proposes a Graph Neural Network architecture to model the relations between multi-view 3D Gaussians predicted by generalizable 3DGS methods. The proposed method works in particularly better for large number (e.g., 4, 8, 16 etc) of input images, compared with previous methods like pixelSplat and MVSplat that simply combine per-view pixel-aligned Gaussians. The proposed method effectively reduces the redundancy of 3D Gaussians when more input images are given. Experiments are conducted on standard RealEstate10K and ACID datasets and the proposed method performs significantly better than pervious methods for large number of input views.", "questions": "Please refer to the weaknesses.", "limitations": "Yes, the limitations are discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper tackles a common problem in existing generalizable 3DGS models like pixelSplat and MVSplat, where they simply combine per-view 3D Gaussians. This leads to redundancy for more input images. The proposed solution could be applicable to different models and improve their performance.\n\nThe presentation is clear and the experiments are well-designed and extensive. The proposed method achieves strong results on standard benchmarks.", "weaknesses": "A few implementation details are not completely clear. For example, how are the input views selected for different number of input views? Are they sampled to be more densely within some predefined frame range or are they sampled to cover larger regions of the scene? This also leads to another question if the proposed method could work on large-scale scenes since the model could handle many frames now? For instance, if the inputs are 10-20 images capturing a room, could the full room be reconstructed reasonably? I am also wondering if the authors plan to release the source code, which I think would benefit the reproducibility for the research community.\n\nFor results of different number of input views in Table 1 and Figure 4, are they obtained with a single model, or the author trained a specific model for each specific number of input views? \n\nFor efficiency analysis, this paper reported the FPS metric. However, I assume the FPS is measured only for the rendering part right? How about the network inference time and memory consumption, especially for more input images? Since the proposed method is a feed-forward model, where the network forward time should also be considered for benchmarking efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below we address specific questions.\n\n**1. The selection of input views**\n\nAs the number of input views increases, they are sampled from larger regions of the scene. Thank you for point out this question. We will clarify this setting in Section 4.1 (line 157) in our final version.\n\n**2. Large-scale scenes**\n\nWe further evaluate our method on longer videos sampled from large-scale scenes in RealEstate10K. The visualization results are shown in Figure 1 in the one-page pdf. Our method can reconstruct large-scale outdoor scenes and render better novel views than previous methods, benefiting from multiple input views. We also validate our method on indoor full room scenes. With 16 images capturing a room, we reconstruct the full room reasonably, while pixelSplat and MVSplat still suffer from the redundancy of Gaussians, due to the fact that the large-scale scenes require more input views. We report the quantitative results of each scene in the following table for detailed comparison.\n\nTable 1: Quantitative comparison of different methods on large-scale scenes.\n\n| Sence ID | pixelSplat | MVSplat | Ours |\n|:--------:|:----------:|:-------:|:----:|\n|  Re10k: fea544b472e9abd1 |    12.66    |   12.88   |  18.88 |\n|  Re10k: 21e794f71e31becb |    15.04    |   14.76   |  20.62 |\n|  Re10k: de45926738229f67 |    17.13    |   20.21   |  25.07 |\n|  ACID: d4453ce709bd53e1 |     11.64    |   13.14   |  20.44 |\n|  ACID: ee09e048af8deba6 |    17.77    |   16.86   |  26.55 |\n|  ACID: 3fcb7b6b398b4064 |    22.75    |   24.55   |  32.03 |\n\n**3. Release of the source code**\n\nWe will release our paper on arXiv and the source code on github if our paper is accepted.\n\n**4. Questions about the different number of input views in Table 1 and Figure 4**\n\nFor results of different number of input views, they are obtained with a single model. In our opinion, it is convenient and flexible for users to use one model to process different number of input views. The results also demonstrate that our model has the capability to reconstruct scenes with different number of input views.\n\n**5. Efficiency analysis**\n\nWe agree with you that the analysis of training and inference latency of our proposed method is important. We discuss this topic in our general rebuttal. We will add this discussion on training and inference latency in Section 4.2 (line 190) as you suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3rlCPc9Hg5", "reviewer": "Reviewer_zafq", "summary": "This paper introduces the Gaussian Graph Network (GGN), a novel approach for generalizable 3D Gaussian Splatting (3GDS) reconstruction. The authors identify a problem with previous generalizable 3DGS work: they regress pixel-aligned Gaussians and combine Gaussians from different views directly, resulting in an excessive number of Gaussians when the model processes multi-view inputs. To address this issue, they propose using a graph network that identifies the relationships between the Gaussians generated from different views and merges them if they are similar in positions and features. Thus, GGN can reduce the number of Gaussians for better efficiency and reconstruction quality. Experiments show that GGN outperforms baselines in both efficiency and quality and scales well when the input views increase.", "questions": "1. How is the inference efficiency compared to the baselines?\n2. How is the inference efficiency when the number of input views increases?", "limitations": "The limitations are discussed in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The idea of merging redundant Gaussians in generalizable 3DGS is novel and practical. Identifying the relationships between Gaussians using a graph network to merge them is also a novel idea.\n2. The results are satisfactory. Compared to the baselines, GGN achieves better reconstruction results with fewer Gaussians and scales well when the input views increase.\n3. The paper is well-written and the experiments are complete.", "weaknesses": "1. The inference efficiency of GGN is not shown in the experiments. Since the number of Gaussians is quite large and increases with more input views, the efficiency of GGN is a concern. While the authors provide the results of rendering time and the number of merged Gaussians, these data do not reflect the inference efficiency of the model.\n\nOverall, this paper introduces a novel and practical idea for generalizable 3DGS reconstruction. I think this paper pushes the boundaries of generalizable 3D reconstruction and should be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We agree with you that the analysis of training and inference latency of our proposed method is important. We discuss this topic in our general rebuttal. We will add this discussion on training and inference latency in Section 4.2 (line 190) as you suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vt5EHdkA9X", "reviewer": "Reviewer_Xdv1", "summary": "This paper presents Gaussian Graphs to construct the relations of different Gaussian groups, introduces a Gaussian Graph Network to process Gaussian Graphs.", "questions": "1. As far as I can see, the main work of this paper includes the construction of Gaussian Graphs and Gaussian Graph Network. But they are very similar to the common GNN and workflow of GNN. What's the difference between this paper and other works that combine GNN and GS have been proposed, such as SAGS and Hyper-3DG?\n2. Could this work be compared with NeRF-based generalizable architectures in the experiments?", "limitations": "I think this paper is of low innovation, as methods have combined GS and GNN. The construction of Gaussian Graphs and Gaussian Graph Network are very common.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Experimental results are sufficient and convincing. This work is easy to follow.", "weaknesses": "1. As far as I can see, the main work of this paper includes the construction of Gaussian Graphs and Gaussian Graph Network. But they are very similar to the common GNN and workflow of GNN. The main innovation of this paper may be the combination of GNN and GS. However, many works that combine GNN and GS have been proposed, such as SAGS and Hyper-3DG. So I think this paper is of low innovation.\n2. The authors provide only a few experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. Below we address specific questions.\n\n**1. The difference between this paper and other works that combine GNN and GS**\n\nThank you for pointing out these relevant works SAGS [1] and Hyper-3DG [2]. We would like to highlight our key differences with these works.\n\nOne of the key differences is the construction of Gaussian graphs. SAGS sets each point as a node and constructs graphs for KNN points, while Hyper-3DG first obtains patch features by aggregating gaussian features within each patch, and then sets each patch as a node with its feature. However, SAGS is still an optimization-based method, where the Gaussian-level graph is less efficient due to the large number of Gaussian points, while the patch-level graph in Hyper-3DG fails to support message passing to exploit Gaussian-level relations.  \n\nIn our work, we propose a hierarchical method to construct Gaussian Graphs - we formulate each view as a node but also fully preserve the Guassian-level structures in each node. We fully exploit the important view-related information, i.e. (1) group of pixel-aligned Gaussians belonging to the same view and (2) projection relations between different views. This information enables us to capture accurate and sparse cross-view Gaussian-level relation in an efficient manner with the edge matrix $E^{i\\rightarrow j}$.\n\nBased on our carefully-designed hierarchical structure, we specifically design linear layers for feature learning by extending the scalar weight of an graph edge to a matrix. Furthermore, we propose a new pooling strategy to fuse Gaussians based on their spatial positions to avoid the redundancy of Gaussians in pixelSplat and MVSplat, while SAGS and Hyper-3DG do not introduce pooling layers in their graph networks.\n\n[1] Ververas, E., Potamias, R. A., Song, J., Deng, J., & Zafeiriou, S. (2024). SAGS: Structure-Aware 3D Gaussian Splatting. arXiv preprint arXiv:2404.19149.\n\n[2] Di, D., Yang, J., Luo, C., Xue, Z., Chen, W., Yang, X., & Gao, Y. (2024). Hyper-3DG: Text-to-3D Gaussian Generation via Hypergraph. arXiv preprint arXiv:2403.09236. \n\n**2. Comparison with NeRF-based generalizable methods**\n\nWe reported the comparison with NeRF-based methods on 2-view settings in Table 2 of our paper. We also conduct further experiments with different number of input views, as shown in the following table. Experimental results show that our method have better rendering quality and inference speed, which is benefit from our efficient and generalizable Gaussian representation learning.\n\n\nTable 1: Comparison of NeRF-based methods on RealEstate10K with different number of input views.\n\n|           | 4 view |                    | 8 view |                 | 16 view |                |\n|-----------|--------|--------------------|--------|-----------------|---------|----------------|\n|           | PSNR   | Inference Time     | PSNR   | Inference Time  | PSNR    | Inference Time |\n| pixelNeRF [3] | 21.82  | 5.38           | 21.84  | 5.77            | 21.85   | 6.48           |\n| MuRF [4]      | 24.30  | 0.36           | 24.78  | 0.55            | 25.56   | 1.79           |\n| Ours          | 24.76  | 0.14           | 25.15  | 0.44            | 26.18   | 1.40           |\n\n[3] Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2021). pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4578-4587).\n\n[4] Xu, H., Chen, A., Chen, Y., Sakaridis, C., Zhang, Y., Pollefeys, M., ... & Yu, F. (2024). Murf: Multi-baseline radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20041-20050).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful feedback. We tried to address all the questions for each reviewer in the rebuttal session below. 

Here we discuss about the **training and inference efficiency** as  mentioned by Reviewer sL12, LnYN and zafq. As the point-aligned gaussian points are quite a large number, if we set each gaussian point as a node in the graph, we do face the problem of inefficiency. In this paper, we formulate each view as a node but also fully preserve the Gaussian-level structures in each node. We exploit the important view-related information, i.e. (1) group of pixel-aligned Gaussians belonging to the same view and (2) projection relations between different views. This information enables us to capture accurate and sparse cross-view Gaussian-level relations in an efficient manner. Based on this, our specifically designed layers for the Gaussian Graph are operated on such Gaussian groups for information fusion. 

We further conduct experiments to analysis the training and inference latency on RealEstate10K. All the experiments are conducted on a single NVIDIA A6000 GPU. Since the inference of pixelSplat with 16 input views cannot be done in parallel on a 48G A6000, we split them into batches to get Gaussians. 

As shown in Table 1 and Table 2, our method uses less training and inference time compared with pixelSplat, benefiting from faster encoding and rendering speed. Due to the additional construction of the Gaussian Graph and application of Gaussian Graph Network, our method uses a little more training and inference time than MVSplat. We also report time consumption of each part of our methods in Table 3, where the Gaussian Graph Network dominates in time consumption and the rendering time increases slowly with more input views. As illustrated in Table 4, the inference time of three methods grows at a quadratic rate with the increase of image resolution due to pixel-aligned Gaussians. 

For memory analysis, as shown in Table 5, our method has 80% off on training memory cost compared with pixelSplat. The comparison of inference memory cost between our method and pixelSplat draws to the similar conclusion. The structure of Gaussian Graph leads to a little increase of memory cost compared to MVSplat, which can be almost ignored as the number of views increases. 

Table 1: Training time (h) of different methods.

|    Methods    | pixelSplat | MVSplat | Ours |
|:-------------:|:----------:|:-------:|:----:|
| Training time |     183    |    78   |  111 |


Table 2: Inference time (ms) of different methods with different number of input views.

|          | pixelSplat | MVSplat |   Ours  |
|:--------:|:----------:|:-------:|:-------:|
|  2 views |    137.3   |   60.6  |  75.6   |
|  4 views |    298.8   |  126.4  |  148.1  |
|  8 views |    846.5   |  363.2  |  388.8  |
| 16 views |   2938.9   |  1239.8 | 1267.5  |

Table 3: Inference time (ms) of different parts in our method.

|          | Image Encoder |   GGN   | Parameter Predictor | Rendering |
|:--------:|:-------------:|:-------:|:-------------------:|:---------:|
|  2 views |      8.37     |  28.21  |        34.51        |    4.47   |
|  4 views |     10.01     |  88.89  |        44.56        |    4.64   |
|  8 views |     14.09     |  311.24 |        58.75        |    4.75   |
| 16 views |     21.15     | 1165.65 |        75.88        |    4.84   |

Table 4: Inference time (ms) of different methods with different view resolutions.

|          | pixelSplat | MVSplat |   Ours  |
|:--------:|:----------:|:-------:|:-------:|
|  256$\times$256 |    137.3   |   60.6  |  75.6   |
|  512$\times$512 |    574.2   |   235.3 |  319.5  |
|  1024$\times$1024 |  2445.4  |   943.3 |  1329.9 |

Table 5: Training memory cost (GB) of different methods.

|        Methods       | pixelSplat | MVSplat | Ours |
|:--------------------:|:----------:|:-------:|:----:|
| Training Memory Cost |    37.5    |   5.9   |  7.7 |

Table 6: Inference memory cost (GB) of different methods with different number of input views.

|          | pixelSplat | MVSplat | Ours |
|:--------:|:----------:|:-------:|:----:|
|  2 views |     6.1    |   1.6   |  3.2 |
|  4 views |    11.8    |   3.2   |  4.8 |
|  8 views |    29.0    |   8.0   |  8.6 |
| 16 views |    73.7    |   20.0  | 21.4 |

Please kindly let us know if you have any follow-up questions or areas needing further clarification. Your insights are valuable to us, and we are ready to provide any additional information that could be helpful.

Meta-review: This paper is a clear paper. It receives 1x accept, 1x weak accept and 1x borderline accept. The reviewers agreed that the paper proposes a novel and practical idea of merging redundant Gaussians in generalizable 3DGS. The proposed framework is a good choice for fusing pixel-aligned gaussian points from frameworks like PixelSplat or MVSplat. Grouping and fusing these points in 3D space with a graph-based strategy sounds reasonable and effective in solving the artifact issue caused by PixelSplat. The experimental results are sufficient and convincing. This work is easy to follow. The weaknesses are mostly clarifications on the efficiency and scalability, and experimental settings are fully addressed by the authors during rebuttal and the discussion phase. The meta-reviewer follows the suggestions of the reviewers to accept the paper.
Decision: Accept (poster)