Gaussian Graph Network: Learning Efficient and Generalizable Gaussian Representations from Multi-view Images

Authors: shengjun  zhang
xin  fei
fangfu  liu
haixu  song
yueqi  duan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 03:45:43

Abstract: 3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis performance. While conventional methods require per-scene optimization, more recently several feed-forward methods have been proposed to generate pixel-aligned Gaussian representations with a learnable network, which are generalizable to different scenes. However, these methods simply combine pixel-aligned Gaussians from multiple views as scene representations, thereby leading to artifacts and extra memory cost without fully capturing the relations of Gaussians from different images. In this paper, we propose Gaussian Graph Network (GGN) to generate efficient and generalizable Gaussian representations. Specifically, we construct Gaussian Graphs to model the relations of Gaussian groups from different views. To support message passing at Gaussian level, we reformulate the basic graph operations over Gaussian representations, enabling each Gaussian to benefit from its connected Gaussian groups with Gaussian feature fusion. Furthermore, we design a Gaussian pooling layer to aggregate various Gaussian groups for efficient representations. We conduct experiments on the large-scale RealEstate10K and ACID datasets to demonstrate the efficiency and generalization of our method. Compared to the state-of-the-art methods, our model uses fewer Gaussians and achieves better image quality with higher rendering speed.

Main paper: 
 introduction: 
 Novel view synthesis is a fundamental problem in computer vision due to its widespread applications, such as virtual reality, augmented reality, robotics and so on. Remarkable progress has been made using neural implicit representations [33, 42, 43] , but these methods suffer from expensive time consumption in training and rendering [28, 13, 1, 40, 65, 11, 17, 34] . Recently, 3D Gaussian Splatting (3DGS) [21] has drawn increasing attention for explicit Gaussian representations and real-time rendering performance. Benefiting from rasterization-based rendering, 3DGS avoids dense points querying in scene space, so that it can maintain high efficiency and quality.Since 3DGS relies on per-subject [21] or per-frame [31] parameter optimization, several generalizable methods [68, 6, 4, 45] are proposed to directly regress Gaussian parameters with feed-forward networks. Typically, these methods [6, 4] generate pixel-aligned Gaussians with U-Net architectures, epipolar transformers or cost volume representations for depth estimation and parameter predictions, and directly combine Gaussian groups obtained from different views as scene representations. However, such combination of Gaussians leads to superfluous representations, where the overlapped regions are covered by similar Gaussians predicted separately from multiple images. While a simple solution is to delete redundant Gaussians, it ignores the connection among Gaussian groups. As We report PSNR as well as the number of Gaussians for pixelSplat [4] , MVSplat [6] and GNN under different input settings.illustrated in Figure 1a , pixelSplat [4] and MVSplat [6] suffer from artifacts with several times as many Gaussians as ours, while the deletion of similar Gaussians hurts rendering quality.To tackle the challenge, we propose Gaussian Graphs to model the relations of Gaussian groups from multiple views. Based on this structure, we present Gaussian Graph Network (GGN), extending conventional graph operations to Gaussian domain, so that Gaussians from different views are not independent but can learn from their neighbor groups. Precisely, we reformulate the scalar weight of an edge to a weight matrix to depict the interactions between two Gaussian groups, and introduce a Gaussian pooling strategy to aggregate Gaussians. Under this definition, previous methods [4, 6] can be considered as a degraded case of Gaussian Graphs without edges. As shown in Figure 1b , our GGN allows message passing and aggregation across Gaussians for efficiency representations.We conduct extensive experiments on both indoor and outdoor datasets, including RealEstate10K [69] and ACID [26] . While the performance of previous methods declines as the number of input views increases, our method can benefit from more input views. As shown in Figure 1c , our model outperforms previous methods under different input settings with higher rendering quality and fewer Gaussian representations. Our main contributions can be summarized as follows:• We propose Gaussian Graphs to construct the relations of different Gaussian groups, where each node is a set of pixel-aligned Gaussians from an input view. • We introduce Gaussian Graph Network to process Gaussian Graphs by extending the graph operations to Gaussian domain, bridging the interaction and aggregation across Gaussian groups. • Experimental results illustrate that our method can generate efficient and generalizable Gaussian representations. Our model requires fewer Gaussians and achieves better rendering quality.2 Related Works 
neural implicit representations: 
 Early researches focus on capturing dense views to reconstruct scenes, while neural implicit representations have significantly advanced neural processing for 3D data and multi-view images, leading to high reconstruction and rendering quality [32, 38, 64, 43] . In particular, Neural Radiance Fields (NeRF) [33] has garnered considerable attention with a fully connected neural network to represent complex 3D scenes. Subsequently, following works have emerged to address NeRF's limitations and enhance its performance. Some studies aim to solve the long-standing problem of novel view synthesis by improving the speed and efficiency of training and inference [17, 13, 34, 40, 28, 65, 11] . Other research focuses on modeling complex geometry and view-dependent effects to reconstruct dynamic scenes [8, 22, 39, 48, 55, 25, 52, 47, 23, 51] . Additionally, some studies [50, 46, 61, 56] have worked on reconstructing large urban scenes to avoid blurred renderings without fine details, which is a challenge for NeRF-based methods due to their limited model capacity. Furthermore, other works [36, 49, 54, 59] apply NeRF to novel view synthesis with sparse input views by incorporating additional training regularizations, such as depth, correspondence, and diffusion models.More recently, 3D Gaussian Splatting (3DGS) [21] has drawn significant attention in the field of computer graphics, especially in the realm of novel view synthesis. Different from the expensive volume sampling strategy in NeRF, 3DGS utilizes a much more efficient rasterization-based splatting approach to render novel views from a set of 3D Gaussian primitives. However, 3DGS may suffer from artifacts, which occur during the splatting process. Thus, several works [62, 12, 19, 24] have been proposed to enhance the quality and realness of rendered novel views. Since 3DGS requires millions of parameters to represent a single scene, resulting in significant storage demands, some researches [30, 35, 14, 10, 20] focus on reducing the memory usage to ensure real-time rendering while maintaining the rendering quality. Other works [70, 58] are proposed to reduce the amount of required images to reconstruct scene. Besides, to extend the capability of 3D Gaussians from representing static scenes to 4D scenarios, several works [53, 9, 63, 57] have been proposed to incorporate faithful dynamics which are aligned with real-world physics.Optimization-based methods train a separate model for each individual scene and require dozens of images to synthesis high-quality novel views. In contrast, feed-forward models learn powerful priors from large-scale datasets, so that 3D reconstruction and view synthesis can be achieved via a single feed-forward inference. A pioneering work, pixelNeRF [66] , proposes a feature-based method to employ the encoded features to render novel views. MVSNeRF [5] combines plane-swept cost volumes which is widely used in multi-view stereo with physically based volume rendering to further improve the quality of neural radiance field reconstruction. Subsequently, many researches [2, 3, 15, 18, 37, 41, 27] have developed novel view synthesis methods with generalization and decomposition abilities. Several generalizable 3D Gaussian Splatting methods have also been proposed to leverage sparse view images to reconstruct novel scenes. While Splatter Image [45] and GPS-Gaussian [68] regress pixel-aligned Gaussian parameters to reconstruct single objects or humans instead of complex scenes, pixelSplat [4] takes sparse views as input to predict Gaussian parameters by leveraging epipolar geometry and depth estimation. MVSplat [6] constructs a cost volume structure to directly predict depth from cross-view features, further improving the geometric quality.However, these methods follow the paradigm of regressing pixel-aligned Gaussians and combine Gaussians from different views directly, resulting in an excessive number of Gaussians when the model processes multi-view inputs. In comparison, we construct Gaussian Graphs to model the relations of Gaussian groups. Furthermore, we introduce Gaussian Graph Network with specifically designed graph operations to ensure the interaction and aggregation across Gaussian groups. In this manner, we can obtain efficient and generalizable Gaussian representations from multi-view images.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of our work in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 The overall framework is illustrated in Figure 2 . After predicting positions and features of Gaussian groups from input views with extracted feature maps, we build a Gaussian Graph to model the relations. Then, we process the graph with Gaussian Graph Network via our designed graph operations on Gaussian domain for information exchange and aggregation across Gaussian groups. We leverage the fused Gaussians features to predict other Gaussian parameters, including opacity α, covariance matrix Σ and colors c. Given multiple input images, we extract image features and predict the means and features of pixel-aligned Gaussians. Then, we construct a Gaussian Graph to model the relations between different Gaussian nodes. We introduce Gaussian Graph Network to process our Gaussian Graph. The parameter predictor generates Gaussians parameters from the output Gaussian features.Given N input images I = {I i } ∈ R N ×H×W ×3 and their corresponding camera parameters C = {c i }, we follow the instructions of pixelSplat [4] and MVSplat [6] to extract image features:EQUATIONwhere Φ image is a 2D backbone. We predict the means and features of pixel-aligned Gaussians:EQUATIONwhere Φ depth and Φ f eat stand for neural networks to predict depth maps and Gaussian features, and ψ unproj is the unprojection operation.We build a Gaussian Graph G with nodesV = {v i } = {(µ i , f i )} 1≤i≤N .To model the relations between nodes, we define its adjacency matrix A = [a ij ] N ×N as:EQUATIONwhere ψ overlap (v i , v j ) computes the overlap ratio between view i and view j. To limit further computational complexity, we prune the graph by preserving edges with top n weights and ignore other possible edges. The degree matrix D = [d ij ] N ×N satisfies d ii = j a ij . Thus, the scaled adjacency can be formulated as:EQUATION)Linear Layers. Assuming that a conventional graph has N nodes with features {g i } ∈ R N ×C and the scaled adjacency matrix A = [ã ij ] N ×N , the basic linear operation can be formulated as:EQUATIONwhere W ∈ R C×D is a learnable weight. Different from the vector nodes g i in conventional graphs, each node of our Gaussian Graph contains a set of pixel-aligned Gaussians v i = {µ i , f i }. Therefore, we extend the scalar weight a sk of an edge to a matrixE s→k = [e s→k ij] HW ×HW , which depicts the detailed relations at Gaussian-level between v s and v k . For the sake of simplicity, we defineEQUATIONAlgorithm 1 Gaussian Graph NetworkInput: Multi-view images I = {I i }, camera parameters C = {c i }, the number of graph layers h Output: Gaussian parameters (µ, Σ, α, c) F ← Φ image (I, C) µ i ← ψ unproj (Φ depth (F i ), c i ) , f i ← Φ f eat (F i ) i = 1, 2, • • • , N Ã ← D -1 A for k ← 1 to h do f i ← σ ãij E j→i f j W i = 1, 2, • • • , N end for (µ, f ) ← l s=1 ϕ pooling (G s ) R, S, α, c ← ψ R (f ), ψ S (f ), ψ α (f ), ψ c (f ) Σ ← RSS ⊤ R ⊤where µ s,i is the center of the i-th Gaussian in Gaussian group v s , and ψ proj is the projection function which returns coordinates of the occupied pixel. In this manner, the linear operation on a Gaussian Graph can be defined as:EQUATION)Pooling layers. If we have a series of connected Gaussian Graphs {G s } 1≤s≤l , where G s ⊆ G, then we operate the specific pooling operation on each G s and combine them together:EQUATIONIf G s only contains one node v s , ϕ pooling (G s ) = v s . Otherwise, we start with a random selected nodev s i = (µ s i , f s i ) ∈ G s .Since G s is a connected graph, we can randomly pick up its neighbor v s j with corresponding camera parameters c s j . We define a function ψ similarity :EQUATIONwhere v s j,m is the m-th Gaussian in node v s j . We define the merge function ψ merge :EQUATIONThen, we merge two nodes together to get a new nodeEQUATIONIn this manner, we aggregate a connected graph G s step by step to one node. Specifically, previous methods can be considered as a degraded Gaussian Graph without edges:EQUATION)where ϕ pooling degenerates to simple combination of nodes.Parameter prediction. After aggregating the Gaussian Graph (µ, f ) = ψ pooling (G), we predict the rotation matrix R, scale matrix S, opacity α and color c:EQUATIONwhere ϕ R , ϕ S , ϕ α and ϕ c are prediction heads. The covariance matrix can be formulated as:EQUATION)Our Gaussian Graph Network is illustrated in Algorithm 1, where σ stands for non-linear operations, such as ReLU and GeLU. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 We validate our proposed Gaussian Graph Network for novel view synthesis. In Section 4.1, we compare various methods on RealEstate10K [69] and ACID [26] under different input settings. In Section 4.2, we analyze the efficiency of our representations. In Section 4.3, we further validate the generalization of our Gaussian representations under cross dataset evaluation. In Section 4.4, we ablate our designed operations of Gaussian Graph Network.Datasets. We conduct experiments on two large-scale datasets, including RealEstate10K [69] and ACID [26] . RealEstate10K dataset comprises video frames of real estate scenes, which are split into 67,477 scenes for training and 7,289 scenes for testing. ACID dataset consists of natural landscape scenes, with 11,075 training scenes and 1,972 testing scenes. For two-view inputs, our model is trained with two views as input, and subsequently tested on three target novel views for each scene. For multi-view inputs, we construct challenging subsets for RealEstate10K and ACID across a broader range of each scenario to evaluate model performance. We select 4, 8 and 16 views as reference views, and evaluate pixelSplat [4] , MVSplat [6] and ours on the same target novel views.Implementation details. Our model is trained with two input views for each scene on a single A6000 GPU for 300,000 iterations, utilizing the Adam optimizer. Following the instruction of previous methods [4, 6] , all experiments are conducted on 256 × 256 resolutions for fair comparison. The image backbone is initialized by the feature extractor and cost volume representations in MVSplat [6] .The number of graph layer is set to 2. The training loss is a linear combination of MSE and LPIPS [67] losses, with loss weights of 1 and 0.05.Results. As shown in Table 1 and Table 2 , our Gaussian Graph benefits from increasing input views, whereas both pixelSplat [4] and MVSplat [6] exhibit declines in performance. This distinction highlights the superior efficiency of our approach, which achieves more effective 3D representation Figure 3 : Visualization results on RealEstate10K [69] and ACID [26] benchmarks. We evaluate all models with 4, 8, 16 views as input and subsequently test on three target novel views.with significantly fewer Gaussians compared to pixel-wise methodologies. For 4 view inputs, our method outperforms MVSplat [6] by about 4dB on PSNR with more than 2× fewer Gaussians.For more input views, the number of Gaussians in previous methods increases linearly, while our GGN only requires a small increase. Considering that previous methods suffer from the redundancy of Gaussians, we adopt these models with multi-view inputs and preserve Gaussians from two input views for rendering. Under this circumstance, pixelSplat [4] and MVSplat [6] achieve better performance than before, because the redundancy of Gaussians is alleviated to some degree. However, the lack of interaction at Gaussian level limits the quality of previous methods. In contrast, our Gaussian Graph can significantly enhance performance by leveraging additional information from extra views. Visualization results in Figure 3 also indicate that pixelSplat [4] and MVSplat [6] tend to suffer from artifacts due to duplicated and unnecessary Gaussians in local areas, which increasingly affects image quality as more input views are added. Figure 4 : Efficiency analysis. We report the number of Gaussians (M), rendering frames per second (FPS) and reconstruction PSNR of pixelSplat [4] , MVSplat [6] and our GGN.In addition to delivering superior rendering quality, our Gaussian Graph network also enhances efficiency in 3D representations. As illustrated in Figure 4 , when using 24 input images, our model outperforms MVSplat by 8.6dB on PSNR with approximately one-tenth the number of 3D Gaussians and more than three times faster rendering speed. Additionally, we compare the average inference time of our model with pixel-wise methods in Table 3 . Our GGN is able to efficiently remove duplicated Gaussians and enhance Gaussian-level interactions, which allows it to achieve superior reconstruction performance with comparable inference speed to MVSplat.To further demonstrate the generalization of Gaussian Graph Network, we conduct cross-dataset experiments. Specifically, all models are trained on RealEstate10K [69] or ACID [26] datasets, and are tested on the other dataset without any fine-tuning. Our method constructs the Gaussian Graph according to the relations of input views. As shown in Table 4 , our GGN consistently outperforms pixelSplat [4] and MVSplat [6] on both benchmarks.Figure 5 : Visualization of model performance for cross-dataset generalization on RealEstate10K [69] and ACID [26] benchmarks. To investigate the architecture design of our Gaussian Graph Network, we conduct ablation studies on RealEstate10K [69] benchmark. We first introduce a vanilla model without Gaussian Graph Network. Then, we simply adopt Gaussian Graph linear layer to model relations of Gaussian groups from multiple views. Furthermore, we simply introduce Gaussian Graph pooling layer to aggregate Gaussian groups to obtain efficient representations. Finally, we add the full Gaussian Graph Network model to both remove duplicated Gaussians and enhance Gaussian-level interactions.Gaussian Graph linear layer. The Gaussian Graph linear layer serves as a pivotal feature fusion block, enabling Gaussian Graph nodes to learn from their neighbor nodes. The absence of linear layers leads to a performance drop of 0.43 dB on PSNR.Gaussian Graph pooling layer. The Gaussian Graph pooling layer is important to avoid duplicate and unnecessary Gaussians, which is essential for preventing artifacts and floaters in reconstructions and speeding up the view rendering process. As shown in Table 5 , the introduction of Gaussian Graph pooling layer improves the rendering quality by 4.9dB on PSNR and reduces the number of Gaussians to nearly one-fourth. 
conclusion and discussion: 
 In this paper, we propose Gaussian Graph to model the relations of Gaussians from multiple views.To process this graph, we introduce Gaussian Graph Network by extending the conventional graph operations to Gaussian representations. Our designed layers bridge the interaction and aggregation between Gaussian groups to obtain efficient and generalizable Gaussian representations. Experiments demonstrate that our method achieves better rendering quality with fewer Gaussians and higher FPS.Limitations and future works. Although GGN produces compelling results and outperforms prior works, it has limitations. Because we predict pixel-aligned Gaussians for each view, the representations are sensitive to the resolution of input images. For high resolution inputs, e.g. 1024 × 1024, we generate over 1 million Gaussians for each view, which will significantly increase the inference and rendering time. GGN does not address generative modeling of unseen parts of the scene, where generative methods, such as diffusion models can be introduced to the framework for extensive generalization. Furthermore, GGN focuses on the color field, which does not fully capture the geometry structures of scenes. Thus, a few directions would be focused in future works.Visualization results draw the same conclusion that our method generates efficient and generalizable Gaussian representations. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We report information on the computer resources in Section 4 and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss potential societal impacts in appendix D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: Our paper does not use existing asserts.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Our method focuses on generalizable novel view synthesis which can be used for application ranging from virtual reality to robotics. However, it can also have potential negative societal impact. Our method relies on large amounts of data and with large computational resources, which potentially has a negative impact on global climate change. What's more, accurate rendering of a scene may raise privacy concerns that need to be addressed carefully. Figure 6 : Additional visualization results on RealEstate10K [69] and ACID [26] benchmarks. We evaluate all models with 4, 8, 16 views as input and subsequently test on three target novel views. Justification: We provide claims made in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We have reviewed the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper is a clear paper. It receives 1x accept, 1x weak accept and 1x borderline accept. The reviewers agreed that the paper proposes a novel and practical idea of merging redundant Gaussians in generalizable 3DGS. The proposed framework is a good choice for fusing pixel-aligned gaussian points from frameworks like PixelSplat or MVSplat. Grouping and fusing these points in 3D space with a graph-based strategy sounds reasonable and effective in solving the artifact issue caused by PixelSplat. The experimental results are sufficient and convincing. This work is easy to follow. The weaknesses are mostly clarifications on the efficiency and scalability, and experimental settings are fully addressed by the authors during rebuttal and the discussion phase. The meta-reviewer follows the suggestions of the reviewers to accept the paper.

Reviews: review: paper_summary: paper_summary: The paper presents an incremental design built upon existing generalizable GS reconstruction frameworks (e.g., PixelSplat, MVSplat) to fusion overlapped pixel-aligned gaussian points from multiple images with a graph-based operator and pooling. This additional design presents a faster and better rendering effect than PixelSplat and MVSplat.
strength_weakness: strength_weakness: ***[Efficiency and Scalability]*** 

Although the paper presents a better rendering speed (a natural outcome of reducing gaussian points), it lacks a discussion regarding learning efficiency: the training latency of the proposed design. We know the point-aligned gaussian points are actually quite a large number, especially when increasing the number and resolution of images. However, classical graph-based methods are usually not efficient in handling a large number of points. Further, the limitation on training efficiency might further restrict the stability of the proposed design. Consequently, I hope the author can consider adding a discussion and benchmarking on a breakdown of training and inference latency of the additional graph-based network. It is fine if the current proposed design is not so efficient, but it is definitely worth a discussion.

On the other hand, the proposed graph network is fundamentally seeking a way to process and fuse points in 3D space. The point cloud processing community already has years of accumulation on handling this kind of data efficiently and effectively. One suggestion is to further enhance performance with the latest efforts in point cloud processing, such as Point Transformer V3.
questions: questions: Already discussed in weaknesses regard to the training efficiency and scalability.
limitations: limitations: The authors also mentioned the limitation in model efficiency, especially when increasing the number and resolution of images. Yet this part is actually more important and worth some number to measure this issue. Detailed discussion in weaknesses.
paper_summary: paper_summary: This paper proposes a Graph Neural Network architecture to model the relations between multi-view 3D Gaussians predicted by generalizable 3DGS methods. The proposed method works in particularly better for large number (e.g., 4, 8, 16 etc) of input images, compared with previous methods like pixelSplat and MVSplat that simply combine per-view pixel-aligned Gaussians. The proposed method effectively reduces the redundancy of 3D Gaussians when more input images are given. Experiments are conducted on standard RealEstate10K and ACID datasets and the proposed method performs significantly better than pervious methods for large number of input views.
strength_weakness: strength_weakness: A few implementation details are not completely clear. For example, how are the input views selected for different number of input views? Are they sampled to be more densely within some predefined frame range or are they sampled to cover larger regions of the scene? This also leads to another question if the proposed method could work on large-scale scenes since the model could handle many frames now? For instance, if the inputs are 10-20 images capturing a room, could the full room be reconstructed reasonably? I am also wondering if the authors plan to release the source code, which I think would benefit the reproducibility for the research community.

For results of different number of input views in Table 1 and Figure 4, are they obtained with a single model, or the author trained a specific model for each specific number of input views? 

For efficiency analysis, this paper reported the FPS metric. However, I assume the FPS is measured only for the rendering part right? How about the network inference time and memory consumption, especially for more input images? Since the proposed method is a feed-forward model, where the network forward time should also be considered for benchmarking efficiency.
questions: questions: Please refer to the weaknesses.
limitations: limitations: Yes, the limitations are discussed.
paper_summary: paper_summary: This paper introduces the Gaussian Graph Network (GGN), a novel approach for generalizable 3D Gaussian Splatting (3GDS) reconstruction. The authors identify a problem with previous generalizable 3DGS work: they regress pixel-aligned Gaussians and combine Gaussians from different views directly, resulting in an excessive number of Gaussians when the model processes multi-view inputs. To address this issue, they propose using a graph network that identifies the relationships between the Gaussians generated from different views and merges them if they are similar in positions and features. Thus, GGN can reduce the number of Gaussians for better efficiency and reconstruction quality. Experiments show that GGN outperforms baselines in both efficiency and quality and scales well when the input views increase.
strength_weakness: strength_weakness: 1. The inference efficiency of GGN is not shown in the experiments. Since the number of Gaussians is quite large and increases with more input views, the efficiency of GGN is a concern. While the authors provide the results of rendering time and the number of merged Gaussians, these data do not reflect the inference efficiency of the model.

Overall, this paper introduces a novel and practical idea for generalizable 3DGS reconstruction. I think this paper pushes the boundaries of generalizable 3D reconstruction and should be accepted.
questions: questions: 1. How is the inference efficiency compared to the baselines?
2. How is the inference efficiency when the number of input views increases?
limitations: limitations: The limitations are discussed in the paper.
paper_summary: paper_summary: This paper presents Gaussian Graphs to construct the relations of different Gaussian groups, introduces a Gaussian Graph Network to process Gaussian Graphs.
strength_weakness: strength_weakness: 1. As far as I can see, the main work of this paper includes the construction of Gaussian Graphs and Gaussian Graph Network. But they are very similar to the common GNN and workflow of GNN. The main innovation of this paper may be the combination of GNN and GS. However, many works that combine GNN and GS have been proposed, such as SAGS and Hyper-3DG. So I think this paper is of low innovation.
2. The authors provide only a few experimental results.
questions: questions: 1. As far as I can see, the main work of this paper includes the construction of Gaussian Graphs and Gaussian Graph Network. But they are very similar to the common GNN and workflow of GNN. What's the difference between this paper and other works that combine GNN and GS have been proposed, such as SAGS and Hyper-3DG?
2. Could this work be compared with NeRF-based generalizable architectures in the experiments?
limitations: limitations: I think this paper is of low innovation, as methods have combined GS and GNN. The construction of Gaussian Graphs and Gaussian Graph Network are very common.

Comments: title: Thanks for your valuable feedback
Inference time of each component in the image encoder
Correction of Table 3
Time and memory cost of the image encoder
Efficiency comparison with Point Transformer
comment: Thank you, Reviewer sL12, for your insightful comments. I completely agree with your points and would like to echo your thoughts. Indeed, I also feel that combining point cloud processing with Gaussian splatting has strong potential, and I am excited to see what our community will develop in this direction :)

Best Regards,

Reviewer LnYN
Thanks for the rebuttal. My concerns are well addressed. Thus, I keep my original rate to accept this paper. 

BTW: I am also considering the possibility of combining point cloud processing with Gaussian splatting. We have spent many years exploring how to handle unstructured data effectively and efficiently, and it's encouraging to see that authors and reviewer LnYN are also interested in this potential.
Thanks for your valuable suggestions. We will clarify the details of view selection and evaluation for different number of input views in Section 4.1. We will prepare a website page, which shows multiple video results, and we will release it with our final paper. We will also add the time and memory consumption in the global response to Section 4.2.
I have read all the reviews and the rebuttal. First, I would like to thank the authors for taking time in preparing the rebuttal and providing additional clarifications in the discussion. Second, as the questions or concerns I had in my earlier review have all been answered or addressed, I keep my initial rating (Weak Accept). However, I would suggest to add the following missing information in the final version:

- Details of view selection and evaluation for different number of input views
- Video results of more input views, which would make it easier to perceive the global consistency of the reconstruction. Although a few rendered images are provided in the rebuttal, it's less clear (compared to videos) how the model works for more input views.
- Time and memory consumption of different number of views, as shown in the global response.
Thanks the authors for the feedback. Everything is clear to me now and it's great to identify the bottleneck in the full pipeline.
Thanks for your comments. We are sorry to confuse you due to the unclear markdown table. Below we address specific questions.

**Q:** Is the U-Net used for depth estimation from the cost volume considered in the inference time?

**A:** Yes. In the previous table, the time taken by the U-Net used for depth estimation is added to the inference time. Both the cost volume refinement and the depth refinement parts employ 2D U-Net. We report their inference time in our new table separately.

**Q:** Why does the cost volume take much more inference time than expected?

**A:** The cost volume construction contains two parts: (a) prepare projected data with respect to Eq. (2) in MVSplat, (b) compute the correlations. The part (a) dominates in the time consumption, since it computes projected features for each image pair and each depth candidates, leading to high inference time. The inference time cost by computing the correlations is not the same as the inference time of cost volume as we reported in the previous table. Actually, the inference time of computing correlations is much less than the inference time of part (a).

**Q:** Does the image encoder refers to everything before GGN?

**A:** It seems that the markdown table is not clear enough. $\Phi_{image}$ in Eq. (1) of GGN refers to multi-view feature extraction, cost volume construction and cost volume refinement. $\Phi_{depth}$ in Eq. (2) of GGN refers to depth estimation and depth refinement. We report the inference time of all these components in the following table. We will further clarify this part in our final version (line 113). 

|          | Multi-view feature extraction | Cost volume construction |        | Cost volume refinement | Depth estimation | Depth refinement|
|----------|-------------------------------|--------------------------|--------|------------------------|------------------|----------------|
|          |                               | prepare projected data   | compute the correlations |         | | |
| 2 views  | 8.37          | 6.77       | 0.09  | 6.56   | 0.27 | 7.99 |
| 4 views  | 10.01         | 50.11      | 0.19  | 6.89   | 0.27 | 8.08 |
| 8 views  | 14.09         | 237.40     | 0.37  | 7.11   | 0.28 | 8.28 |
| 16 views | 21.15         | 1022.59    | 0.96  | 7.37   | 0.29 | 8.56 |
Aha, it seems that the "Image Encoder" refers to everything before GGN right? The formatting of markdown makes it a bit confusing. If this is the case, could the authors show a breakdown of the inference time for each component in the Image Encoder? Thanks.
Thanks for your response. I am a bit confused. Initially I thought the "Image Encoder" contains everything before the GGN. Now it seems that the "Image Encoder" only refers to the feature extractor (Swin). However, from the new table, there is still one component missing: the U-Net used for depth estimation from the cost volume. Is it considered when measuring the inference time? In addition, it seems to me that the cost volume takes much more inference time than expected, since the cross-view attention in Swin is kind of similar to the cost volume computation (i.e., cross-view matching), but Swin does not take too much time. Could the authors further elaborate how to understand this behavior? Thanks.
Thanks for your comments. We follow the construction of the image encoder in MVSplat, which employs Swin Transformer and cost volume techniques to extract features. The cost volume part does cost a lot of time and memory when the number of views increase, as you mentioned. In Table 3 of our general rebuttal, we only reported the inference time of Swin Transformer as the image encoder and added the inference time taken by cost volume techniques to GGN, leading to uncorrect results. We sincerely apologize for this mistake. Thanks for your correction. We further conduct efficiency experiments and correct the inference time of each part more detailedly. Our GGN does not take much time as reported in Table 3, while the image encoder actually dominates in time consumption.

|          | Image Encoder |             |        | GGN     |  
|----------|---------------|-------------|--------|---------|
|          | Swin Trans.   | Cost Volume | All    |         | 
| 2 views  | 8.37          | 14.31       | 22.68  | 7.52    | 
| 4 views  | 10.01         | 57.38       | 67.39  | 22.07   | 
| 8 views  | 14.09         | 245.94      | 260.03 | 58.62   | 
| 16 views | 21.15         | 1028.98     | 1050.13| 131.97  |
Thanks for your comments. We report the inference time (ms) as well as memory cost (GB) in the following table. Due to the limitation of rebuttal time, we only add a simple part to our model, which leads to a little increase on inference time and memory cost. We will consider how to fully utilize the efficient point cloud processing methods instead of such simple combination as our future topic to further promote our research. Thanks for your inspiration again.
|      |Ours  |              |      |Model A|              |      |
|------|------|--------------|------|-------|--------------|------|
|Views |PSNR  |Inference Time|Memory|PSNR   |Inference Time|Memory|
|4     | 24.76| 148.1        |4.8   |24.83  |157.3         |5.4   |
|8     | 25.15| 388.8        |8.6   |25.26  |410.4         |10.1  |
|16    | 26.18| 1267.5       |21.4  |26.30  |1334.6        |26.0  |
Thanks the authors for the detailed response. Could the authors share more details about how the image encoder accepts more input views since I guess the multi-view epipolar attention in both pixelSplat and MVSplat would cost a lot of time and memory when the number of views increase? Did the author perform standard multi-view epipolar attention (for example, pair-wise) or is there any modification made here? Thanks.
Thanks the authors for the response. The comparison with Point Transformer looks interesting (Table 2), I am wondering how the proposed GGN compares with Point Transformer in terms of inference time and memory consumption with respect to different views? It would be great if the authors could also report the time and memory comparisons in Table 2 as well, thanks.
Thank you for the new experimental results. You have addressed my concerns, and I have improved the final rating.
Hi reviewers,

Thank you for your hard work in reviewing the paper!
Please check out the authors' responses and ask any questions you have to help clarify things by Aug 13.

--AC