PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views

## Abstract

Abstract We propose PixelGaussian , an efficient feed-forward framework for learning generalizable 3D Gaussian reconstruction from arbitrary views.
Most existing methods rely on uniform pixel-wise Gaussian representations, which learn a fixed number of 3D Gaussians for each view and cannot generalize well to more input views.
Differently, our PixelGaussian dynamically adapts both the Gaussian distribution and quantity based on geometric complexity, leading to more efficient representations and significant improvements in reconstruction quality.
Specifically, we introduce a Cascade Gaussian Adapter to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scorer.
CGA leverages deformable attention in context-aware hypernetworks to guide Gaussian pruning and splitting, ensuring accurate representation in complex regions while reducing redundancy.
Furthermore, we design a transformer-based Iterative Gaussian Refiner module that refines Gaussian representations through direct image-Gaussian interactions.
Our PixelGaussian can effectively reduce Gaussian redundancy as input views increase.
We conduct extensive experiments on the large-scale ACID and RealEstate10K datasets, where our method achieves state-of-the-art performance with good generalization to various numbers of views.
Code: https://github.com/Barrybarry-Smith/PixelGaussian .

## 1 Introduction

Novel view synthesis (NVS) seeks to reconstruct a 3D scene from a series of input views and generate high-quality images from previously unseen viewpoints. High-quality and real-time reconstruction and view synthesis are crucial for autonomous driving (Tonderski et al., ; Khan et al., ; Tian et al., ; Huang et al., ), robotics perception ( Wilder-Smith et al., ;  Jiang et al., ) and virtual or augmented reality (Yang et al., ;  Zheng et al., ).

NeRF-based methods ( Mildenhall et al., ;  Hu et al., ;  Liu et al., ;  Neff et al., ) have achieved remarkable success by encoding 3D scenes into implicit radiance fields, yet sampling volumes for NeRF rendering is costly in both time and memory. Recently,  Kerbl et al. () proposed to represent 3D scenes explicitly using a set of 3D Gaussians, enabling much more efficient and high-quality rendering via a differentiable rasterizer. Still, the original 3D Gaussian Splatting requires separate optimization on each single scene, which significantly reduces inference efficiency.
To tackle this problem, recent researches have aimed at generating 3D Gaussians directly from a feed-forward network without any per-scene optimization ( Charatan et al., ;  Chen et al., ;  Liu et al., ;  Szymanowicz et al., ;  Zheng et al., ). Typically, these approaches adhere to a paradigm where a fixed number of Gaussians is predicted for each pixel in the input views. The Gaussians derived from different views are then directly merged to construct the final 3D scene representation. However, such a paradigm limits the model performance as the Gaussian splats are uniformly distributed across images, making it difficult to capture local geometric details effectively. Additionally, as the number of input views increases, directly merging Gaussians can degrade reconstruction performance due to severe Gaussian overlap and redundancy across views.

To address this, we propose PixelGaussian, which enables dynamic adaption on both 3D Gaussian distribution and quantity. To be specific, we first uniformly initialize Gaussian positions following  Chen et al. () to accurately localize the Gaussian centers. To identify geometry complexity across images, we then compute a relevance score map for each input view from image features in an end-to-end manner. Under the guidance of score maps, we construct a Cascade Gaussian Adapter (CGA), which leverages deformable attention (Xia et al., ) to control the pruning and splitting operations. After CGA, more Gaussians are allocated to regions with complex geometry for precise reconstruction, while unnecessary and duplicate Gaussians across views are pruned to reduce redundancy and improve efficiency. Since these Gaussian representations still fall short in fully capturing the image details, we further introduce a transformer-based Iterative Gaussian Refiner (IGR) to refine 3D Gaussians through direct image-Gaussian interactions. Finally, we employ rasterization-based rendering using the refined Gaussians to generate novel views at target viewpoints.

We conduct extensive experiments on ACID () and RealEstate10K (Zhou et al., ) benchmarks for large-scale 3D scene reconstruction and novel view synthesis.
PixelGaussian outperforms existing methods on different input views with a comparable inference speed.
Notably, existing generalizable 3D Gaussian splatting methods (pixelSplat (Charatan et al., ) and MVSplat (Chen et al., )) fail to achieve good results when transferring to more input views while our method demonstrates consistent performance.
This is because existing pixel-wise methods generate uniform pixel-aligned Gaussian predictions, and our model mitigates Gaussian overlap and redundancy across views by dynamically adjusting their distribution based on local geometry complexity.
Visualizations and ablations further demonstrate that both CGA and IGR blocks are crucial in adapting Gaussian distribution, enabling the proposed PixelGaussian to capture geometry details and achieve better reconstruction accuracy.

## 2 Related Work

Multi-View Stereo.
Multi-View Stereo (MVS) aims to reconstruct a 3D representation from multi-view images of a given scene or object. Since accurate depth estimation is essential for reliable 3D reconstruction from 2D inputs, most MVS methods ( Gu et al., ;  Ding et al., ;  Yao et al., ) require ground truth depth for supervision in training process. Additionally, point-based MVS approaches generally separate the processes of depth estimation and point cloud fusion processes. Recently, inspired by efficient Gaussian representations proposed by Kerbl et al. (), Chen et al. () introduces to directly predict depth for pixel-wise Gaussians from a cost volume structure without requiring depth supervision, significantly improving model scalability and flexibility. Therefore, following a similar approach, we construct a lightweight cost volume to facilitate depth estimation, which serves as an efficient initialization for 3D Gaussians in our PixelGaussian.

Per-scene 3D Reconstruction.
Neural Radiance Fields (NeRF) have revolutionized the field of 3D reconstruction by representing scenes as implicit neural fields (Mildenhall et al., ). Subsequent researches have focused on overcoming the limitations of the original NeRF to improve its performance and broaden its applicability. Some researches aim to improve the efficiency for novel view synthesis ( Hu et al., ;  Fridovich-Keil et al., ;  Yu et al., ;  Liu et al., ;  Neff et al., ). Moreover, several studies concentrate on capturing intricate geometry and temporal information to achieve accurate and dynamic reconstruction ( Li et al., ;  Du et al., ;  Pumarola et al., ;  Tian et al., ;  Wang et al., ). Compared to implicit NeRF-based methods, 3D Gaussian Splatting (3DGS) (Kerbl et al., ) represents a 3D scenario as a set of explicit 3D Gaussians, enabling a rasterization-based splatting rendering process that is significantly more efficient in both time and memory. Given that 3DGS still requires millions of 3D Gaussians to represent a single scene, numerous studies have focused on achieving real-time rendering and minimizing memory usage ( Fan et al., ;  Katsumata et al., ;  Lu et al., ). Additionally, some researches focus on enhancing the reconstruction quality of 3DGS by employing multi-scale rendering (Yan et al., ), advanced shading models (Jiang et al., ) or incorporating physically based properties for realistic relighting (Gao et al., ). However, these methods still require per-scene optimization and rely on dense input views, which can be computationally expensive and limit their scalability for large-scale or dynamic scenes.

Generalizable 3D Reconstruction.
PixelNeRF (Yu et al., ) pioneers the approach of predicting pixel-wise features directly from input views to reconstruct neural radiance fields. Following methods incorporate volume or transformer architectures to improve the performance of feed-forward NeRF models ( Chen et al., ;  Xu et al., ;  Miyato et al., ;  Sajjadi et al., ;  Du et al., ). However, these feed-forward NeRF approaches typically demand substantial memory and computational resources due to the expensive per-pixel volume sampling process (Wang et al., ; Johari et al., ; Barron et al., ; Garbin et al., ; Reiser et al., ; Müller et al., ). With the advent of 3DGS, PixelSplat (Charatan et al., ) initiates a shift towards feed-forward Gaussian-based reconstruction. It takes sparse input views to directly predict pixel-wise 3D Gaussians by leveraging epipolar geometry to learn cross-view features. MVSplat (Chen et al., ) constructs a cost volume structure for depth estimation, which significantly boosts both model efficiency and reconstruction quality. Additionally, MVSGaussian (Liu et al., ) further improves model performance by introducing an efficient hybrid Gaussian rendering process. Moreover, SplatterImage (Szymanowicz et al., ) and GPS-Gaussian (Zheng et al., ) predict pixel-wise 3D Gaussians for object-centric or human reconstruction.

However, these feed-forward methods are constrained by the pixel-wise Gaussian prediction paradigm, which limits the model’s performance as the Gaussian splats are uniformly distributed across images. Such a paradigm inadequately captures intricate geometries, while also causing Gaussian overlap and redundancy across views, ultimately resulting in severe rendering artifacts. In comparison, PixelGaussian consists of a Cascade Gaussian Adapter (CGA), allowing for dynamic adaption on both Gaussian distribution and quantity. Visualizations demonstrate that CGA is capable of allocating more Gaussians in areas rich in geometric details, while reducing duplicate Gaussians in similar regions across input views. Furthermore, we introduce an Iterative Gaussian Refiner (IGR), enabling direct interaction between 3D Gaussians and local image features via deformable attention. Experimental results show that IGR effectively leverages image features to guide Gaussians in capturing the full information contained within the images, significantly enhancing the model’s ability to capture local intricate geometry.

## 3 Proposed Approach

In this section, we present our method to learn generalizable Gaussian representations from arbitrary views. Given an arbitrary set of input images $\mathcal{I}=\{I_{i}\}_{i=1}^{N}\in\mathbb{R}^{N\times H\times W\times 3}$ and corresponding camera poses $\mathcal{C}=\{C_{i}\}_{i=1}^{N}$, our PixelGaussian aims to learn a mapping $\mathcal{M}$ from images to 3D Gaussians for scene reconstruction:

$$ $\mathcal{M}:\{(I_{i},C_{i})\}_{i=1}^{N}\mapsto\{(\mu_{j},s_{j},r_{j},\alpha_{j },sh_{j})\}_{j=1}^{N_{K}},$ (1) $$

where $N_{K}$ is the total number of 3D Gaussians, which adaptively varies depending on the scene context. Each Gaussian is parameterized by its position $\mu_{j}$, scaling $s_{j}$, rotation $r_{j}$, opacity $\alpha_{j}$ and spherical harmonics $sh_{j}$.

As illustrated in Figure 2, we first use a lightweight cost volume for depth estimation and Gaussian position initialization. We then introduce Cascade Gaussian Adapter (CGA), which dynamically adapts both Gaussian distribution and quantity based on local geometric complexity. Finally, we explain how Iterative Gaussian Refiner (IGR) enables direct image-Gaussian interactions, further refining Gaussian distribution and representations for enhanced reconstruction.

Figure: Figure 2: Overview of PixelGaussian. Given multi-view input images, we initialize 3D Gaussians using a lightweight image encoder and cost volume. Cascade Gaussian Adapter (CGA) then dynamically adapts both the distribution and quantity of Gaussians. By leveraging local image features, Iterative Gaussian Refiner (IGR) further refines Gaussian representations via deformable attention. Finally, novel views are rendered from the refined 3D Gaussians using rasterization-based rendering.
Refer to caption: x2.png

### 3.1 Gaussian Initialization

Position Initialization.
Following the instructions of MVSplat (Chen et al., ), we first extract image features via a 2D backbone consisting of CNN and Swin Transformer (Liu et al., ). Specifically, CNN encodes multi-view images to corresponding feature maps, while Swin Transformer performs both self-attention and cross-view attention to better leverage information cross views. Then, we obtain the aggregated multi-view features $\mathcal{F}=\{F_{i}\}_{i=1}^{N}$.

To initialize Gaussian positions precisely, we construct a lightweight cost volume (Yao et al., ) for depth estimation, denoted as $\Phi_{depth}$. We then predict Gaussian centers as follows:

$$ $\mu=P^{-1}(\Phi_{depth}(\mathcal{F}),\mathcal{C})$ (2) $$

where $P^{-1}(\cdot)$ stands for unprojection operation.

Parameter Initialization.
For each Gaussian center $\mu_{j}$, we randomly set corresponding scaling $s_{j}\in\mathbb{R}^{3}$, rotation $r_{j}\in\mathbb{R}^{4}$, opacity $\alpha_{j}\in\mathbb{R}^{1}$, spherical harmonics $sh_{j}\in\mathbb{R}^{C}$ within a proper range. we then get the initial Gaussians set $\mathcal{G}=\{(\mu_{j},s_{j},r_{j},\alpha_{j},sh_{j})\}_{j=1}^{HW}\in\mathbb{R
}^{HW\times(11+C)}$.

### 3.2 CGA: Cascade Gaussian Adapter

After obtaining the initial Gaussian set $\mathcal{G}$, we introduce Cascade Gaussian Adapter (CGA) driven by a multi-view keypoint scorer $\Psi$, as shown in Figure 3(a). CGA contains a set of context-aware hypernetworks $\mathcal{H}$ which dynamically control and guide the following Gaussian pruning and splitting operations. This approach ensures that regions with complex geometry details are represented by a greater number of Gaussians, while areas with poor geometry can be represented with fewer Gaussians. In parallel, CGA effectively removes redundant Gaussians to prevent Gaussian overlap across views. Compared to previous pixel-wise methods, which rigidly allocate a fixed number of Gaussians per pixel, our design dynamically adapts both distribution and quantity of Gaussians based on geometric complexity. This flexibility allows for a more accurate capture of local geometry and mitigates the problem of Gaussian overlap, thereby improving the overall quality of reconstruction.

Figure: Figure 3: Illustration of the proposed CGA and IGR Blocks. (a) CGA comprises a keypoint scorer followed by a series of hypernetworks that produce context-aware thresholds to guide the splitting and pruning of Gaussians. (b) IGR further facilitates direct image-Gaussian interactions, enabling Gaussian representations to capture and extract local geometric features more effectively.
Refer to caption: x3.png

Given the aggregated features $\mathcal{F}$ derived in Section 3.1, $\Psi$ computes relevance score maps $\mathcal{R}=\{R_{i}\}_{i=1}^{N}\in\mathbb{R}^{N\times H\times W}$, where each score map $R_{i}$ is obtained by a learnable weighted average of contributions from different views:

$$ $\mathcal{R}=\Psi(\mathcal{F})=\textit{softmax}\left(\textit{MLP}\left(\sum_{i= 1}^{N}\alpha_{i}\cdot F_{i}\right)\right),\quad\alpha_{i}=\frac{\exp(\beta_{i} )}{\sum_{j=1}^{N}\exp(\beta_{j})},$ (3) $$

where $A=[\alpha_{1},\alpha_{2},\ldots,\alpha_{N}]^{T}\in\mathbb{R}^{N}$ represents the contribution factor of each view, and is determined by learnable parameters $\beta_{i}(i=1,2,...,N)$.

We first introduce a set of hypernetworks $\mathcal{H}=\{H_{k}\}_{k=1}^{K}$ to generate context-aware thresholds. CGA is composed of $K$ stages, where each stage $H_{k}$ takes score maps $\mathcal{R}$ along with Gaussian set $\mathcal{G}_{k}=\{(\mu_{j}^{(k)},s_{j}^{(k)},r_{j}^{(k)},\alpha_{j}^{(k)},sh_{
j}^{(k)})\}_{j=1}^{N_{k}}\in\mathbb{R}^{N_{k}\times(11+C)}$ as input, and outputs thresholds $\tau_{high}^{(k)},\tau_{low}^{(k)}\in\mathbb{R}$ for splitting and pruning. As illustrated in equation 4, we first sample and embed Gaussian set $\mathcal{G}_{k}$ into Gaussian score queries $\mathcal{Q}_{r}^{(k)}$. Then we project sampled reference points $\mu^{(k)}$ onto score maps $\mathcal{R}$ with corresponding camera parameters $\mathcal{C}$. Finally, we update queries $\mathcal{Q}_{r}^{(k)}$ with weighted scores from $\mathcal{S}$ and get both thresholds through a simple MLP. Initially, we set $\mathcal{G}_{1}=\mathcal{G}$.

$$ $\tau_{high}^{(k)},\tau_{low}^{(k)}=\mathcal{H}_{k}(\mathcal{G}_{k},\mathcal{R} ,\mathcal{C})=\textit{MLP}(\sum_{i=1}^{N}\alpha_{i}\cdot\textit{DA}(\mathcal{Q }_{r}^{(k)},R_{i},P(\mu^{(k)},C_{i}))),$ (4) $$

where $DA(\cdot),P(\cdot)$ denote the deformable attention function and projection operation, respectively.

Then, we obtain Gaussian-wise scores by projecting Gaussian centers onto score maps $\mathcal{R}$. To elaborate, let $S_{k}=\{s_{ij}^{(k)}\}\in\mathbb{R}^{N\times N_{k}}$ be the score matrix for Gaussian set $\mathcal{G}_{k}$, where each score $s_{ij}^{(k)}$ is the value at the projection point of the $j-th$ Gaussian center in $R_{i}$, or 0 0 if it does not project onto any region in $R_{i}$. The final Gaussian-wise scores $S_{k}^{avg}$ are then computed by averaging scores across different views:

$$ $S_{k}^{avg}=S_{k}^{T}\cdot A,$ (5) $$

Once Gaussian-wise scores are obtained, regions with higher scores, indicating more complex geometry details, undergo splitting operation to allocate more Gaussians for finer representations. For regions with lower scores, we apply an opacity-based pruning operation, gradually reducing Gaussian opacity and scaling to minimize their impact and reduce redundancy.

Splitting. For Gaussian $g_{j}^{(k)}\in\mathcal{G}_{k}$ with score higher than $\tau_{\text{high}}^{(k)}$, we generate $M$ separate new Gaussians for more detailed representations:

$$ $G_{j}^{(k)}=\textit{SplitNet}(g_{j}^{(k)})\in\mathbb{R}^{M\times(11+C)},$ (6) $$

where SplitNet$(\cdot)$ is a simple MLP-based network that ensures all parameters within proper range. The newly generated Gaussians are then directly concatenated with the existing Gaussian set $\mathcal{G}_{k}$.

Pruning. For Gaussian $g_{j}^{(k)}\in\mathcal{G}_{k}$ with score lower than $\tau_{\text{low}}^{(k)}$, we apply an opacity-based pruning operation rather than directly removing it. Specifically, we set a predefined opacity threshold $\tau_{\alpha}$. If the Gaussian opacity is greater than $\tau_{\alpha}$, we gradually reduce its opacity and scaling:

$$ $\alpha_{j}^{(k)}\rightarrow\gamma_{\alpha}\cdot\alpha_{j}^{(k)},\quad s_{j}^{( k)}\rightarrow\gamma_{s}\cdot s_{j}^{(k)},$ (7) $$

where $\gamma_{\alpha}<1$ and $\gamma_{s}<1$ are reduction factors. Otherwise, the current Gaussian is removed entirely from Gaussian set $\mathcal{G}_{k}$.

After K-stage adaptation in the Cascade Gaussian Adapter, the initial uniform 3D Gaussian representations are transformed into adaptive forms. Gaussians are redistributed according to geometric complexity, resulting in a more efficient and context-aware representation.

### 3.3 IGR: Iterative Gaussian Refiner

Though CGA allows for a more optimal Gaussian distribution, the Gaussian representations still fall short in capturing the full information contained in the images. Inspired by the efficiency demonstrated by GaussianFormer (Huang et al., ) in occupancy prediction, we design a transformer-based Iterative Gaussian Refiner (IGA) to further extract local geometric information from input views, as shown in Figure 3(b). In this process, we leverage deformable attention to enable direct image-Gaussian interactions, enhancing the ability for 3D Gaussians to more accurately capture intricate geometry details in reconstruction and view synthesis.

IGR is composed of $B$ attention and refinement blocks. In Section 3.2, CGA adapts the original Gaussian set $\mathcal{G}$ to $\mathcal{G}=\mathcal{G}_{K}$. To continue, we first sample and embed $\mathcal{G}$ into Gaussian queries $\mathcal{Q}$. In each block, deformable attention is first applied between Gaussian queries $\mathcal{Q}$ and multi-view features $\mathcal{F}$ to update Gaussian representations. This is followed by a refinement stage where a residual module further fine-tunes the queries. The overall process of IGR can be formulated as:

$$ $\mathcal{Q}_{b}=\Phi_{ref}(\sum_{i=1}^{N}\alpha_{i}\cdot DA(\mathcal{Q}_{b-1}, F_{i},P(\mu^{(b)},C_{i})))\quad b=1,2,\dots,B,$ (8) $$

where $DA(\cdot),\Phi_{ref}(\cdot),P(\cdot)$ denote the deformable attention function, refinement layer and projection operation, $F_{i},C_{i},\alpha_{i}$ represents the image feature, camera parameters and contribution factor of input view $I_{i}$, respectively. $\mathcal{Q}_{b}(b=1,2,...,B)$ stands for output queries from the $b-th$ IGR block, and $\mu^{(b)}$ is the Gaussian center of current stage. Initially, we set $\mathcal{Q}_{0}=\mathcal{Q}$.

Finally, the refined Gaussian queries are decoded into Gaussian parameters $\mathcal{G}_{f}$ through a simple MLP to ensure all parameters within proper range, and then can be used for rasterization-based rendering at novel viewpoints.

$$ $\mathcal{G}_{f}=\{(\mu_{j}^{f},s_{j}^{f},r_{j}^{f},\alpha_{j}^{f},sh_{j}^{f}) \}_{j=1}^{N_{K}}=\textit{MLP}(\mathcal{Q}_{B}).$ (9) $$

Our full model takes ground-truth target RGB images at novel viewpoints as supervision, allowing for efficient end-to-end training. The training loss is calculated as a linear combination of MSE and LPIPS (Zhang et al., ) losses, with loss weights of 1 and 0.05, respectively.

Compared to the uniform pixel-wise paradigm, our PixelGaussian approach dynamically adapts both the Gaussian distribution and quantity within the Cascade Gaussian Adapter. Additionally, the Iterative Gaussian Refiner refines Gaussian representations to capture intricate geometric details in the input views. This design achieves more efficient Gaussian distributions while mitigating overlap and redundancy common in pixel-wise methods.

## 4 Experiments

### 4.1 Experimental Settings

Datasets. To assess the performance of our model, we conduct experiments on two extensive datasets: ACID () and RealEstate10K (Zhou et al., ). The ACID dataset consists of video frames capturing natural landscape scenes, comprising 11,075 scenes in the training set and 1,972 scenes in the test set. RealEstate10K provides video frames from real estate environments, with 67,477 scenes allocated for training and 7,289 scenes reserved for testing. The model is trained with two reference views, and four novel views are selected for evaluation. During testing, however, we perform multiple experiments where 2, 3, and 4 views are selected for reference, while four novel views selected for evaluation in each scenario.

Implementation Details. We set the resolutions of input images as 256x256. In Cascade Gaussian Adapter (CGA), we apply $K=3$ stages of cascade Gaussian adaption. As for the splitting operation, the SplitNet generates $M=1$ separate new Gaussians, whereas the pruning process uses reduction factors $\gamma_{\alpha}=\gamma_{s}=0.5$ and opacity threshold $\tau_{\alpha}=0.3$. We use $B=3$ blocks in Iterative Gaussian Refiner (IGR) to extract local geometry from input views. We implement our PixelGaussian with Pytorch and train the model on 8 NVIDIA A6000 GPUs for 300,000 iterations with Adam optimizer. More training details are provided in Section A.2.

### 4.2 Main Results

**Table 1: Results of Novel View Synthesis on the RealEstate10K and ACID benchmarks. We report the average PSNR and LPIPS (Zhang et al., ) on the test set, where all models are trained with 2 reference views and inferred with 2, 3, and 4 reference views.**
| Datasets | Methods | 2$\rightarrow$2 Views | 2$\rightarrow$3 Views | 2$\rightarrow$4 Views |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$ |  |  |
| RealEstate10K | pixelNeRF | 20.25 | 0.556 | 21.15 | 0.525 | 21.68 | 0.518 |
| MuRF | 25.95 | 0.146 | 26.23 | 0.137 | 26.40 | 0.134 |  |
| pixelSplat | 25.67 | 0.145 | 22.35 | 0.242 | 20.12 | 0.283 |  |
| MVSplat | 26.25 | 0.130 | 22.94 | 0.236 | 20.74 | 0.268 |  |
| PixelGaussian | 26.72 | 0.126 | 26.79 | 0.123 | 26.85 | 0.122 |  |
| ACID | pixelNeRF | 20.66 | 0.530 | 21.33 | 0.518 | 21.40 | 0.506 |
| MuRF | 27.94 | 0.163 | 28.22 | 0.154 | 28.35 | 0.149 |  |
| pixelSplat | 28.06 | 0.152 | 23.73 | 0.276 | 20.15 | 0.294 |  |
| MVSplat | 28.26 | 0.144 | 23.85 | 0.268 | 20.32 | 0.275 |  |
| PixelGaussian | 28.63 | 0.140 | 28.72 | 0.137 | 28.78 | 0.137 |  |

Novel View Synthesis. As shown in Table 1 and Figure 4, our proposed PixelGaussian consistently outperforms previous NeRF-based methods and pixel-wise Gaussian feed-forward networks across all settings with 2, 3, and 4 reference views. Notably, as the number of input views increases, the reconstruction performance of both pixelSplat (Charatan et al., ) and MVSplat (Chen et al., ) degrades significantly, while PixelGaussian shows a slight improvement. This is because previous methods directly merge multiple views by back-projecting pixel-wise Gaussians to 3D space based on depth maps. Without the capability to adapt the quantity and distribution of Gaussians dynamically, pixel-wise methods often produce duplicated Gaussians with significant overlap, and their spatial positioning is suboptimal. In contrast, PixelGaussian is able to optimize both the distribution and quantity of Gaussians via CGA, while IGR blocks facilitate direct interaction between Gaussian queries and local image features, resulting in more accurate reconstructions.

Figure: Figure 4: Visualization results on ACID and RealEstate10K benchmarks. Pixel-wise methods suffer from Gaussian overlap due to suboptimal Gaussian distributions, whereas PixelGaussian enables dynamic Gaussian adaption and improved local geometry refinement.
Refer to caption: x4.png

Multi-View Comparison. We further compare model performance and Gaussian quantities of different methods across various input views in Table 2. Though our method requires more Gaussians than MVSplat (Chen et al., ) with 2 input views due to more frequent splitting than pruning, it achieves better reconstruction with fewer Gaussians as the number of views increases. In regions with richer geometric details, CGA blocks first split more Gaussians for finer representations, followed by IGR to further refine these Gaussians using deformable attention on local image features to better capture and reconstruct geometric details. Meanwhile, CGA prunes duplicate and overlapping Gaussians across views to control the growth of overall Gaussian quantity as the number of input views increases.

**Table 2: Comparison of PSNR and Gaussian Quantity on RealEstate10K Dataset. We present the average PSNR and the number of Gaussians (K) for inference using 2, 4, and 6 input views.**
| Methods | 2$\rightarrow$2 Views | 2$\rightarrow$4 Views | 2$\rightarrow$6 Views |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | # Gaussians | PSNR$\uparrow$ | # Gaussians | PSNR$\uparrow$ | # Gaussians |  |
| pixelSplat | 25.67 | 393 K | 20.12 | 786 K | 19.36 | 1179 K |
| MVSplat | 26.25 | 131 K | 20.74 | 262 K | 20.24 | 393 K |
| PixelGaussian | 26.72 | 188 K | 26.85 | 240 K | 26.89 | 278 K |

Efficiency Analysis. We explore the efficiency of PixelGaussian compared with dominant pixel-wise methods on a single NVIDIA A6000 GPU. All models are inferred with 4 input views on the RealEstate10K (Zhou et al., ) dataset. We report the average inference latency, memory cost, number of Gaussians and rendering FPS in Table 3. Undeniably, PixelGaussian requires higher latency and memory usage than MVSplat (Chen et al., ) due to the extra cost of CGA and IGR blocks. However, this trade-off allows PixelGaussian to achieve higher rendering FPS by utilizing fewer Gaussians and far more better reconstruction quality as the number of input view increases.

**Table 3: Efficiency analysis. Inference on RealEstate10K (Zhou et al., ) dataset using 4 input views, reporting average latency, memory cost, Gaussian quantity, and rendering FPS.**
| Methods | Latency | Memory | # Gaussians | Rendering FPS |
| --- | --- | --- | --- | --- |
| pixelSplat | 298 ms | 11.78 G | 786 K | 64 |
| MVSplat | 126 ms | 3.17 G | 262 K | 133 |
| PixelGaussian | 235 ms | 4.39 G | 240 K | 140 |

### 4.3 Expeimental Analysis

In this section, we further investigate and conduct experiments to demonstrate the effectiveness of our PixelGaussian. We first visualize the score maps $\mathcal{S}$ and Gaussian density maps. Then, we present the cascade adaption process of CGA. Finally, we conduct ablation studies on our model. These experiments show that CGA dynamically adapts both the distribution and quantity of Gaussians according to geometric complexity, while IGR further extract local features via direct image-Gaussian interactions, offering significant improvements over traditional pixel-wise methods.

Score Maps and Density Maps. As shown in 3, relevance score maps $\mathcal{S}$ are derived from multi-view image features through the keypoint scorer $\Psi$. To further understand the significance of $\mathcal{S}$ and its impact on the subsequent Gaussian splitting and pruning operations, we visualize the relevance score maps on the RealEstate10K (Zhou et al., ) dataset. Furthermore, we project the centers of the adaptive Gaussian after CGA $(i.e.\{\mu_{j}^{f}\}_{j=1}^{N_{K}})$, onto each input view as Gaussian density maps. Figure 5 illustrates that in this end-to-end learning framework, keypoint scorer $\Psi$ is able to learn such score maps where local regions with richer and more complex geometric details tend to receive higher score, which guides the following SplitNet to allocate more Gaussians for more precise representations. In contrast, regions with fewer geometric details receive lower scores, leading to pruning to reduce representation complexity while preserving overall representation efficiency.

Figure: Figure 5: Visualization of score maps and Gaussian distributions on RealEstate10K dataset. Cascade Gaussian Adapter dynamically adjusts Gaussian distribution and quantity based on score maps. More Gaussians are allocated to detail-rich regions for more precise representations, while pruning minimizes Gaussian redundancy and overlap across views.
Refer to caption: x5.png

Cascade Gaussian Adaption. To present the cascade Gaussian adaption process in CGA blocks more explicitly, we visualize Gaussian density maps during model inference on the RealEstate10K (Zhou et al., ) dataset with 2 input views. As illustrated in Figure 6, the original Gaussian set $\mathcal{G}$ is initialized on a pixel-wise basis. Following this, score maps $\mathcal{S}$ guide hypernetworks $\mathcal{H}$ to generate context-aware thresholds, which in turn direct the subsequent Gaussian splitting and pruning operations. More Gaussians are allocated in regions with richer geometric details, while duplicate Gaussians across views tend to be pruned, leading to a more efficient and optimal Gaussian distribution for scene representation.

Figure: Figure 6: Visualization of Gaussian adaption process in Cascade Gaussian Adapter. During inference on RealEstate10K (Zhou et al., ) dataset, CGA progressively concentrates more Gaussians on geometrically complex regions, while pruning duplicate Gaussians across views to control the growth of overall Gaussian quantity.
Refer to caption: x6.png

Ablation Study. To further investigate the architecture of PixelGaussian, we conduct ablation studies by inferring our model on RealEstate10K (Zhou et al., ) test dataset with 4 input views. We first introduce a vanilla model, where the initial Gaussian set $\mathcal{G}$ is directly used to render novel views. Then, we adopt rigid CGA blocks without context-aware Hypernetworks $\mathcal{H}$, which means Gaussian set $\mathcal{G}$ goes through splitting and pruning based on fixed thresholds $(\tau_{high}^{(k)}=0.8,\tau_{low}^{(k)}=0.2,k=1,2,...,K)$. We further add HyperNetworks $\mathcal{H}$ to generate context-aware thresholds. Finally, we adopt IGR blocks to refine the Gaussian representations via image-Gaussian interactions. As shown in Table 4, HyperNetworks $\mathcal{H}$ utilizes score maps $\mathcal{S}$ to generate context-aware thresholds, enabling a more dynamic and efficient Gaussian distribution for scene representation compared to rigid splitting and pruning. Furthermore, IGR blocks refine the Gaussian set iteratively via deformable attention between Gaussians and image features, enhancing their ability to describe and reconstruct intricate local geometric details.

**Table 4: Ablations on the components of PixelGaussian. We report the average PSNR, LPIPS (Zhang et al., ), and the number of Gaussians (K) of model inference.**
| Methods | PSNR$\uparrow$ | LPIPS$\downarrow$ | #Gaussians |
| --- | --- | --- | --- |
| Vanilla | 20.34 | 0.272 | 262 K |
| + Rigid Cascade Gaussian Adapter | 22.46 | 0.220 | 225 K |
| + HyperNetworks $\mathcal{H}$ | 25.80 | 0.140 | 240 K |
| + Iterative Gaussian Refiner | 26.85 | 0.122 | 240 K |

## 5 Conclusion and Discussions

In this paper, we have presented PixelGaussian to learn generalizable 3D Gaussian reconstruction from arbitrary input views. The core innovation of our approach is context-aware Cascade Gaussian Adapter (CGA), which dynamically splits Gaussians in regions with complex geometric details and prunes redundant ones. Further, we incorporate deformable attention within Iterative Gaussian Refiner (IGR), facilitating direct image-Gaussian interactions to improve local geometry reconstructions. Compared to previous uniform pixel-wise methods, PixelGaussian is able to dynamically adapt both Gaussian distribution and quantity guided by the complexity of local geometry details, allocating more to detailed regions and reducing redundancy across views, thus leading to better performance in reconstruction and view synthesis.

Discussions and Limitations. Although PixelGaussian can adjust the distribution of 3D Gaussians dynamically, the initial Gaussians are still derived from pixel-wise unprojection. When we initialize the Gaussian centers completely at random, the model fails to converge. Moreover, deformable attention in IGR consumes substantial computational resources when the number of Gaussians is extremely large, highlighting the need for a more efficient approach to represent 3D scenes with fewer Gaussians. Furthermore, PixelGaussian is unable to perceive the unseen parts of 3D scenes beyond the input views, suggesting the potential need to incorporate generative models.

## Appendix A Appendix

### A.1 Preliminary

#### A.1.1 3D Gaussian Splatting

3D Gaussian Splatting (Kerbl et al., ) represents a 3D scene as a set of explicit Gaussian primitives as follows:

$$ $\mathcal{G}=\{g_{i}|g_{i}=(\mu_{i},\Sigma_{i},\alpha_{i},sh_{i})\}_{i=i}^{N}$ (10) $$

where each Gaussian has a center $\mu_{i}$, a covariance $\Sigma_{i}$, an opacity $\alpha_{i}$ and spherical harmonics $sh_{i}$. Furthermore, given the scaling matrix $S$ and rotation matrix $R$, we can calculate the covariance matrix:

$$ $\Sigma=RSS^{T}R^{T}$ (11) $$

As explicit Gaussian primitives $\mathcal{G}$ can be rendered via an rasterization-based operation, such approach is much more cheaper in both time and memory compared to implicit neural fields ( Mildenhall et al., ;  Barron et al., ;  Tian et al., ;  Chen et al., ;  Garbin et al., ) or voxel-based representations( Sitzmann et al., ;  Ji et al., ;  Xie et al., ;  Tatarchenko et al., ;  Choy et al., ).

#### A.1.2 Deformable Attention

Since the introduction of ViT (Dosovitskiy et al., ), numerous efficient attention mechanisms have been proposed to further enhance the scalability and reduce the complexity of ViT ( Liu et al., ;  Chen et al., ; Wang et al., ; Yang et al., ; Zhang et al., ; Dong et al., ; Jaegle et al., ; Sun et al., ; Zhu et al., ; Yue et al., ; Chen et al., ). Among them, Deformable Attention Transformer (DAT) (Xia et al., ) stands out for its ability to effectively capture multi-scale features and adaptively focus on important regions while maintaining lightweight computational overheads. Therefore, we apply deformable attention in both the Score Hypernetworks $\mathcal{H}$ (equation 4) and Iterative Gaussian Refiner (IGR) (equation 8) in our PixelGaussian model. We elaborate equation 8 as an example.

Given Gaussian centers $\mu$, corresponding queries $\mathcal{Q}$, feature maps $\mathcal{F}=\{F_{i}\}_{i=1}^{N}$ and camera parameters $\mathcal{C}=\{C_{i}\}_{i=1}^{N}$, we first project Gaussian centers to pixel coordinates to get reference points, denote as $\mathcal{R}=\{R_{i}\}_{i=1}^{N}$:

$$ $R_{i}=P(\mu,C_{i}),(i=1,2,...,N)$ (12) $$

where $P(\cdot)$ denotes projection operation. Next, for Gaussian queries $\mathcal{Q}$, we perform deformable sampling from feature maps $\mathcal{F}$ at reference points $\mathcal{R}$ using bilinear interpolation as follows:

$$ $\phi(F_{i},(r_{x},r_{y}))=\sum_{(p_{x},p_{y})}g(r_{x},p_{x})\cdot g(r_{y},p_{y })\cdot F_{i}(p_{x},p_{y})$ (13) $$

where $g(x,y)=max(0,1-|a-b|),(r_{x},r_{y}),F_{i}$ denote reference point and feature map, respectively. Then we can update queries $\mathcal{Q}$ via attention:

$$ $\displaystyle\mathcal{Q}=\sum_{i=1}^{N}\alpha_{i}\cdot(W_{i}\cdot\phi(F_{i},R_ {i}))$ (14) $\displaystyle W_{i}=softmax(\mathcal{Q}\cdot\phi(F_{i},R_{i})^{T})$ (15) $$

where $\alpha_{i},(i=1,2,...,N)$ represents view weights as illustrated in equation 3. Following this paradigm, we incorporate deformable attention in the score hypernetworks $\mathcal{H}$ to generate context-aware thresholds, and in IGR blocks to further refine Gaussian representations using image features.

### A.2 More Implementation Details

Training Details. As mentioned in Section 4.1, our model is trained on 8 NVIDIA A6000 GPUs, The batch size for a single GPU is set to 4, where each batch contains a large-scale 3D scene with two reference views and four inference views. Detailed settings of image backbone to generate aggregated features $\mathcal{F}$, context-aware score hypernetworks $\mathcal{H}$ in Cascade Gaussian Adapter and Iterative Gaussian Refiner are illustrated in Table 7. For MLP-based networks in equation 6 and equation 9, they all follow criteria in Table 7 to ensure all Gaussian parameters within proper range.

**Table 5: Details of Gaussian Parameter Range.**
| Gaussian Parameters | Criterion |
| --- | --- |
| scaling $s\in\mathbb{R}^{3}$ | $s_{x},s_{y},s_{z}\in[0.50,15.00]$ |
| rotation $r\in\mathbb{R}^{4}$ | $||r||=1$ |
| opacity $\alpha\in\mathbb{R}$ | $\alpha\in[0,1]$ |
| harmonics $sh\in\mathbb{R}^{C}$ | decaying coefficients |

Network Architecture. As illustrated in Table 7, the image backbone comprises a ResNet-based CNN (He et al., ) for feature extraction and a Swin Transformer (Liu et al., ) for multi-view attention. For context-aware score hypernetworks $\mathcal{H}$ in CGA, we employ $K=3$ stages for adaptive Gaussian adaption. Then, we adopt $B=3$ deformable attention blocks in IGR for Gaussian representation refinement.

### A.3 More Results

We provide more visualization comparisons of PixelGaussian and previous dominant pixel-wise methods (Charatan et al., ; Chen et al., ) on ACID () and RealEstate10K (Zhou et al., ) benchmarks. Additionally, we provide visualizations of more score maps and Gaussian density maps from our PixelGaussian model. As shown in Figure 7 and Figure 8, PixelGaussian benefits from dynamic adaption of Gaussian distributions, resulting in superior reconstruction compared to previous pixel-wise methods.

Figure: Figure 7: Visualization of score maps and Gaussian distributions of Cascade Gaussian Adapter.
Refer to caption: x7.png

Figure: Figure 8: Visualization results on ACID () and RealEstate10K (Zhou et al., ) benchmarks.
Refer to caption: x8.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Kindly refer to [Weaknesses].
soundness: 3
strengths: * The motivation is reasonable, emphasising the significance of updating the predicted pixel-aligned 3D Gaussians under more input views.
* Experiments on two existing benchmarks verify the efficacy of the introduced PixelGaussian
summary: This work introduces PixelGaussian, a feed-forward 3DGS model that can adaptively update the number of Gaussian via pruning and splitting. In particular, PixelGaussian starts by predicting the pixel-aligned 3D Gaussians. These 3D Gaussians are used as input to the introduced Cascade Gaussian Adapter (CGA) and Iterative Gaussian Refiner (IGR) for further processing, yielding a set of updated 3D Gaussians aligned with the geometry complexity. Experiments on RealEstate10K and ACID demonstrate the effectiveness of PixelGaussian, showing state-of-the-art performance.
weaknesses: * Accessing the quality of depth map and extracted point cloud. As mentioned in L91, this work aims to “mitigate Gaussian overlap and redundancy”. However, this contribution cannot be justified by the RGB image-based results. It is crucial to access the 3D Gaussian quality by visualising the depth map and point cloud, similar to Fig. 4 in MVSplat. In particular, it would be better to compare with pixelSplat and MVSplat in terms of point cloud and depth map, especially using more input views, e.g., 4 input views. It is also suggested that similar comparisons be made among different ablated models, which would make it easier to understand the effectiveness of CGA and IGR.

* Accessing more input views. The significance of refining 3D Gaussians would be more obvious under the settings of more input views. Hence, it would be interesting to see how well the introduced PixelGaussian can perform using more input views, e.g., 12 views, 16 views, or even 32 views, similar to the concurrent work Long-LRM [Ziwen et. al, arXiv:2410.12781].

* Accessing more complex datasets. As shown in Fig. 4, the visual differences between PixelGaussian and other state-of-the-art methods are minor, possibly because RealEstate10K is too simple to demonstrate the effectiveness of Gaussian refinement. Hence, it is highly recommended to report additional comparisons on more complex datasets, e.g., MipNeRF360, Tanks and Temples.

* Accessing cross-dataset generalization. It would be interesting to see how well the introduced CGA and IGR can generalize to other datasets. For example, trained on RealEstate10K but tested on DTU, similar to Fig. 5 in MVSplat.

* The motivation of using Deformable Attention is unclear. It might be better to provide experiments with further analysis to justify why it is better to use Deformable Attention instead of typical attention blocks.

* The paper title and model name are confusing. It might be better to highlight in the title that this work predicts an adaptive quantity of 3D Gaussians. Besides, the model name ‘PixelGaussian’ might unintentionally imply ‘pixel-align Gaussian’, which is the opposite of the objective. It might be better to change it to ‘AdaptiveGaussian’ or some other name that better reflects the main contribution of this work.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1.Is it possible to provide details on the view selection process, including the angular separation between views and whether test views are interpolated between or extrapolated beyond the training views as well as the camera pose distribution for both training and testing sets.

2.Since the author argues their method can be trained on two views and tested on more views. Is it possible to show that such training strategies have better generalization ability for large scale/wide range scenes and compare results with those diffusion-based sparse view reconstruction methods including ReconFusion, ViewCrafter, ReconX following their experiments settings. Also give more discussions on such settings.
soundness: 3
strengths: 1.The paper proposes a generalizable gaussian prediction method with Cascade Gaussian Adapter (CGA) to dynamically adapt the distribution and quantity of Gaussians, which is effectively shown in the experiments part.

2.It further refines Gaussian representations via proposed Iterative Gaussian Refiner with deformable attention.

3.It shows with better distributed gaussians, the final rendering results are better than those methods with per-pixel predicted gaussians and better generalization when trained with 2 views and tested with 2-4 views.
summary: They propose a feed-forward framework 3D Gaussian reconstruction from arbitrary views. With propose CGA and IGR, they can produce well distributed gaussians base on geomtry complexity and generalize better to arbitrary views than other methods. They have validation on benchmarks to show the effectiveness of their method.
weaknesses: 1. While the paper demonstrates improved generalization with increased view numbers during inference through adjusted Gaussian distributions. However, it would make more sense by including results for baseline methods trained specifically on 2, 4, and 6 views.

2. The current experiments primarily showcase scenarios with substantial FOV overlap between input views. To better demonstrate the method's generalization capabilities, consider including more challenging test cases - for instance, evaluating a model trained on 2 views when tested with 2-6 widely-spaced views with minimal overlap. Such examples would validate the method's ability to handle challenging novel view synthesis scenarios.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. As shown in Table 1, the 3D-GS-based PixelGaussian demonstrates marginally improved performance over the NeRF-based MuRF, especially on the ACID benchmark. A more detailed analysis is required to explain the underlying reasons for this performance difference, training and inference efficiency, and memory usage.
2. RealEstate10K and ACID benchmarks were first officially introduced in pixelSplat, but I think the efficiency should be demonstrated in more benchmarks. I would like to see more comparisons in more diverse datasets, such as MuRF's DTU, LLFF, and Mip-NeRF 360 dataset, or even self-captured real-world via cameras or phones (see the teaser in Metric3D [1] and F2-nerf [2]).
3. I would like to see the full efficiency comparison across 2-6 views, in addition to only 4 views in Table 3.
4. The training efficiency comparison with MVSplat should also be analyzed with the same amount of training data and computing resources.
5. I think the 3D-GS adaptions in Fig. 6 and Fig. 1 are not visually-pleasing in the ICLR-level paper and can be drawn and demonstrated better.


[1] Metric3d: Towards zero-shot metric 3d prediction from a single image. ICCV 2023

[2] F2-nerf: Fast neural radiance field training with free camera trajectories. CVPR 2023
soundness: 3
strengths: 1. The proposed CGA and IGR modules are novel in the feed-forward 3D-GS scenarios.
2. The experiments effectively support the proposed method: Table 1 demonstrates visual performance compared to SOTAs, Table 2 illustrates scalability concerning input views, Table 3 highlights inference efficiency achieved through reduced 3D Gaussians, and Table 4 validates the effectiveness of the designed modules.
3. The paper is well-motivated and easy to understand and follow.
summary: PixelGaussian employs a dynamic mechanism to identify the region of interest within the underlying 3D spatial locations, and accordingly adjusts the 3D-GS distribution and quantity to enhance visual quality while reducing memory usage. Experiments on ACID and RealEstate10K validate the effectiveness of the proposed method.
weaknesses: Please see the "Questions" section.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the above weaknesses. If they are adequately addressed, I am willing to increase the rating.
soundness: 3
strengths: 1. Motivations are clear. PixelSplat and MVSplat indeed have problems when the number of views increase. This paper tackles this issue, which I think is very reasonable.

2. Ablation study is thorough and the performance of the proposed method is impressive.

3. Writing is good and easy to understand.
summary: This work first points out the unique issue posed by the use of pixel-aligned 3D Gaussians, which is the redundant 3D Gaussians depicting the same obejcts' surfaces. Due to this issue, existing works, such as MVSplat and PixelSplat, perform bad when the number of views increases, unlike the NeRF-based generalized NVS works, PixelNeRF, IBRNet or MVSNeRF, that benefit from more input views. To handle this issue, the authors propose so called CGA, to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scoring function. Subsequently, a transformer-based IGR is proposed to model interactions between image and 3D Gaussians to realize further refinements. The proposed method is evaluated on large-scale indoor/outdoor datasets and is shown to achieve new state-of-the-art.
weaknesses: 1. It seems that the performance of PixelGaussian is better in 2 view setting as well. Why is this? I understand that for more views, it will the proposed method will perform better than them, but i can not think of a convincing reason for the superior performance in 2 view setting. I am guessing that the model is trained on 8 A6000 GPUs, which incorporates larger effective batch size than MVSplat. So I would say it's not fair to compare directly with MVSplat, thus we can not be so sure that the proposed method would work also well with 2 views. It might even perform worse if trained with similar iterations and batch size. Would it be possible if the authors  provide a more detailed comparison of training conditions between PixelGaussian and MVSplat, including batch sizes, number of iterations, and computational resources used. Additionally, if PixelGaussian trained under similar conditions as MVSplat to ensure a fair comparison in the 2-view setting, what would be the results?

2. In Table 3, the authors highlight that the rendering FPS is higher than MVSPlat. This makes sense, since the # of Gaussians are less compared to MVSplat. However, Since the architecture of the proposed method builds on top of MVSplat, it will likely to infer slower overall, if both 3D reconstruction and nvs were accounted for, which is reported and supported by the latency. So I think the efficiency is not really a big contribution, **unless** the authors showed the differences in higher resolutions. Currently 256x 256 is too small to see the dramatic difference. But the current submission unfortunately does not have efficiency comparisons in higher resolutions, which could've been a strength of this work.  Would it be possible if the authors provide detailed comparisons at higher resolutions (e.g., 512x512, 1024x1024 or higher).  This could be this paper's strength if properly compared.

3. It would've been even more interesting if the paper explored the impacts of increasing the number of input views more and more. I would expect at some point, the performance will saturate, but it will be very interesting to see those. Would it be possible if the authors systematically increases the number of input views (e.g., from 2 to 8 or 10) and report how the performance changes, including where it potentially saturates?

Meta-review: This paper receives borderline ratings of 5,6,5,6. The AC look at the reviews, rebuttal and the discussions, and decide to reject the paper for now. The authors are encouraged to fix the issues and resubmit to future conference. The main concern raised by the reviewers is that the proposed method introduces extra overhead which lead to higher inference latency compared to MVSplat. Moreover, the performance is worse than MVSplat and rendering speed is similar to PixelGaussian when there are fewer than 4 views. These points suggest that the design is not sufficiently effective. Additionally, there are also concerns on the experiments such as more complex datasets, more input views, cross-generalization, etc. Although the two 6's are positive, there is no strong compelling reason to push for acceptance of the paper.

Decision: Reject