PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views

Authors: Xin Fei, Wenzhao Zheng, Yueqi Duan, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, Jiwen Lu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Gaussian splatting, 3D reconstruction, Novel view synthesis

Abstract:
We propose AdaptiveGaussian, an efficient feed-forward framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Most existing methods rely on uniform pixel-wise Gaussian representations, which learn a fixed number of 3D Gaussians for each view and cannot generalize well to more input views. Differently, our AdaptiveGaussian dynamically adapts both the Gaussian distribution and quantity based on geometric complexity, leading to more efficient representations and significant improvements in reconstruction quality. Specifically, we introduce a Cascade Gaussian Adapter (CGA) to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scorer. CGA leverages deformable attention in context-aware hypernetworks to guide Gaussian pruning and splitting, ensuring accurate representation in complex regions while reducing redundancy. Furthermore, we design a transformer-based Iterative Gaussian Refiner (IGR) module that refines Gaussian representations through direct image-Gaussian interactions. Our AdaptiveGaussian can effectively reduce Gaussian redundancy as input views increase. We conduct extensive experiments on the large-scale ACID and RealEstate10K datasets, where our method achieves state-of-the-art performance with good generalization to various numbers of views.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729880256872

thread_type:
official_review

<thread_object>
creator: reviewer_x5k2

<content>
summary:
This work first points out the unique issue posed by the use of pixel-aligned 3D Gaussians, which is the redundant 3D Gaussians depicting the same obejcts' surfaces. Due to this issue, existing works, such as MVSplat and PixelSplat, perform bad when the number of views increases, unlike the NeRF-based generalized NVS works, PixelNeRF, IBRNet or MVSNeRF, that benefit from more input views. To handle this issue, the authors propose so called CGA, to adjust Gaussian distribution according to local geometry complexity identified by a keypoint scoring function. Subsequently, a transformer-based IGR is proposed to model interactions between image and 3D Gaussians to realize further refinements. The proposed method is evaluated on large-scale indoor/outdoor datasets and is shown to achieve new state-of-the-art.

strengths:
1. Motivations are clear. PixelSplat and MVSplat indeed have problems when the number of views increase. This paper tackles this issue, which I think is very reasonable.

2. Ablation study is thorough and the performance of the proposed method is impressive.

3. Writing is good and easy to understand.

weaknesses:
1. It seems that the performance of PixelGaussian is better in 2 view setting as well. Why is this? I understand that for more views, it will the proposed method will perform better than them, but i can not think of a convincing reason for the superior performance in 2 view setting. I am guessing that the model is trained on 8 A6000 GPUs, which incorporates larger effective batch size than MVSplat. So I would say it's not fair to compare directly with MVSplat, thus we can not be so sure that the proposed method would work also well with 2 views. It might even perform worse if trained with similar iterations and batch size. Would it be possible if the authors  provide a more detailed comparison of training conditions between PixelGaussian and MVSplat, including batch sizes, number of iterations, and computational resources used. Additionally, if PixelGaussian trained under similar conditions as MVSplat to ensure a fair comparison in the 2-view setting, what would be the results?

2. In Table 3, the authors highlight that the rendering FPS is higher than MVSPlat. This makes sense, since the # of Gaussians are less compared to MVSplat. However, Since the architecture of the proposed method builds on top of MVSplat, it will likely to infer slower overall, if both 3D reconstruction and nvs were accounted for, which is reported and supported by the latency. So I think the efficiency is not really a big contribution, **unless** the authors showed the differences in higher resolutions. Currently 256x 256 is too small to see the dramatic difference. But the current submission unfortunately does not have efficiency comparisons in higher resolutions, which could've been a strength of this work.  Would it be possible if the authors provide detailed comparisons at higher resolutions (e.g., 512x512, 1024x1024 or higher).  This could be this paper's strength if properly compared.

3. It would've been even more interesting if the paper explored the impacts of increasing the number of input views more and more. I would expect at some point, the performance will saturate, but it will be very interesting to see those. Would it be possible if the authors systematically increases the number of input views (e.g., from 2 to 8 or 10) and report how the performance changes, including where it potentially saturates?

questions:
Please see the above weaknesses. If they are adequately addressed, I am willing to increase the rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730092170047

thread_type:
official_review

<thread_object>
creator: reviewer_z7ms

<content>
summary:
PixelGaussian employs a dynamic mechanism to identify the region of interest within the underlying 3D spatial locations, and accordingly adjusts the 3D-GS distribution and quantity to enhance visual quality while reducing memory usage. Experiments on ACID and RealEstate10K validate the effectiveness of the proposed method.

strengths:
1. The proposed CGA and IGR modules are novel in the feed-forward 3D-GS scenarios.
2. The experiments effectively support the proposed method: Table 1 demonstrates visual performance compared to SOTAs, Table 2 illustrates scalability concerning input views, Table 3 highlights inference efficiency achieved through reduced 3D Gaussians, and Table 4 validates the effectiveness of the designed modules.
3. The paper is well-motivated and easy to understand and follow.

weaknesses:
Please see the "Questions" section.

questions:
1. As shown in Table 1, the 3D-GS-based PixelGaussian demonstrates marginally improved performance over the NeRF-based MuRF, especially on the ACID benchmark. A more detailed analysis is required to explain the underlying reasons for this performance difference, training and inference efficiency, and memory usage.
2. RealEstate10K and ACID benchmarks were first officially introduced in pixelSplat, but I think the efficiency should be demonstrated in more benchmarks. I would like to see more comparisons in more diverse datasets, such as MuRF's DTU, LLFF, and Mip-NeRF 360 dataset, or even self-captured real-world via cameras or phones (see the teaser in Metric3D [1] and F2-nerf [2]).
3. I would like to see the full efficiency comparison across 2-6 views, in addition to only 4 views in Table 3.
4. The training efficiency comparison with MVSplat should also be analyzed with the same amount of training data and computing resources.
5. I think the 3D-GS adaptions in Fig. 6 and Fig. 1 are not visually-pleasing in the ICLR-level paper and can be drawn and demonstrated better.


[1] Metric3d: Towards zero-shot metric 3d prediction from a single image. ICCV 2023

[2] F2-nerf: Fast neural radiance field training with free camera trajectories. CVPR 2023
</content>
</thread_object>

</thread>
<thread>
creation date:
1730108466655

thread_type:
official_review

<thread_object>
creator: reviewer_hfot

<content>
summary:
They propose a feed-forward framework 3D Gaussian reconstruction from arbitrary views. With propose CGA and IGR, they can produce well distributed gaussians base on geomtry complexity and generalize better to arbitrary views than other methods. They have validation on benchmarks to show the effectiveness of their method.

strengths:
1.The paper proposes a generalizable gaussian prediction method with Cascade Gaussian Adapter (CGA) to dynamically adapt the distribution and quantity of Gaussians, which is effectively shown in the experiments part.

2.It further refines Gaussian representations via proposed Iterative Gaussian Refiner with deformable attention.

3.It shows with better distributed gaussians, the final rendering results are better than those methods with per-pixel predicted gaussians and better generalization when trained with 2 views and tested with 2-4 views.

weaknesses:
1. While the paper demonstrates improved generalization with increased view numbers during inference through adjusted Gaussian distributions. However, it would make more sense by including results for baseline methods trained specifically on 2, 4, and 6 views.

2. The current experiments primarily showcase scenarios with substantial FOV overlap between input views. To better demonstrate the method's generalization capabilities, consider including more challenging test cases - for instance, evaluating a model trained on 2 views when tested with 2-6 widely-spaced views with minimal overlap. Such examples would validate the method's ability to handle challenging novel view synthesis scenarios.

questions:
1.Is it possible to provide details on the view selection process, including the angular separation between views and whether test views are interpolated between or extrapolated beyond the training views as well as the camera pose distribution for both training and testing sets.

2.Since the author argues their method can be trained on two views and tested on more views. Is it possible to show that such training strategies have better generalization ability for large scale/wide range scenes and compare results with those diffusion-based sparse view reconstruction methods including ReconFusion, ViewCrafter, ReconX following their experiments settings. Also give more discussions on such settings.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696561667

thread_type:
official_review

<thread_object>
creator: reviewer_bydc

<content>
summary:
This work introduces PixelGaussian, a feed-forward 3DGS model that can adaptively update the number of Gaussian via pruning and splitting. In particular, PixelGaussian starts by predicting the pixel-aligned 3D Gaussians. These 3D Gaussians are used as input to the introduced Cascade Gaussian Adapter (CGA) and Iterative Gaussian Refiner (IGR) for further processing, yielding a set of updated 3D Gaussians aligned with the geometry complexity. Experiments on RealEstate10K and ACID demonstrate the effectiveness of PixelGaussian, showing state-of-the-art performance.

strengths:
* The motivation is reasonable, emphasising the significance of updating the predicted pixel-aligned 3D Gaussians under more input views.
* Experiments on two existing benchmarks verify the efficacy of the introduced PixelGaussian

weaknesses:
* Accessing the quality of depth map and extracted point cloud. As mentioned in L91, this work aims to “mitigate Gaussian overlap and redundancy”. However, this contribution cannot be justified by the RGB image-based results. It is crucial to access the 3D Gaussian quality by visualising the depth map and point cloud, similar to Fig. 4 in MVSplat. In particular, it would be better to compare with pixelSplat and MVSplat in terms of point cloud and depth map, especially using more input views, e.g., 4 input views. It is also suggested that similar comparisons be made among different ablated models, which would make it easier to understand the effectiveness of CGA and IGR.

* Accessing more input views. The significance of refining 3D Gaussians would be more obvious under the settings of more input views. Hence, it would be interesting to see how well the introduced PixelGaussian can perform using more input views, e.g., 12 views, 16 views, or even 32 views, similar to the concurrent work Long-LRM [Ziwen et. al, arXiv:2410.12781].

* Accessing more complex datasets. As shown in Fig. 4, the visual differences between PixelGaussian and other state-of-the-art methods are minor, possibly because RealEstate10K is too simple to demonstrate the effectiveness of Gaussian refinement. Hence, it is highly recommended to report additional comparisons on more complex datasets, e.g., MipNeRF360, Tanks and Temples.

* Accessing cross-dataset generalization. It would be interesting to see how well the introduced CGA and IGR can generalize to other datasets. For example, trained on RealEstate10K but tested on DTU, similar to Fig. 5 in MVSplat.

* The motivation of using Deformable Attention is unclear. It might be better to provide experiments with further analysis to justify why it is better to use Deformable Attention instead of typical attention blocks.

* The paper title and model name are confusing. It might be better to highlight in the title that this work predicts an adaptive quantity of 3D Gaussians. Besides, the model name ‘PixelGaussian’ might unintentionally imply ‘pixel-align Gaussian’, which is the opposite of the objective. It might be better to change it to ‘AdaptiveGaussian’ or some other name that better reflects the main contribution of this work.

questions:
Kindly refer to [Weaknesses].
</content>
</thread_object>

</thread>
<thread>
creation date:
1732554149596

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal

comment:
We thank all the reviewers for their valuable feedbacks. In this general rebuttal session, we try to address the common questions for each reviewer and we will upload the revised paper shortly.

As requested by Reviewer bYDc, z7ms, and X5K2, we further evaluate our model under a broader range of input view settings on both the RealEstate10K and DTU datasets, to provide a more comprehensive assessment of its performance across different scenarios. 

We evaluate our model, trained on the RealEstate10K dataset, for inference on both the RealEstate10K and DTU datasets. For each scene, we select the reference view to cover as wide a view range as possible to compare NVS results on such challenging cases. Due to memory limitations, we evaluate our model with a maximum of 16 input views and we split the inputs into batches to get Gaussians for pixelSplat on the 16-view setting.

Table 1: PSNR comparison of multiple input view settings on challenging cases in RealEstate10K dataset
|          | pixelSplat | MVSplat | PixelGaussian |
|----------|----------- |---------|---------------|
| 4 views | 22.02 | 22.30 | 23.95 |
| 8 views | 19.97 | 20.39 | 24.05 |
| 12 views | 18.92 | 19.69 | 24.18 |
| 16 views | 18.90 | 19.40 | 24.24 |

Table 2: PSNR comparison of multiple input view settings for cross-dataset generalization on DTU dataset
|          | pixelSplat | MVSplat | PixelGaussian |
|----------|------------|---------|---------------|
| 4 views | 12.30 | 12.45 | 13.42 |
| 8 views | 11.94 | 12.10 | 13.46 |
| 12 views | 11.75 | 11.78 | 13.53 |
| 16 views | 11.47 | 11.55 | 13.56 |

From the results, we observe that PixelGaussian slightly benefits from increasing input views in challenging cases while the performance of pixelSplat and MVSplat drops dramatically due to the overlap and redundancy of pixel-aligned Gaussians. Moreover, our model is able to maintain this advantage in cross-dataset generalization. 

Please feel free to reach out if you have any follow-up questions or need further clarification. Your feedback is valuable to us, and we are happy to provide any additional information that may be helpful.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives borderline ratings of 5,6,5,6. The AC look at the reviews, rebuttal and the discussions, and decide to reject the paper for now. The authors are encouraged to fix the issues and resubmit to future conference. The main concern raised by the reviewers is that the proposed method introduces extra overhead which lead to higher inference latency compared to MVSplat. Moreover, the performance is worse than MVSplat and rendering speed is similar to PixelGaussian when there are fewer than 4 views. These points suggest that the design is not sufficiently effective. Additionally, there are also concerns on the experiments such as more complex datasets, more input views, cross-generalization, etc. Although the two 6's are positive, there is no strong compelling reason to push for acceptance of the paper.