Latent Radiance Fields with 3D-aware 2D Representations

Authors: Chaoyi Zhou, Xi Liu, Feng Luo, Siyu Huang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Gaussian Splatting, 3D-aware Representation

Abstract:
Latent 3D reconstruction has shown great promise in empowering 3D semantic understanding and 3D generation by distilling 2D features into the 3D space. However, existing approaches struggle with the domain gap between 2D feature space and 3D representations, resulting in degraded rendering performance. To address this challenge, we propose a novel framework that integrates 3D awareness into the 2D latent space. The framework consists of three stages: (1) a correspondence-aware autoencoding method that enhances the 3D consistency of 2D latent representations, (2) a latent radiance field (LRF) that lifts these 3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field (VAE-RF) alignment strategy that improves image decoding from the rendered 2D representations. Extensive experiments demonstrate that our method outperforms the state-of-the-art latent 3D reconstruction approaches in terms of synthesis performance and cross-dataset generalizability across diverse indoor and outdoor scenes. To our knowledge, this is the first work showing the radiance field representations constructed from 2D latent representations can yield photorealistic 3D reconstruction performance.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729927499775

thread_type:
official_review

<thread_object>
creator: reviewer_cwtt

<content>
summary:
This paper target at Latent 3D reconstruction, and address the domain gap between 2D feature space and 3D representations. They proposed propose a novel framework that comprise (1) a correspondence-aware autoencoding method, (2) a latent radiance field (LRF), and (3) a VAE-Radiance Field (VAE-RF) alignment strategy.

strengths:
Strengths*:
With the proposeld framework, this paper enhances the 3D consistency of 2D latent representations as well as effectively mitigated the gap between the 2D latent space and the natural 3D space.

weaknesses:
Weaknesses*:

1. Compared with feature-GS, it is a good improvement to add a correspondence-aware constraint during VAE encoder finetuning to improve its 3D awareness. However, this approach still cannot guarantee strict multi-view consistency of the encoded multiview features. As a result, after constructing the LRF, there may be a blurred radiance field with significant detail losses. Although the LRF is 3D consistent, the final decoded features may still exhibit noticeable flickering effects due to the lack of view consistency of decoder.


2. I think this paper still needs optimization during the LRF stage instead of using a total feed forward method. When compared with 3DGS and Mip-Splatting, the authors only train them in latent space resolution (8 times lower than the image resolution), which yields very bad visual results. I suggest that the authors consider training the competing methods at full resolution. This paper may not work better than Mip-Splatting when trained with full resolution and very dense view, but it would be interesting to see if it outperforms Mip-Splatting in a sparse-view setting. This might be achieved because of generation capability of Stable Diffusion VAE used in this paper.

3. The author didn't provide a detailed explanation of experiment settings. For example, I wonder how views are sampled in the experiment during training. And how many views are used as input and evaluation repectively. This is important for me to fully evaluate this paper. 

4. Object change in final rendering. As shown in Fig 1, the building in the final rendering is different from the ground truth image. One is in red, the other one is in white. This lead to my concern of identity preserving capability of purposed method. I think this is a problem that needs to be addressed.

questions:
I think the strict view-inconsistency can not be solved fundamentally due to the emplyment of VAE Decoder. But it coule be possible to showcase more view consistent results and evlaute the 3D consistency of proposeld methods. 

The author provide more details about the experiment settings, especially the view sampling strategy and the number of views used in the experiment. I hope see whether the proposed method can be applied to a sparse-view setting.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537067428

thread_type:
official_review

<thread_object>
creator: reviewer_rdan

<content>
summary:
This paper introduces a framework for constructing radiance field representations in latent space, aiming to bridge the domain gap between 2D feature space and 3D representations. The authors propose a three-stage pipeline: (1) a correspondence-aware autoencoding method that enforces 3D consistency in latent space through correspondence constraints, (2) a latent radiance field (LRF) that lifts these 3D-aware 2D representations into 3D space, and (3) a VAE-Radiance Field alignment strategy that improves image decoding from rendered 2D representations.

The key technical contribution is the integration of 3D awareness into 2D representation learning without requiring additional per-scene refinement modules. The authors adapt the 3D Gaussian Splatting framework to operate in latent space, using spherical harmonics to model view-dependent effects. They demonstrate their method's effectiveness on both novel view synthesis and text-to-3D generation tasks across various datasets including MVImgNet, NeRF-LLFF, MipNeRF360, and DL3DV-10K. The authors claim their approach is the first to achieve photorealistic 3D reconstruction performance directly from latent representations while maintaining cross-dataset generalizability.

The work represents an attempt to make latent 3D reconstruction more practical by addressing the geometric consistency issues in existing approaches. The framework is designed to be compatible with existing novel view synthesis and 3D generation pipelines without requiring additional fine-tuning.

strengths:
- The paper follows a standard pipeline structure addressing latent space 3D reconstruction. The method section breaks down into three components: correspondence-aware encoding, latent radiance field construction, and VAE alignment. The ablation study provides basic validation of these components, though more comprehensive analysis would be beneficial.
- While building heavily on existing techniques, the paper demonstrates competent engineering in combining different elements into a working system. The adaptation of correspondence constraints and 3DGS to latent space shows reasonable technical implementation. The provided implementation details outline the basic approach.
- The evaluation includes tests on multiple datasets (MVImgNet, NeRF-LLFF, MipNeRF360, DL3DV-10K), attempting to demonstrate applicability across different scenarios. While the cross-dataset evaluation has limitations, it provides basic evidence of generalization capability. The inclusion of both novel view synthesis and text-to-3D generation shows the method's potential utility, though more thorough evaluations are needed.
- The method functions without per-scene refinement modules, which could be advantageous compared to some previous approaches.

weaknesses:
- The paper fails to provide compelling justification for operating in latent space. While previous works like Latent-NeRF (for text-to-3D generation) established initial groundwork, this paper does not clearly demonstrate additional benefits of its approach. The motivation for operating in latent space remains questionable. The paper shows modest improvements in PSNR/SSIM metrics but doesn't address fundamental questions: What are the computational advantages over image-space methods? How does memory consumption compare? Why is the added complexity of latent space operations justified? The authors should conduct a thorough efficiency analysis, measuring training time, inference speed, and memory usage against image-space baselines. Without such evidence, the practical value of the latent space approach is hard to justify.
- Section 4.1's correspondence mechanism has several fundamental issues. Most critically, the paper fails to address the scale mismatch between COLMAP's pixel-level correspondences and the VAE's latent space. Given that the VAE operates at a lower resolution (likely 8x or 16x downsampled) with larger receptive fields, how are pixel-level correspondences meaningfully mapped to latent features? This mapping is non-trivial: a single latent code typically corresponds to a large receptive field in pixel space, making precise correspondence matching questionable. The paper should answer: How are multiple pixel correspondences within one latent cell handled? How does the receptive field size affect correspondence accuracy? Additionally, basic details are missing: the correspondence filtering criteria, quality metrics, and robustness to matching errors. The use of L1 distance for latent features (Eq. 6) needs justification, especially given the coarse nature of latent correspondences. These technical gaps raise serious concerns about the method's fundamental soundness.
- The use of spherical harmonics in latent space (Eq. 8) is puzzling. Given that the features are already in a learned latent space, why introduce SH basis functions? A direct learnable decoder or simpler view-dependent representation might suffice. Similarly, the VAE-RF alignment stage seems unnecessarily complex - the authors may quantify the alleged distribution shift and explore simpler alternatives. These design choices add complexity without clear benefits.
- The experimental setup has a fundamental flaw: image-space methods are handicapped by low-resolution inputs while the proposed method has access to high-resolution data. This creates an artificial advantage for the proposed method. A fair comparison requires either: testing at matched resolutions, or demonstrating specific benefits under computational constraints. The ablation study skips crucial experiments on correspondence quality, loss function components, and architectural variations. These missing comparisons make it difficult to assess the true value of each component.
- The paper sidesteps important practical concerns. Where are the failure cases? How does the method handle challenging scenes with varying illumination or complex geometry? The text-to-3D generation results lack comparisons with current state-of-the-art methods. The claim of "photorealistic reconstruction" needs validation through proper user studies or established perceptual metrics. Testing on more diverse, challenging scenarios would better demonstrate real-world applicability.

questions:
- How to handle the scale mismatch between COLMAP and latent space? Please clarify: 1) the exact mapping strategy from pixel to latent correspondences; 2) how multiple pixel correspondences within one latent cell are aggregated
- What's the correspondence filtering pipeline? In particular: 1) thresholds used for COLMAP matching 2) any additional filtering criteria in latent space 3) how outlier correspondences are handled.
- During VAE-RF alignment (Section 4.3), how to: 1) balance the training/novel view losses 2) prevent overfitting during decoder fine-tuning.
- Regarding the resolution setup, why choose this specific resolution comparison protocol? Would the method's advantages hold at equal resolutions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619677199

thread_type:
official_review

<thread_object>
creator: reviewer_ntxk

<content>
summary:
The author introduces pixel-to-pixel correspondences across different viewpoints to help the VAE learn a 2D latent space with 3D awareness. Using 3D Gaussian Splatting (3DGS), they perform 3D reconstruction and rendering in the latent space to obtain 2D latent representations from specified camera poses. The rendered results are then decoded back into image space by the decoder to obtain RGB images.

Experimental results demonstrate that the resulting 2D latent space possesses a certain level of 3D perception capability and outperforms existing methods when decoding to higher-resolution images.

strengths:
The author is committed to integrating 3D awareness into the 2D latent space, and the results show a significant degree of success in this endeavor. Additionally, using 3D Gaussian Splatting (3DGS) in modeling the latent space is an intriguing idea.

weaknesses:
The motivation of this paper is somewhat unclear. Is the author aiming to improve reconstruction accuracy, enhance rendering speed, reduce storage space, or achieve some other application? It appears that none of these goals have been fully addressed.

**Reconstruction Accuracy**: When training the comparison methods, the author down-scaled the RGB images to the same resolution as the latent representation before training, which may be considered unfair. The VAE used by the author has been exposed to high-resolution images, while the comparison methods have not. This discrepancy could reduce the reconstruction performance of the comparison methods and impact the paper's credibility.

**Rendering Speed**: In Section 5.1, the author reports the training times for Stage 1 and Stage 3, but not for Stage 2 or for inference time. Therefore, it is challenging to conclude that the proposed method has a faster rendering or training speed compared to other methods.

**Storage Space Reduction**: In Section 5.1, the author mentions the need to "train the same number of latent 3D Gaussian splatting scenes... for Stage-III," indicating that Stage 2 is scene-specific. Compared to 3DGS, this does not seem to save much storage space.

**Other Applications**: In Section 5.3, the author suggests that their work can be used for text-to-3D generation. However, the two methods used for comparison are relatively outdated. It is recommended to compare the method with more recent approaches, such as IPDREAMER [1], to make the claim more convincing.

[1] Zeng, Bohan, et al. "Ipdreamer: Appearance-controllable 3d object generation with image prompts." arXiv preprint arXiv:2310.05375 (2023).

questions:
- Could the author please provide further clarification on the motivation and applicable scenarios for this work? 
- In the experimental section, it would be beneficial to employ fairer comparison methods; using low-resolution training for comparison models is not advisable. 
- Please consider using more recent models to compare the text-to-3D generation capabilities of this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666990632

thread_type:
official_review

<thread_object>
creator: reviewer_duk5

<content>
summary:
This submission tries to resolve the problem of 3D reconstruction in the latent space via a 3-stage idea. The first stage focuses on improving the 3D awareness of the VAE's encoder via a correspondence-aware constraint on the latent space; the second stage builds a latent radiance field (LRF) to represent 3D scenes from the 3D-aware 2D representations; the last stage further introduces a VAE-Radiance Field (VAE-RF) alignment method to boost the reconstruction performance. The results generated by this pipeline out-perform the results generated by many state-of-the-arts.

strengths:
There are many innovations in this work, but I think the best part is the introduction of the 3d awareness into the 2D representation training. In this part, especially the correspondence aware autoencoding is the key to the success of this overall idea.

weaknesses:
There are still some weaknesses prevented me from giving a higher score, especially, the details of how to compute each component of the pipeline. Please see my questions below. In addition, some related references are missing.

questions:
I'm willing to raise my score if questions below is answered:

1. How is \lambda_{ij} computed in equation (6), section 4.1? Basically, how to compute the average pose error and how it contributes to the weight \lambda_{ij}?
2. Since equation (6) becomes multi-objective optimization, does this change largely increase the training convergence time? Did you experience any convergence issues?
3. How is the inference speed of this pipeline?
4. The idea is kind of similar to CaesarNeRF: Calibrated Semantic Representation for Few-shot Generalizable Neural Rendering: https://haidongz-usc.github.io/project/caesarnerf, which also uses calibrated image feature in each 2d latent, could you cite and compare?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732164111727

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their time, effort, and valuable feedback, which have greatly helped us to improve this work. We would like to take this opportunity to clarify the motivations of this work, and three major experiments added during rebuttal period. 

**Motivation:**
This work aims to *bridge the gap between 2D and 3D representations, providing an efficient and generalizable solution for tasks requiring multi-view consistency in the feature space*. Targeting at this, there are two key challenges:
- A key challenge is that *2D latent features often lack 3D view consistency*, which impairs the performance of novel view synthesis (NVS). To address this. We introduce a correspondence-aware autoencoding fine-tuning strategy that enforces multi-view consistency in 2D latent features. This strategy uses a correspondence loss to align 2D latent representations with RGB images by ensuring shared corresponding points, minimizing modifications to the 2D latent space while achieving geometric consistency. 
- Another key challenge is that, *applying RGB-based NVS method to latent features would cause data distribution shift* (see below table). We mitigate this through an alignment fine-tuning process that uses latent radiance fields (LRFs) to fine-tune decoder. This process corrects distribution shifts by training on paired datasets, leading to improved latent representation fidelity and significant PSNR gains, as shown in Table 2 and Figure 6 of the paper.


| KL Loss (Before 3D reconstruction)  | KL Loss (After 3D reconstruction)  |
|------------------------------|----------------------------|
| $3.35*10^4$                    |          $1.32*10^5$       |

 **A comprehensive comparison of running efficiency and performance**: Our method reduces input resolutions, model storage space, and GPU usage for photorealistic NVS, which is particularly useful in cases with limited communication bandwidth and storage. For instance, some individuals may not have GPUs with large memories, where our method is an efficient solution for them to run photorealistic NVS algorithms. 

| Method           | Input resolution to 3D model | Training Time &darr; | GPU Usage &darr; | Storage &darr;       | Rendering FPS &uarr; | Decoding FPS &uarr; | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; |
|------------------|-------------------------|-----------------------|------------------|----------------------|----------------------|---------------------|-------------|-------------|--------------|
| *3DGS*           | *512\*512*              | *5.9 min*             | *3 GB*           | *200.41 MB*          | *100*                | *-*                 | *26.17*     | *0.778*     | *0.091*      |
| 3DGS/8           | 64*64                  | **3.1 min**               | 1 GB             | **59.15 MB**         | **200**              | -                   | 14.03        | 0.352       | 0.541        |
| 3DGS-VAE         | 64*64                  | 4.8 min               | 2 GB             | 250.97 MB            | 80                   | 20                  | 20.57       | 0.595       | 0.346        |
| Latent-NeRF      | 64*64                  | 27.2 min              | 10 GB            | 350.50 MB            | 0.09                 | 20                  | 18.16       | 0.530       | 0.432        |
| Ours             | 64*64                  | 3.9 min           | **1 GB**         | 96.42 MB             | 180                  | 20                  | **22.45**       | **0.667**       | **0.197**        |



**Sparse view reconstruction**: Thanks to reviewer Cwtt's insightful comment, we are excited to find that the improved 3D feature consistency enables better reconstruction of missing information for few-shot NVS task. Our method is even better than the state-of-the-art 3DGS methods with standard input image resolutions. The results are shown in the below table. The experiment is conducted on the LLFF dataset with 3 input views. 

| Method (LLFF, 3 views)                                      | PSNR &uarr; | SSIM &uarr; | LPIPS &darr; |
|-------------------------------------------|-------------|-------------|--------------|
| 3DGS-VAE                                  | 13.06       | 0.283       | 0.5708       |
| 3DGS (standard resolution)                | 13.79       | 0.331       | 0.468        |
| Mip-Splatting (standard resolution)       | 13.70       | 0.315       | 0.486        |
| Ours                                      | **15.51**   | **0.379**   | **0.465**    |

**More recent text-to-3D generation**: We have conducted comparison with more recent text-to-3D generation method, as shown in Sec. 5.3 of the revised paper. As shown in Fig. 5, our method can boost the performance under extremely complicated text prompts, and achieve complex geometry while preserving the multi-view consistency. Our method achieves consistent generation across views.

 Please refer to our comments for each reviewer for more detailed responses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces an innovative approach that integrates 3D awareness into 2D latent space representations, specifically through a correspondence-aware autoencoding mechanism. This integration is aimed at enhancing the 3D consistency and the natural interaction between 2D latent and 3D spaces. The authors then build LRF in latent space and decode it to images.  

***Strengths:***
- The method's integration of 3D awareness into 2D latent space, which is viewed as innovation enhancing the model's performance.
- Effective adaptation of various existing techniques such as 3DGS and correspondence constraints to improve 3D consistency in latent space.
- Testing across multiple datasets demonstrates the generalization capability and potential usage in different scenarios including novel view synthesis and text-to-3D generation.

***Weaknesses:***
- The paper's motivation and specific goals (such as improvement in reconstruction accuracy, rendering speed, or storage space reduction) are not clearly defined or convincingly addressed.
- Some reviewers noted the lack of detail in various components of the pipeline.
- Questions about the computational advantages or the justification for added complexity in operating within latent space

The authors have provided detailed rebuttal with further experiments and explanation. After careful consideration and discussion, we are pleased to inform this paper is accepted. The decision to accept the paper is from following considerations:
- Novel approach to incorporating 3D awareness into 2D latent space
- Despite some limitations, the paper provides expriments on multiple datasets, showing better generalization and potential for practical applications.
- The authors effectively addressed many of the initial concerns raised during the rebuttal period.

Overall, while the paper has some areas that could benefit from further refinement, its strengths in innovation, and empirical validation are sufficient to merit acceptance. The paper provides a way to think about and integrate 3D data within 2D latent frameworks and we think there might be potential for future direction. The authors are suggeted to carefully polish the paper, from motivation presentation to advantage demonstration, based on the reviewers' comments.