Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder

Authors: Antoine Schnepf, Karim Kassab, Jean-Yves Franceschi, Laurent Caraffa, Flavian Vasile, Jeremie Mary, Andrew I. Comport, Valerie Gouet-Brunet
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Latent NeRF, NeRF, Autoencoder, Inverse Graphics, Nerfstudio, 3D

Abstract:
While pre-trained image autoencoders are increasingly utilized in computer vision, the application of inverse graphics in 2D latent spaces has been under-explored. Yet, besides reducing the training and rendering complexity, applying inverse graphics in the latent space enables a valuable interoperability with other latent-based 2D methods. The major challenge is that inverse graphics cannot be directly applied to such image latent spaces because they lack an underlying 3D geometry. In this paper, we propose an Inverse Graphics Autoencoder (IG-AE) that specifically addresses this issue. To this end, we regularize an image autoencoder with 3D-geometry by aligning its latent space with jointly trained latent 3D scenes. We utilize the trained IG-AE to bring NeRFs to the latent space with a latent NeRF training pipeline, which we implement in an open-source extension of the Nerfstudio framework, thereby unlocking latent scene learning for its supported methods. We experimentally confirm that Latent NeRFs trained with IG-AE present an improved quality compared to a standard autoencoder, all while exhibiting training and rendering accelerations with respect to NeRFs trained in the image space. Our project page can be found at https://ig-ae.github.io .

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730041674631

thread_type:
official_review

<thread_object>
creator: reviewer_irnm

<content>
summary:
This paper introduces an Inverse Graphics Autoencoder (IG-AE) to explore the potential of inverse graphics in 2D latent spaces, which is relatively underutilized in computer vision. The authors highlight that inverse graphics applied in latent spaces can reduce both training and rendering complexity while enabling compatibility with other 2D latent-based methods. A key challenge addressed is that traditional image latent spaces lack 3D geometry, preventing direct application of inverse graphics. IG-AE addresses this by aligning the latent space of an image autoencoder with jointly trained latent 3D scenes. This enables a "latent NeRF" training pipeline, implemented within the Nerfstudio framework, making latent scene learning accessible for methods supported by the framework. Experiments show that latent NeRFs trained with IG-AE offer improved quality compared to conventional autoencoders, while also providing faster training and rendering compared to NeRFs trained in image space.

strengths:
1. **Novel Use of Inverse Graphics in Latent Space**: The authors claim that they are the first to explore this direction, which explores the relatively untapped area of applying inverse graphics in 2D latent spaces, which reduces training and rendering complexity and offers compatibility with other latent-based 2D methods.

2. **Integration of 3D Geometry into Latent Spaces**: The authors address the issue of the lack of 3D geometry in standard image latent spaces by regularizing the autoencoder with 3D information, aligning its latent space with jointly trained 3D scenes. This approach enhances the representation capability of latent spaces for 3D scene understanding. From many perspectives, these efforts will be meaningful in this area.

3. **Latent NeRF Training Pipeline**: Two new training pipelines are proposed. By creating a latent NeRF training pipeline and integrating it within the Nerfstudio framework, the paper unlocks efficient latent scene learning and makes it accessible to a broader set of methods, providing a practical tool for further research.

4. **Open-Source Contribution**: The open-source extension to the Nerfstudio framework adds value to the research community by facilitating reproducibility and further experimentation, and the proposed method can naturally work on any new NeRF representations.

weaknesses:
1. **The writing is pretty hard to follow.** I tried to understand this paper by reading over and over again, but still find it very hard to follow. Since this direction is relatively new, I strongly suggest the author to revise the writing in the later version for clearer elaboration.

2. **The motivation of the proposed method is unclear.** From my understanding, the proposed method try to align the NeRF rendering to a pretrained autoencoder (Ostris KL-f8-d16 VAE here), but what are the benefits of doing this? Though promising PSNR/LPIPS metrics are achieved as in Tab. 1 and Tab. 2, does the proposed IG-AE / AE supports zero-shot novel view synthesis on the imagenet dataset? Regarding Tab. 2, I don't quite understand how the author evaluate the novel view synthesis pipeline here, since no visual results are included in the main paper. All the visual results are just reconstructing the input view.

3. **Lack of comparison with existing methods**. Though the author claims this idea is quite novel and they are among the first to propose this solution, many existing methods have already shown similar spirits, including 

* SRT: Scene Representation Transformer.
* NeRF-VAE: A Geometry Aware 3D Scene Generative Model
* LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation

Their 3D (V)AE models support 3D view synthesis on the open vocabulary single-view inputs. The discussions / comparisons with these methods are needed to demonstrate the soundness of the proposed pipeline.

questions:
1. In section 3, is the same VAE used as in section 4? The implementation details have not mentioned this.
2. How is the "3D aligned" auto-encoder useful in the downstream tasks?
3. In Tab. 2, how is the NVS task performed? Given a view as the input to your encoder, then synthesis novel views directly? Would you include more visual results?
4. In your AE/VAE design, the latent space is still a "3D-aware" image, so why not using a truly 3D-aware latent such as the latent triplane in LN3Diff / Direct3D?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730275427304

thread_type:
official_review

<thread_object>
creator: reviewer_jlb8

<content>
summary:
This paper propose a two-stage pipeline to train a latent space which is suitable for fitting NeRFs. In the first stage, a set of latent tri-planes are individually trained for each scene in the training data. The latent tri-planes are fitted by minimizing both the volumetric rendering error in latent space and the pixel difference in RGB space through a image decoder. In the second stage, an autoencoder is trained on all scenes in the training data, jointly with all individual tri-planes. The fidelity of the autoencoder is additionally regularized by training on additional real images. Experiments demonstrated that the proposed latent space is able to fitting NeRFs better than a image-based latent space.

strengths:
- The paper attempts to research on an important problem of 3D-aware latent spaces.

- The proposed method overall makes sense: the introduce of individually trained latent tri-planes provides a auxiliary variable which serves as a "3D-aware" guidance for auto-encoded latents. 

- Experiments show that the proposed IG-AE is better for training NeRFs than vanilla AE.

- The method can easily integrated into NeRFStudio with an open-source extension.

weaknesses:
My majority concerns are as follows:

(1) While the proposed method is interesting and makes sense, the claimed property of "3D-aware" latent space cannot be fully justified from the given experiment results:
- (a) The proposed method is only tested on dataset with limited variations. For evaluation NeRFs, ShapeNet dataset is not a best choice as it contains simple shapes and textures without any non-Lambertian effects. Additionally, the test dataset contains only three categories for Shapenet and Objeverse dataset. Evaulations on more complicated dataset with non-lambertian effects, such as Realistic Synthetic 360 [1] would help to more thoroughly demonstrate the "3D-aware" property of the latent space across a wider range of scenarios and make the results more convincing.
- (b) Even under the limited test dataset, the performance of the proposed method are not convincing enough. Despite the IG-AE outperforms the vanilla AE counterpart, the quality degrades significantly compares to the RGB version. This raises the question that whether the image fidelity is enough for the IG-AE.
- (c) The "3D-aware" property cannot be easily judged from given metrics and results in the paper. All quantitative results are averaged across views - it is hard to know whether the "consistency" between different synthesized views are preserved. One suggestion would be using some perceptal metrics to evaluate the consistency between different generated view, e.g., the CLIP feature similarity used in [2] and [3]. There are also very few qualitative results provided. Providing more qualitative results such as videos showing a rotated object can be also useful to visualize the view consistency.
- (d) The effect of the original auto-encoder utilized for training IG-AE is not well explored. As discussed in Section 3.2 in [4], a latent space with low channel-wise depth (e.g., down to 4) and a slightly higher (but still significantly lower than RGB input resolution, 32 or 64 for example) will encourage local dependency over the autoencoder’s image and latent spaces. Under such circumstance, the latent representation is a near patch level representation of its corresponding RGB image, making it nearly equivariant to spatial transformations of the scene. In other words, it automatically (at least to some extent) has the property of "3D-aware" for training NeRFs. I would suggest the paper add more discussion and/or experiments regarding the 2D autoencoder used.

(2) There has been many attempts for training NeRFs (or similar implicit 3D representations) in a space with reduced resolution, followed with upsampler operations in 2D space. For example:
- Rodin [5] first generates a low-resolution tri-planes with 3D aware convolutions and an upsampler. 
- StyleNeRF [6] generates a low-resolution latent features using NeRF and then upsample these features in 2D space with a StyleGAN-like structures.

None of these related methods are discussed and compared in this paper, making the judgement of "our approach is the first to propose a 3D-aware latent space" difficult. The suggestion would be add more discussion to clarify how the proposed approach differs from or improves upon these existing methods that used reduced resolution spaces for modeling NeRFs.

References:

[1] Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[2] Qian, Guocheng, et al. "Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors." The Twelfth International Conference on Learning Representations.

[3] Melas-Kyriazi, Luke, et al. "Realfusion: 360 reconstruction of any object from a single image. In 2023 IEEE." CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vol. 1. 2023.

[4] Metzer, Gal, et al. "Latent-nerf for shape-guided generation of 3d shapes and textures." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Wang, Tengfei, et al. "Rodin: A generative model for sculpting 3d digital avatars using diffusion." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[6] Gu, Jiatao, et al. "Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis." arXiv preprint arXiv:2110.08985 (2021).

questions:
(1) In the first stage each latent tri-plane is trained individually, hence each tri-plane will have its own finetuned version of the image decoder. In section 4.2, when jointly training the IG-AE with the latent triplanes, which initialization is used for the decoder for IG-AE?

(2) Regarding the experiment setup: how many held-out scenes in each category (Cake, Figurine and House) for the Objeverse dataset? Does the same category appears in the training data of IG-AE ?

(2) In Table 5, it is shown that the InstantNGP exhibits both longer training time and rendering time paired with IG-AE (including the RGB decoding time for IG-AE). Given that the quality under IG-AE has no advantages, what are the advantages for the proposed IG-AE method compared with directly training NeRF using InstantNGP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705644305

thread_type:
official_review

<thread_object>
creator: reviewer_rtxt

<content>
summary:
This paper introduces the concept of 3D-awareness into the latent space of autoencoders, through learning a latent NeRF and an Inverse Graphics AutoEncoder (IG-AE). The proposed latent NeRF is general and can be used as standard extension to different previous NeRF architectures, with implementation code in the supplementary. The proposed IG-AE jointly train latent NeRFs and standard AE to achieve 3D awareness in latent space, on both synthetic and real data. The experimental comparisons and analysis are mostly thorough and comprehensive, while still with some issues in justifying its motivations.

strengths:
1. 3D-awareness of 2D image generation from autoencoders is an important issue, and this paper seems to be the first work to address it.
2. The proposed latent NeRF operates fully within the latent space, which is a standardized solution and can work as an open-source extension to established NeRF architectures. The authors submitted the code in the supplementary.
3. The training framework of latent NeRF and IG-AE is sensible, with detailed ablation study to justify the loss design.
4. The paper is very well-written and easy to follow, with method design and experiments to support its motivation and arguments.

weaknesses:
Although I agree with the importance of 3D-awareness of 2D autoencoders and appreciate the authors' efforts, I still have some concerns/questions for the proposed method to address 3D-awareness with latent NeRF:
1. Is a latent NeRF really necessary? Does it bring more advantage or more damage to the standard 2D AEs, especially when the scenes are quite complicated? Learning a scene with a NeRF model tends to smooth out high-frequency details (easier to be 3D-inconsistent), which is also true for TV loss as discussed in Line 363. Is it possible that the latent NeRF learning and TV loss would force the encoder to remove high-frequency contents during joint training, which can not be recovered by the decoder? One example is the cake in Figure 6, IG-AE is over-smoothed comparing to RGB. Could you conduct quantitative evaluation to show the level of loss in high frequencies?
2. Additionally, if the intension is to bring 3D-awareness to 2D autoencoders, why not add NeRF constraints on the final RGB images instead of the latents? Maybe this would cause less information loss? Is it possible to compare the trade-off between using latent NeRF and NeRF in final RGB space?
3. On the condition of the lost high-frequency details, is this level of "3D-awareness" still helpful for the 2D autoencoders? The encoding and decoding reconstruction on natural images by 2D encoders seems to be fine for now, as shown in Figure 9 and other literatures. The problem emerges when it comes to novel image generation rather than simple reconstruction, which is untouched in this paper.
4. The experiments seem to be limited to simple objects? The Objaverse and Shapenet objects should be much simpler than the original domain of the pre-trained "Ostris KL-f8-d16" VAE. Is it possible to show the performance on more complicated scenes with rich high-frequency details?
5. If the focus is to to improve standard 2D autoencoders then the comparisons to AE should be the reconstruction of multiview images?
6. If the justification is novel view synthesis, the baseline methods should be other (latent) NeRF methods instead of AE? Since AE itself is not for this task. Table 5 does show some advantages in training and rendering time, but with significant sacrifice on NVS performance.

questions:
1. Is there any convergence issue during IG-AE training? Do you simply jointly train all modules with all losses?
2. I haven't checked the computation of latent NeRF, so I'm not sure if there would be any problems.
3. Is this paper an improved version of the one in the supplementary? There seems to be quite some similarities.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732035795957

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank the reviewers for dedicating their time and efforts to provide useful and relevant feedback about our work. We especially value that:
- generally, the reviewers have recognized the importance of the issue of 3D-awareness of 2D image autoencoder raised in this paper (RTXt s.1, jLB8 s.1) and the novel use of inverse graphics in the latent space (RTXt s.1, iRnM s.1);
- technically, they have appreciated our training framework (jLB8 s.2) and our detailed ablation study (RTXt s.3), as well as our open-source extension of Nerfstudio (RTXt s.2, jLB8 s.4, iRnM s.4);
- structurally, they have generally understood the paper (jLB8 s.2) and found it "very well written and easy to follow, with method design and experiments to support its motivation and arguments" (RTXt s.4). 

We provided an individual response to each reviewer, and additionally address common concerns below. We will provide the revised version of the paper as soon as possible, as to facilitate post-revision discussions. Supplementary videos can be (anonymously) found at https://anon-supp-iclr25.github.io. We look forward to discussing with the reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732569007174

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Revision

comment:
We thank the reviewers once again for the valuable reviews and discussions. As promised, we have submitted our paper revision with the changes discussed in the rebuttal, highlighted in blue in our revision.
- *(Reviewers RTXt (w.1), iRnM (w.2), jLB8)* We have **revised the introduction to better illustrate the motivation** of our work, as well as the advantages and applications brought by latent NeRFs.
- *(Reviewers jLB8 (w.2), iRnM (w.3))* We have **added a new section to our related work** that contrasts our work from other works tackling feature fields or scene embeddings in the literature, including the suggested references. Additionally, we have added a section and a summarizing table in the appendix (section A, table 3), providing a more detailed discussion of recent works and highlighting their distinctions.
- *(Reviewers RTXt (w.4), jLB8 (w.1.a))* We have revised our dataset section (5.2) to **better motivate our choice of dataset**, and to illustrate why it is a suitable choice for the evaluation of latent NeRF architectures and the 3D-awareness of our latent space.
- Moreover, **we have added additional experiments** to answer the reviewer's concerns and questions:
    - *(Reviewer jLB8 (w.1.c))* Table 8 (5.3, table in appendix D.2) reports NVS metrics computed directly on latent images, **further demonstrating the 3D-awareness of our latent space**.
    - *(Reviewer jLB8 (w.1.d))* Table 4 (appendix E) justifies our choice of autoencoder by **evaluating other autoencoders** on our task of latent NeRF training. This set of experiments further explains our choice of autoencoder, which is a compromise between NVS performance and training/rendering speeds.
    - *(Reviewer RTXt (w.2))* Table 9 (appendix D.2) reports **autoencoding performance on synthetic images**.
- Additionally, we have already provided the reviewers with supplementary illustrations available at https://anon-supp-iclr25.github.io/ that showcase animated videos highlighting the 3D-awareness of our latent space and the frame-to-frame consistency of our NeRFs.

We remain available for further discussions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732645828907

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Regarding the paper's scope and downstream applications

comment:
We would like to thank the reviewers for the animated and constructive discussion, which we believe has greatly helped us to improve our paper and clarified most of the raised concerns. One remaining concern shared by Reviewers RTXt and jLB8 is the lack of experiments on downstream applications. We would like to clarify our position on this matter in a separate thread for the sake of clarity.

Firstly, we reiterate and summarize our paper's contributions as acknowledged, to our understanding, by all reviewers.
- We identify a **new research direction, 3D-aware image latent spaces**, with **application perspectives clearly grounded in the literature**:
    1. efficiency gains by manipulating latent instead of RGB NeRFs for 3D tasks, where performance is hindered by 3D-inconsistent latent spaces;
    2. interoperability between NeRFs (3D) and pretrained latent-based methods (2D) which, until now, has necessitated special adapters due to 3D-inconsistent latent spaces.
- To tackle this challenging task -- as the only baseline, a naive autoencoder, does not work well even on simple cenes --, we propose **the first 3D-aware autoencoder**.
- We experimentally **confirm for the task of novel view synthesis**:
    - the 3D awareness of our autoencoder and its positive consequence on performance;
    - the efficiency gains of latent NeRFs.

We have debated with the reviewers over the following question:
> **Should the paper include experiments on downstream applications referenced in the motivation to show its advantages?**

**We already show the mentioned speed advantages (perspective 1)** (discussed in Sections 5.2 and D.2). We acknowledge that we do not include experiments in downstream tasks (perspective 2) and that, despite our performance gains relative to the current AE baseline, we do not match the performance of the RGB methods.

Still, given the **difficulty and novelty of the task** and the already demonstrated benefits, we think our submission can be considered as **a first step in this direction**, leaving other downstream applications outside our scope.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732803504782

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Supplementary Experiments

comment:
This is to inform the reviewers that we have added **experiments on 15 additional Objaverse objects** in a newly revised version of our paper, as suggested by reviewer jLB8. These experiments compare IG-AE and AE across the four NeRF representations for each object, **which adds 60 new comparisons**.

The quantitative results for these new experiments can be found in **tables 5 to 9** in the appendix (page 18, in brown).
For the qualitative results, we have **added new animated videos on our anonymous website** (https://anon-supp-iclr25.github.io/), and will be adding new comparative figures with visuals in the camera-ready version of our paper.

These results further support our previous claims of having a 3D-aware latent space better suited for latent NeRFs.

We remain open for further discussions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors present a method to align a 2D image autoencoder with a 3D latent space. The authors demonstrate that the resulting latent space leads to better NVS quality with less 2D artifacts.

The reviewers converged to positively-leaning scores of 6/6/6 over the course of discussion, appreciating the interesting new idea and novelty of this work and acknowledging the clear shown benefit. At the same time, questions regarding relevancy remain.

I decided to follow the reviewers recommendation and propose to accept the paper. I think this is an interesting paper that can spark some novel ideas.