Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder
Authors: Antoine Schnepf
Karim Kassab
Jean-Yves Franceschi
Laurent Caraffa
Flavian Vasile
Jeremie Mary
Andrew I. Comport
Valerie Gouet-Brunet
Venue: ICLR 2025 Poster (2025)
Keywords: Latent NeRF
NeRF
Autoencoder
Inverse Graphics
Nerfstudio
3D
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We learn a 3D-aware image latent space in which we train latent NeRFs.

Abstract: While pre-trained image autoencoders are increasingly utilized in computer vision, the application of inverse graphics in 2D latent spaces has been under-explored. Yet, besides reducing the training and rendering complexity, applying inverse graphics in the latent space enables a valuable interoperability with other latent-based 2D methods. The major challenge is that inverse graphics cannot be directly applied to such image latent spaces because they lack an underlying 3D geometry. In this paper, we propose an Inverse Graphics Autoencoder (IG-AE) that specifically addresses this issue. To this end, we regularize an image autoencoder with 3D-geometry by aligning its latent space with jointly trained latent 3D scenes. We utilize the trained IG-AE to bring NeRFs to the latent space with a latent NeRF training pipeline, which we implement in an open-source extension of the Nerfstudio framework, thereby unlocking latent scene learning for its supported methods. We experimentally confirm that Latent NeRFs trained with IG-AE present an improved quality compared to a standard autoencoder, all while exhibiting training and rendering accelerations with respect to NeRFs trained in the image space. Our project page can be found at https://ig-ae.github.io .

Published as a conference paper at ICLR 2025
BRINGING NERFS TO THE LATENT SPACE: INVERSE GRAPHICS AUTOENCODER
Antoine Schnepf" 1.2 Karim Kassab" 1,,, JJean-Yves Franceschi's Laurent Caraffa',
Flavian Vasile', Jeremie Mary', Andrew Comport 2, Valérie Gouet-Brunet!
Equal contribution
Critco AT Lab, Paris, France
2 Université Côte d' Azur, CNRS, I3S, France
3 LASTIG, Université Gustave Eiffel. IGN-ENSG, F-94160 Saint-Mandé
ABSTRACT
While pre-trained image autoencoders are increasingly utilized in computer vision, the application of inverse graphics 2D latent spaces has been under-explored. Yet, besides reducing the training and rendering complexity, applying inverse graphics in the latent space enables valuable interoperability with other latent-based 2D methods. The major challenge is that inverse graphics cannot be directly applied such image latent spaces because they lack an underlying 3D geometry. this paper, we propose an Inverse Graphics Autoencoder (IG-AE) that specifically addresses this issue. To this end, we regularize an image autoencoder with 3D- geometry by aligning its latent space with jointly trained latent 3D scenes. We utilize the trained IG-AE to bring NeRFs to the latent space with a latent NeRF training pipeline, which we implement in open-source extension of the Nerfstu- dio framework, thereby unlocking latent scene learning for its supported methods. We experimentally confirm that Latent NeRFs trained with IG-AE present im- proved quality compared to standard autoencoder, all while exhibiting training and rendering accelerations with respect NeRFs trained in the image space. Our project page can be found at httpss:/iget //ig-a///eeeeee/e gi github.it io.
Inverse Graphics
Render
E
Image space
3D-aware latent space
D
3D-aware latent space
Image space
IG-AE
underlying
underlying
3D space
3D space (latent)
(a) Image space analogy.
(b) Inverse Graphics Autoencoder (IG-AE).
Figure I: 3D-aware latent space. We draw inspiration from the relationship between the 3D space and image space and introduce the concept of a 3D-aware latent space. We propose an Inverse Graphics Autoencoder (IG-AE) that encodes images into 3D-aware latent images, hence preserving 3ID-consistency. We use these latents to train scene representations in the 3D-aware latent space.

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
Latent image representations are increasingly used in computer vision tasks. Most recent methods utilize latent representations for various tasks such as image generation (Rombach et al., 2022; Esser al.. 2021), and image segmentation (Long et al., 2015; Ohgushi al., 2020). as they allow to represent images in a compact form which reduces their representation complexity.
Similarly to 2D computer vision, leveraging latent image representations for 3D tasks would bring nu- merous advantages. First, training NeRFs on lower-resolution latent images would enable significant speed-ups in both rendering and training times. While efforts towards NeRF speed improvements have traditionally focused on working directly on NeRF architectures, this approach would instead tackle the space in which NeRF models are trained. This means that would offer a broader impact across various NeRF models, extending beyond single specific NeRF improvement. Second, latent NeRFs unlock an onter-operayility between NeRFs and pre-traing trained latent- based methods. For in- stance, previous works have enabled this inter-operability in a task-specific, ad-hoc manner to unlock applications like scene editing (Park et al., 2024) and generation (Metzer et al., 2023).
Despite the potential of latent NeRFs, the adoption of latent image representations for 3D tasks, such as Novel View Synthesis (NVS), remains limited due to the inherent incompatibilities between image latent spaces and 3D modeling methods. These incompatibilities stem from the lack of underlying 3D geometry in latent spaces on the one hand, and the necessity of 3D-consistency among images for solving the inverse graphies problem on the other hand. practice, encoding two 3D-consistent views of an object does not lead to 3D-consistent latent representations; instead, it produces latent images with features that vary inconsistently across views. This prohibits the direct application of scene learning methods such as NeRF (Mildenhall et al., 2020) on such representations.
To sidestep these incompatibilities, the aforementioned methods have resorted to the ad-hoc imple- mentation of custom, scene-dependent "adapters" (Khalid et al., 2023) or "refinement layers" (Park et al., 2024). However, these solutions are only workarounds and do not address the root cause the problem: the need for universal "3D-aware" latent space induced by image autoencoder that preserves 3D-consistency, Such an autoencoder would enable a broader compatibility between image autoencoders and NeRFs, unlocking the potential of latent NeRFs. In this paper, we explore this direction for the first time.
We start by proposing a technique that enables learning NeRF models in the latent space. Our training pipeline comprises two stages: Latent Supervision supervises NeRFs with latent image representations, and RGB Alignment aligns the learned latent scene with the RGB space. Yet, its application in a standard latent space results in sub-optimal latent NeRFs, due to their aforementioned incompatibilitie. To address this, we draw inspiration from the relationship between the image space and 3D space (Fig. la) and introduce 3D-aware latent spaces, which augment regular image latent spaces by incorporating an underlying 3D geometry. We present an Inverse Graphics Autoencoder (IG-AE) that embeds 3D-aware latent space (Fig. 1b). To achieve this, we apply 3D regularization on the latent space of an autoencoder by jointly training with synthetic set of latent 3D scenes. During this process, we align the encoded views with the renderings of the latent 3D scenes, which enforces 3D-consistency. Additionally, we propose an autoencoder preservation process that simul- tancously autoencodes real and synthetic data while performing training, which allows to conserve the reconstructive performance of the autoencoder.
We propose open-source extension of Nerfstudio (Tancik et al., 2023) that enables learning its supported NeRF models in latent spaces, thereby unlocking a streamlined approach for latent NeRF learning. This enables us evaluate various current NeRF architectures on the task of learning scenes in latent spaces. Our results highlight the effectiveness of our IG-AE in latent scene learning as compared to a standard AE. We consider this work to be the first milestone towards foundation inverse graphics autoencoders, and aspire that our open-source Nerfstudio extension promotes further research in this direction.
An overview of our contributions is given below.
We introduce the concept of 3D-aware latent spaces that are compatible with 3D tasks,
We present an Inverse Graphics Autoencoder (IG-AE) that maps images to a aw-aee latent space, all while preserving autoencoder performances,
2

---

Published as a conference paper at ICLR 2025
We propose standardized method to train NeRF architectures in latent spaces,
We propose open-source extension of Nerfstudio that support training supported NeRF models in the latent space, in an effort to streamline future work involving latent NeRFs.
We experimentally show that IG-AE enables improved latent NeRF quality with respect to the baseline AE and decreased training times with respect regular NeRFs.
2 RELATED WORK
Latent NeRFs. While NeRFs (Mildenhall et al., 2020) were originally conceived to work in the image space, they have also been extended to work in other feature spaces (Tschernezki et al., 2022; Kobayashi et al., 2022). Notably, Latent NeRFs are extensions of NeRFs to the latent spaces of image autoencoders. They have been particularly explored in works targeting applications such as scene editing (Park et al., 2024; Khalid et al., 2023) and text-to-3D generation (Metzer et al., 2023), which are not directly feasible with regular NeRFs. To cireumvent the incompatibilities between NeRFs and latent spaces, these works have resorted to special adapter layers that correct NeRF renderings into standard latent image representations. For novel view symthesis, Aumentado-Armstroon et al. (2023) employ hybrid NeRFs that are trained to simultaneously render both RGB and latent components. This is done to supervise the NeRF geometry during training, while only keeping latent components at inference, which enables good NVS performances and rendering speed-ups ups. In this work, we aim to train NsRFs that operate fully within the latent space. We address the incompatibility between NeRFs and latent spaces by tackling its root cause: the need for inverse graphics autoencoder that preserves 3D consistency, or in other words, the need for a a3 3D-aware latent space.
Scene embeddings. While methods like Latent NeRFs and other feature fields modify the NeRF rendering space by replacing scene images with "feature images", other approaches embed the entire scene information into "scene embedding". Recent works (Wang et al., 2023; Kosiorek et al., 2021; Sajjadi et al., 2022: Lan et al., 2024) focus on training encoders to transform scene information (c.g. images or text) into embeddings that fully encapsulate the 3D scene. NeRFs can then be obtained directly (without training) from such scene embeddings. In contrast, our approach aligns more closely with feature field methods (Tschernezki et al., 2022; Kobayashi et al., 2022), where NeRFs are trained to render "feature images" derived from scene images (e.g. DINO or CLIP features), enabling applications such segmentation and object retrieval. However, we train NeRFs to render latent image representations obtained from an autoencoder fine-tuned for 3D tasks, to tackle the task of 3D reconstruction. For elaborate distinction across recent works, refer the reader to Appendix A.
Nerfstudio. Subsequent to the introduction of NeRF (Mildenhall et al., 2020), multitude of NeRF models emerged to improve upon the original architecture by accelerating training times (Müller al., 2022; Kerbl al., 2023), improving rendering quality (Barron et al., 2021; Chen et al., 2022), as well as targeting other limitations (Fridovich- al.. 2021). With the escalation of NeRF methods, Nerfstudio (Tancik et al., 2023) emerged as a unified PyTorch (Paszke et al., 2019) framework in which NeRF models are implemented using standardized implementations, making it straightforward for researchers and practitioners integrate various NeRF models into their projects. Today, most well-known NeRF techniques are implemented in Nerfstudio (Müller et al., 2022; Barron et al., 2021: Mildenhall et al., 2020; Kerbl al., 2023), thus allowing for friction-free experience when testing and comparing novel techniques. While Nerfstudio currently provides streamlined approach for NeRFs, it limits the representation of 3D scenes to the natural color space, hence limiting both the research and development of Latent NeRFs. In this work, we propose an open-soured extension of the Nerfstudio framework that enables training supported NeRF methods in custom latent space, thus facilitating our current as well as future research in this area. Furthermore, we integrate our IG-AE into this extension, which enables us to evaluate various current NeRF architectures the task of latent scene learning.
er al., 2023; Martin-Bruall et al., 2021; Yu
LATENT NERF
In this section, we start by presenting latent NeRFs. Then, we propose a general latent NeRF training method, which will later serve to train NeRFs in the latent space of our IG-AE. Provided an encoder
3

---

Published as a conference paper at ICLR 2025
TA
AE
IG-AE
AE
IG-AE
Ground Truth
(a) Latent images.
(b) Decoded latent NeRF renderings.
Figure 2: Comparison IG-AE and a standard AE. Encoding 3D-coonistent images using an AE leads to 3DD-incosisstent latent images. When trained on such latents, NeRF renderings present artifacts when decoded. IG-AE presents a 3D-aware latent space with 3D-consistent latent images. Latent NeRFss trained with IG-AE eliminate these artifacts and more closely match the ground truth.
Eg and decoder De, our method can train any NeRF architecture latent space via two stages. Latent Supervision is adaptation of the standard NeRF training framework to the latent space which replaces RGB images with latent images. RGB Alignment fine- munetunes the decoder to align the learned latent scene with the RGB space, which experiments proved to be essential for strong NVS performances in the RGB space.
3.1 DEFINITION
Latent NeRFs are conceptually similar to standard NeRFs, with the primary difference being that they model NeRFs are simple extensions of standard NeRF methods, where the rendering resolution and the number of output channels are modified in accordance with the latent space dimensions. Let Fø be latent NeRF method with trainable parameters 0, where Fg (VanillaNNeRF (Mildenhall et al., 2020), Instant-NGP (Müller al., 2022), TensoRF (Chen al., 2022), K-Planes (Fridovich-Be et al., 2023)). For conciseness, we consider Fg to be generic NeRF method and abstract from method-specific nuances. Given a camera pose p, one can render a novel view as such:
the latent space of autoencoder opposed to the RGB space. As such, latent
Ep = Folp)
x Dxp
where 2, is the rendered latent image of shape (h, w, c), and T. is the decoded RGB image of shape (H1, W) We define > 1 as the resolution downscale factor when going from the RGB space the latent space: (h, w) (#/t, W/1). As NeRF rendering pipelines implement classic volume rendering (Kajiya Von Herzen, 1984) where pixels are independently rendered, latent NeRFs reduce the rendering complexity by factor of 1 as compared to standard NeRFs. This makes latent NeRFs highly attractive, as they alleviate key bottleneck in NeRF training, while maintaining the same target resolution after decoding.
3.2 TRAINING
Our latent NeRF training scheme consists of two stages, as illustrated in Fig. 3.
Latent Supervision. In this stage, we consider the direct adaptation of NeRF training in the latent space, i.c. training NeRF latent image representations rather than RGB images. Let LF, be the training loss corresponding to the NeRF method F3. Ly, comprises of a pixel-level reconstructive objective that matches the NeRF renderings to ground truth views, well as other method-specific
4

---

Published as a conference paper at ICLR 2025
Latent NeRF Training Pipeline
Encoding
Latent NeRF Inference Pipeline
p
NeRF Method
F8 Field, Ray Sumpler, Renderer, etc.
E$
D
xp
Fp
T p
Stage 1: Latent Supervision
LLS 0
Stage 2: RGB Alignment
Lalign (0, ab)
Latent NeRF Training. We train a Latent NeRF two stages. First, we train the chosen NeRF method Fo on posed encoded latent images using its proprietary loss Lr, that matches rendered latents žp and encoded latents 2p. Subsequently, we align with the scene in the RGB space by adding decoder fine-tuning via Lalign that matches ground truth images Tp and decoded renderings Tp.
Figure 3:
regularization terms. Latent Supervision consists of minimizing the following objective:
(o)
Σ PEP
Lr, Z0;
where Ep and 2p are respectively the encoded latent representation of the RGB ground truth and the rendered latent image, with pose p. denotes the of training camera poses. Note that, to avoid redundantly encoding training views, we start this stage by encoding all the training images X' ppp)pe into latent image representations 2 (2p ppp), and then caching them.
RGB alignment. While Latent Supervision effectively captures 3D structure in the latent space, even minor inaccuracies in latent NeRF renderings can be magnified during the decoding process. These inaccuracies stem from various sources of етгог within the pipeline, mainly: errors originating from the latent space and autoencoding performances (i.e. imperfect AE), and errors associated with 3D modeling (i.e. imperfect NeRF). Hence, in order to alleviate the effect of these errors, finish our latent NeRF training by an RGB alignment process, where we tne-tune the decoder Dy and the latent scene Fg to align with the RGB space. In practice, we minimize the following objective:
Lalign (0, 1) = Σ E1ap 1p1
PEP
where 2p is the RGB ground truth, and āp (āp) ) is the decoded latent NeRF rendering. Consequently, the latent NeRF not only exhibits good NVS performances the latent space, but also when decoding its renderings to the RGB space.
Overall, we divide latent NeRF training into Latent Supervision and RGB Alignment. Latent Supervision is an accelerated approach that learns 3D structures in the latent space thanks to reduced image resolutions, and RGB alignment enhances NVS quality in the RGB space by using RGB supervision and a decoder fine-tuning, while still rendering the NeRF in the latent space. Fig. (AE columns) illustrates the results our latent NeRF training method when applied a standard latent space. Latent NeRFs trained in standard latent space with our method learn a coarse geometry. However, due to the aforementioned incompatibilities between latent spaces and NeRFs, the decoded renderings present artifacts in the RGB space. To address this, shift our focus to resolving these incompatibilities in the next section, and present an inverse graphics autoencoder embedding a 3w-a latent that compatible with learning latent NeRFs.

---

Published as a conference paper at ICLR 2025
3D Regularization
LRGB
Learnable Latent Scenes
7
Tri-Planes
Renderer
Ti
T,
Tv
R
- latent
RGB Scenes
M
Renderer
D 4b
M₁
MN
Zag
Sop
E c
(synth)
Real Images
I
I
I
I
Ij
i
L
(real)
AE Preservation
Figure 4: IG-AE Training. We jointly learn a set of latent synthetic scenes T, and supervise the latent images 2x,p of an autoencoder with rendered 3D-consistent latents žx,p using Listent- We match decoded latent renderings E..p with the ground truth scene renderings Fs,p using LRGB- We preserve autoencoder performances on synthetic and real data respectively through ((syath) and C(real)
4 INVERSE GRAPHICS AUTOENCODER
We present IG-AE, autoencoder that encodes 3D-consistent images into 3D-consistent latent representations. To attain such an autoencoder, it is necessary that its latent space encodes the RGB space while also retaining underlying 3D geometry. In this section, 3D-consistnncy (Section 4.1), and then elaborate on how we train an IG-AE: we utilize synthetic data construct learnable of latent scenes which aims to supervise the latent space of an AE with 3D-consistent latents (Section 4.2), all while preserving autoencoding performances (Section 4.3). Fig. 4 presents overview of method.
start by defining
4.1 3D-CONSISTENCY
The notion of 3D consistency ensures that corresponding points or features in different images represent the same point or object in the scene, despite variations in viewpoint, lighting or occlusion. We denote the posed images X' (#, p E SE(3)), where SF(3) is the Special Euclidean group containing all camera poses in 3D. X' are posed 3D-consistent images if and only if there exists a 3D model M such that:
Vp SE(3), Render(M,p)) = Zp
(4)
Note that NeRF model inherently produces 3D-consistent images as it is an implicit model M rendered via classic volume rendering (Kajiya & Von Herzen, 1984). Also note that while 3D consistency is natural for posed images X { {xphpep obtained from scene in the image space, does not naturally extend to the latent space, as latent representations two 3D-consistent images not necessarily 3D consistent. Our 3D-aware latent space designed to mitigate this discrepancy.
4.2 RE-RULLARRZZAIOO
To achieve 3D consistteyy in the latent space, learn IG-AE by aligning the latent encodings of 3D-consistent images with reference 3D-consistent latent images. However, this cannot be directly achieved such 3D-consistent latent images available. To this end, learn set of 3D latent

---

Published as a conference paper at ICLR 2025
scenes with NeRFs from which 3D-consistent latents can be easily rendered. In fact, while NeRFs cannot replicate 3D- inconsistent latents from a given autoencoder, training them via CF, (which typically employs MSE reconstructive objective) leads to convergence towards a common coarse geometry that most satisfies the inputs, while maintaining 31D-consistency. We utilize this property in our approach to obtain 3D-consistent latent images with which we supervise our autoencoder. To learn our latent scenes in practice, we adopt Tri-Plane representations (Chan et al., 2022), as their simple architecture ensures a low memory footprint and a fast training.
Let M {M,, MN be a dataset of 3D scenes, and X. (4s.hher be the set of renderings of scene M,, E M from an array of training views P. We denote the set of latent scenes as the set of Tri-Planes T- = {Ti,... .... TN J. where r comprises both scene- scene-specific trainable parameters, as well as shared parameters present the feature decoder of our common Tri-Plane renderer Ry-
Before training our IG-AE, we start by encoding views of our scenes M into standard (i.e. 3D- inconsistent) latent views, and training our Tri-Planes on these latents. Subsequently, we proceed to train our IG-AE while continuing to jointly train our Tri-Planes. For each scene M.. we encode each view Fspp into a latent image ps and render the corresponding 3D-aware latent image žs, from latent scene T3 as follows:
Za,p Es (Ta,))
EeP = (T..
(5)
where Eg is the encoder with trainable parameters 6. We then define a loss function Liatent that aligns these latent representations:
Llatent (Ø, Zs,p 2e.p) = || Ee.p
(6)
This loss function updates both the encoder parameters á and the latent scene parameters T to minimize the distance between the latent representations. This means that, on the one hand, the latent scene T, is updated to align with the 3D-inconsistent latent images 2s,p, and hence learn coarse geometry common among latent images of Mg. On the other hand, the encoder parameters á are updated to produce latent images that more closely reassemble the latent scene renderings غ Ex.p or in other words, produce 3D-consistent images.
We additionally define LROB as loss function that mirrors Liatent in the RGB space:
LRGB(U, Taps, Ts.p) = || - -. 21,
where T.pp (Z.pp) is the decoded latent scene rendering. LEGB updates both the decoder parameters 1 and the latent scene parameters that the latent scene aligns with the RGB scene when decoded. This is to ensure the optimal decoding of 3D-consistent latents as well as NVS performances in the RGB space.
Therefore, our 3D regularization objective minimizes the following loss:
EAD (e, T)
Σ e.p
Mistent (6, Ti Zo.ps Za.p) + ARGB (v), Ti Tap Tz,p)]
(8)
where Ajatent and ÅRGB are hyper-paramette
Overall, our IG-AE is trained by aligning the latent space of an autoencoder with the space of latent images that is inferable via Tri-Planes, while also ensuring the proper mapping from the latent space to the RGB space. In practice, we synthetic scenes from Objaverse (Deitke et al., 2023) to learn 3D structure in the latent space, as they present well-defined geometry and error-free camera parameters. Fig. illustrates comparison between a standard AE latent space and our 3D-aware latent space. While latent NeRFss trained with an AE present artifacts when decoding their renderings, those trained in the latent space of our IG-AE do not exhibit such artifacts. Yet, our experiments show that solely applying our 3D-regularization optimization objective leads to degradation of the AE's generalization capabilities and autoencoding performances. On that account, we present in the next section additional components of our training that target reconstructive objective.
4.3 AUTOENCODER PRESERVATION
As 3D regularization does not incorporate autoencoder reconstruction, this section focuses on preserving the autoencoding performance of the AE both synthetic and real data. To this end,
7

---

Published as a conference paper at ICLR 2025
additionally jointly learn reconstructive objective. First, add autoencoder loss to align the ground truth RGB views with the reconstructed views after encoding and decoding:
C (syytth) (d, 1/; xa sxp) - 1=..,p L. 2t
where E.pp (2spp)) is the reconstruction of the ground truth image. This ensures that the autoencoder still reconstructs images from the scenes.
To avoid overfitting on synthetic data, we additionally inject real images from Imagenet (Deng et al., 2009) into our training pipeline, on which we also define reconstructive objective. We denote == fI,,.., J the dataset of real images. We define the reconstructive loss on an image I I as:
C(real) (Ø, ,h 1, 1) - 11, 1113 + Ap + ATVLTv(Ks))
(10)
where Ap and λrv the encoding of 1,. Lp is a perceptual loss that refines the autoencoder reconstructions by aligning the reconstructed and original images in the feature space of a pre-trained network. Lrv is total variation loss that acts as a regularization term to encourage spatial smoothness by penalizing high- frequency variations in the latent images. We found that leads latent images that more closely reassemble our 3D-consistent latents. This is done to prevent the IG-AE from converging towards dual-mode solution, where it only 3D-consistent for synthetic scenes while functioning as a normal AE for real scenes. More details about TV are present in Appendix C.
ppper--arameters. I, (ky, is the reconstruction of Ij. and k2 (I,
Therefore, our autoencoder preservation loss is defined as follows:
Lo )b)
au
X (synth)
Σ
(syyth ac
(Ø, (0) Z..p. Rx + X (real)
Σ C(real) Cireal) (0, 1b.
ac
(11)
(real) (syuth) and Age are hyper-parameters. where A
4.4 IG-AE TRAINING OBJECTIVE
Overall, we present a joint objective for IG-AE that regularizes its latent space with 3D geometry while preserving autoencoding performances. The objective minimizes the following loss:
SIG-AB(ó, v. = La vb r) + (6, 1))
(12)
In summary, our training strategy exploits the well-defined geometry of synthetic data and utilizes LaD learn latent scenes and regularize the latent space of an AE with 3D geometry, all while maintaining the reconstruction performance of the AE via Ene and real data.
5 EXPERIMENTS
We start by training IG-AE on synthetic scenes from Objaverse, as described in Section 4. Subsequently, we train Nerfstudio models in our 3D-aware latent space on scenes from out-of- distribution datasets, as well as held-out scenes from Objaverse. To evaluate our method, we compare the novel view synthesis performance of the Nerfstudio models when trained our 3D-aware latent space (IG-AE), standard latent space (AE), and the RGB space. Moreover, evaluate the auto-encoding performances of our IG-AE on held-out real images. Finally, we assess our design choices by presenting ablation study of our method, where omit either 3D-regularization components or our AE preservation components. The details of our Nerfstudio extension, as well as IG-AE and latent NeRF trainings, are available in Appendix B.
5.1 DATASETS
For 3D-reullarization, adopt Objaverse (Deitke et al., 2023), synthetic dataset which is standard when large-scale scale and diverse 3D data is needed (Liu et al., 2023; Shi et al., 2024).
Objaverse provides meshes of synthetic objects, and thus presents artifact in its scenes and no approximation errors in camera parameters. We utilize N = 500 objects from Objaverse. Each object rendered from V 300 views 128 128 resolution.
8

---

Published as a conference paper at ICLR 2025
AE
IG-AE
AE
IG-AE
AE
IG-AE
Vanilla-NeRF
GT
GT
GT
Instant-NGP
TensoRF
K-Planes
Figure 5: Qualitative results. Visualization of decoded latent NeRF renderings trained with standard AE and an IG-AE on scenes from three cittof-dditribution datasets. Latent NeRFs trained with AE exhibit artifacts in decoded renderings that not present in those trained with IG-AE.
Table I: Main Results ShapeNet datasets. All results obtained by training NeRFs with Latent NeRF Training Pipeline, and are averaged over 4 scenes from each dataset. Our 3D-aware latent space generalizes to out-of-distribution datasets and is more suited for latent NeRF training.
Bags dataset
Hats dataset
Vases dataset
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
28.81
0.954
0.046
28.75
0.960
0.032
32.00
0.968
0.023
Vanilla-NeRF
IG-AE
29.57
0,960
0.044
29.31
0.967
0.033
33.06
0.973
0.021
Instant-NGP AE 24.29
0.892
0.113
23.48
0.893
0.096
26.49
0.917
0.081
IG-AE
25.90
0.923
0.062
25.30
0.925
0.048
28.44
0.942
0.043
26.19
0.930
0.060
25.90
0.932
0.044
29.16
0.948
0.038
TensoRF
IG-AE
28.40
0,953
0.038
28.09
0,957
0.029
31.45
0,966
0.021
AE
28.00
0.946
0.041
27.68
0,951
0.031
31.56
0.964
0.023
K-Planes
IG-AE
29.22
0.957
0.038
28.81
0.962
0.027
32.79
0.971
0.019
For AE preservation, we adopt Imagenet (Deng et al., 2009), a large dataset of diverse real images. We utilize L 000 images, which we pre-process by doing a square cropping followed by downscaling to 128 x 128 resolution that matches our rendered scenes.
For NeRF evaluations, we utilize synthetic, object-level data as it aligns with the training domain. As such, we train NeRFs on held-out scenes from Objaverse, and on scenes from three out-of- distribution datasets: Shapenet Hats, Bags, and Vases (Chang et al., 2015). Note that, due to the challenges latent NeRFs face with simple object-level scenes, our evaluation focuses such scenes, and excludes more complex or real scenes. We discuss this in Section 5.4.
5.2 RESULTS
We report for each experiment the PSNR (1), the SSIM (1), and the LPIPS (L) (Zhang et al., 2018). Quantitative and qualitative results can be found in Table and Fig. 5, where we utilize our latent NeRF training pipeline to train various Nerfstudio models on out-of-distribution datasets from ShapeNet, both using our IG-AE and standard AE. Models trained with IG-AE showcase superior NVS performance compared those trained with standard AE. This shows that IG-AE embeds 3D-aware latent space better suited for NeRF training and capable of generalizing to new datasets. Additional results on held-out Objaverse scenes can be found in Appendix D.3. To further support this, Table 6 presents latent NVS metrics computed on latent images. Latent NeRFs showcase significantly better latent NVS performance with IG-AE, further confirming its awwaeeeess.. Table 4 presents the NVS performance in the two stage of Latent NeRF training. In Table 8, provide comparison of latent NeRF training with classical RGB training. Latent NeRF training showcases speedups both in terms of training and rendering times, but lower NVS performance, which discuss in Section 5.4.

---

Published as a conference paper at ICLR 2025
Table 2: Ablations. Comparison between IG-AE and the baseline AE on two tasks. First, compare their NVS performances when used to train latent Vanilla-NeRF. Second, we compare their reconstruction performances when auto-eneoding held-out images from ImageNet. Our method presents both strong NVS and reconstruction performances as compared to its ablations, validating our design choices. The bold and underlined entries indicate the best and second-bes results.
NVS
Reconstruction
Cake
Figurine
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
33.18
0,962
0.053
31.46
0.972
0.017
27.69
0.856
0,023
IG-AE (no 3D)
32.09
0.951
0.063
30.94
0.965
0.021
29.83
0.898
0.013
IG-AE (no Pr)
33.87
0,966
0.050
33.73
0.982
0.012
17.66
0.410
0.279
IG-AE
34.38
0.966
0.051
33.17
0.979
0.012
29.57
0.887
0.015
5.3 ABLATIONS
To justify our design choices, we conduct an ablation study in which we alternatively omit our 3D regularization pipeline and our AE preservation components. Ablation results on held-out Objaverse scenes for Vanilla NeRF can be found in Table 2. "IG-AE (no Pr)" removes the AE preservation components from our training by deactivating Lae. "IG-AE (no 3D)" omits our 3D regularizatiom by deactivating Lan. As illustrated, IG-AE (no Pr) presents deteriorated autoencoding performance due to overfitting on the task 3D-regularization. IG-AE (no 3D) showcases good autoencoding performances, but lower NVS performances, the latent space here is not 3D-awar due to the lack of 3D-regularization. Our method (IG-AE) demonstrates both good NVS and autoencoding reconstruction performances compared its ablations, thereby justifying the components of our pipeline. Note that ablating both our 3D regularization components and our AE preservation components leads back the case of standard autoencoder (AE). Additional ablations done with other NeRF models can be found in Table 5 in Appendix D.2.
5.4 LIMITATION
Table 8 compares latent NeRF training with classical RGB training. Despite speedups both in terms of training and rendering times, latent NeRFs have a lower NVS performance. This is due to limitation in the representation of high frequencies in latent NeRFs. We speculate that this issue arises because enforcing 3D consistency in the latent space tends to prioritize regularizing low-frequency structures over high-frequency details. This leads to the loss of tigh-freeuency details when decoding latent NeRF renderings back to the RGB space, as it can be seen in Fig. 5. Furthermore, this currently hinders the applicability of latent NeRFs on complex scenes with abundant high-frequency details, such real-world scenes. We leave this to future work, 3D-aware latent space via an image autoencoder, and is compatible with various NeRF architectures.
approach is the first propose
CONCLUSION
In this paper, we propose IG-AE, the first image autoencoder embedding 3D-aware latent space. Moreover, we present latent NeRF training pipeline that brings NeRF architectures to the latent space. We integrate our pipeline into an open-source extension of Nerfstudio, thereby enabling latent NeRF training for supported architectures. Extensive experiments show the notable improvements our 3D-aware latent space brings as compared to training NeRFs in a standard latent space, well as the efficiency improvements it brings with respect to training NeRFss the RGB space. In concluding this paper, several directions of future work arise, towards foundation inverse graphics autoencoders. This includes, for instance, the exploration of approaches improving the representation of high-freug gigh-rreuuenyy details when decoding latent NeRFs. We hope that our proposed Nerfstudio extension, as well as our open- apen-source codebase, will promote further research in this direction.
consider this work to be the first milestone
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Tristan Crmmentado-Ammsrronn Ashkan Mirzaei, Marcus A Brubaker, Jonathan Kelly, Alex Levin- shtein, Konstantinos G Derpanis, and Igor Gillischenski. Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations, arXiv preprint arXiv:2310.17880) 2023.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Bartin-Brualla, and Pratul P. Srinivasan. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.
1855-5864, October 2021.
Jaret Burkett. Ostris VAE KL-18-d16. vae-k1-f8-d1 6, 2024. Accessed: 2024-09-25.
https://huggingfacc.cooo
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient Aroometry-Awarr 3D Generative Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 16123-16133, June 2022.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An Information-Rice 3D Model Repository. arXiv preprint arXiv: 152200012, 2015.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial Radiance Fields. In European Conference on Computer Vision (ECCV), 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A Universe of Annotated 3D Objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogmition (CVPR), pp. 13142-13153, June 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High-Resoluiin Image Synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873-12883, June 2021.
Sara Fridovich-Keil Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields Without Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5501-5510 June 2022.
Sara Fridovich-Keil Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12479-12488, June 2023.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. StyleneRF: A style-based 3d aware gener- ator for high-resolution image synthesis. In International Conference on Learning Representations, 2022.
James Kajiya and Brian P. Von Herzen. Ray Tracing Volume Densities. SIGGRAPH Comput. Graph., 18(3):165 -174, January 1984. doi: 10. 10.1145/96496..88888 8005594
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions Graphics, 42(4), July 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LERF: Language Embedded Radiance Fields. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 19729-19739, October 2023.
11

---

Published as a conference paper at ICLR 2025
Umar Khalid, Hasan Iqbal, Nazmul Karim, Jing Hua, and Chen Chen. LatentEditor: Text Driven Local Editing of 3D Scenes. arXiv preprint arXiv:2312.0933 2023.
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for Editing via Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 2331 23311-2333. Curran Associates, Inc., 2022.
Feature Field Distillation.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokra, and Danilo Jimenez Rezende. NeRF-VAE: A Geometry Aware 3D Scene Generative Model. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 5742-5752.
PMLR, 18-24 Jul 2021.
Yushi Lan, Fangzhou Hong, Shuai Yang. Shangchen Zhou, Xuyi Meng, Bo Dai, Xingang Pan, and Chen Change Loy. LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation. In ECCV, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1-to-3 Zero-shot one image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9298-9309, October 2023.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully Convolutional Networks for Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2015.
Ricardo Martin Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7210-7219, June 2021.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-NeRF for Shape-Guides Guided Generation of 3D Shapes and Textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12663-12673 12673, June 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, ACM Trans. Graph., 41(4):102:11102:155, July 2022. doi: 10. 00.1145522822333330122
Toshiaki Ohgushi, Kenji Horiguchi, and Masao Yamanaka. Road Obstacle Detection Method Based on an Autoencoder with Semantic Segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV), November 2020.
JangHo Park, Gihyun Kwon, and Jong Chul Ye. ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF. In The Twelfth International Conference on Learning Representations, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein. Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Hiigh-erformance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- Resolution Image Synthesis With Latent Diffusson Models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695 June 2022.
12

---

Published as a conference paper at ICLR 2025
Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms. Physica D: Nonlinear Phenomena, 60(1):259-265 1992. ISSN 0167-2789. doi: http:///ooorr//.000000011111/1//
Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry- View Synthesis Through Set-Latent Scene Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 0229-6238, June 2022.
Novel
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. MVDream: Multi-view diffusion for 3d generation. In The Twelfth International Conference on Learning Representations, 2024.
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister and Angioo Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
V. Tschernezki, Laina, D. Larlus, and A. Vedaldi. Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations. In 2022 International Conference on 3D Vision (3DV), pp. 3DV57658. DDN57665.2022.000055
Los Alamitos, CA, USA, sep 2022. IEEE Computer Society. doi: 10.1109/
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, and Baining Guo. RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4563-4573, June 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural Radiance Fields From One or Few Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 4578-4587, June 2021.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
13

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Table Comparison with related work. Our work is the first train various NeRF architectures in the latent space of an autoencoder, where NeRFs are able to render images that match AE latent, thanks to the aw-ae latent space our IG-AE.
3D alignment
Method
Rendering space
rendering and reference spaces
NeRF Repr.
Task
NeRF-VAE
RGB
yes (RGB)
Vanilla-NeRF
Generation
Rodin
RGB
yes (RGB)
Tri-Planes
Generation
LN3Diff
RGB
yes (RGB)
Tri-Planes
Generation
StyleNeRF
Feature
Vanilla-NeRF
Generation
VAE Latent (Train)
Latent-NeRF
yes (RGB)
J-NGP
Generation
RGB (Inference)
LERF
RGB & CLIP
yes (RGB)
Vanilla-NeRF
Distillation
N3F
RGB & DINO
no (DINO)
Tri-Planes
3D Distillation
Latent-Editor
VAE Latent
no (VAE)
Vanilla-NeRF
3D Editing
ED-NeRF
VAE Latent
no (VAE)
TensoRF
3D Editing
RGB & VAE Latent (Train)
RLS
no (VAE)
Vanilla-NeRF
Reconstruction
VAE Latent (Inferenee)
Vanilla-NeRF,
Ours
IG-AE Latent (3D-aware)
yes (IG-AE)
K- K-Planes, TensoRF, Instant-NGP
Reconstruction
This section is dedicated compare some recent works utilizing NeRFs with scene embeddings and feature images. Table 3 summarizes this section.
As discussed in the related work (Section 2), some methods encode scene information (e.g, images, text) into scene embedding that then serves to produce NeRF. Such works do not tackle changing the RGB rendering space of NeRFs.
NeRF-VAE (Kosiorek et al., 2021) trains a VAE to encode scene images into a scene embedding. This embedding is then used to condition NeRF model. They can additionally sample such embeddings in the VAE latent space, which allows for scene generation.
Rodin (Wang et al., 2023) enables scene generation with an optional conditioning via images and/or text. They encode the input image and/or textual prompt into scene embedding, which then used to condition a diffusion model that generates a Tri-Plane representing the scene.
LN3Diff (Lan et al., 2024) enables fast ext/single-imaae-to-e to-3D generation. For single- image-to-3D, the image is encoded into scene embedding, which used condition DiT transformer that generates a Tri-Plane representing the scene.
Other works change the rendering space NeRFs, which makes them more closely related to our work, with still some key differences.
StyleNeRF (Gu et al., 2022) proposes a high-resolution generative model with multi-view consistency. They achieve style-based generation via NeRFs that render feature images followed by decoder. However, these features can only be obtained via the NeRF rendering procedure.
Latent-NeRI (Metzer et al., 2023) tackles text-to-3D object generation. It trains latent NeRFs such that their renderings match the posterior distribution of Stable Diffusion (Rom- bach et al., 2022) under a descriptive text prompt. After this optimization, the latent NeRF is converted into an RGB NeRF to resolve the issues arising from training NeRFs standard latent space.
LERF (Kerr et al., 2023) distills CLIP features in NeRFs while simultaneously learning the RGB components of the scene, which allows for language queries in 3D.
14

---

Published as a conference paper at ICLR 2025
N3F (Tschemezki et al., 2022) distills DINO features into NeRF representation, while simultaneously learning the RGB components of the scene. This allows to apply image-level tasks (e.g. scene editing, object segmentation) NeRF renderings. The DINO features rendered by NeRFs represent a 3D-consistent version of the original DINO features.
Latent-Editon (Khalid et al., 2023) and ED-NeRF (Park et al., 2024) achieve scene editing by training NeRFs in VAE latent space. To make this possible, they implement special layers ("adapter" and "refinement layer") that correct the NeRF renderings to match latent images.
RLS Soimentado-Armstooon et al., 2023) is the closest our work. It employs hybrid NeRFs that are trained to simultaneously render both RGB and latent components of scene, where the latent components are learned via a VAE decoder. At inference, it exclusively renders the latent components. However, due to the lack of 3D-aware latent space, it still requires RGB components at training time to supervise the geometry of the scene.
The presented methods either tackle a different task or use a different rendering space, which makes them not directly comparable with our work. This means that our work the first to propose training purely latent NeRFs, where no RGB components are utilized, for the reconstructive task.
B IMPLEMENTATION DETAILS
B.1 NERESTUDIO EXTENSION
We extend Nerfstudio support latent scene learning. Specifically, incorporate our latent NeRF training pipeline as well as IG-AE in Nerfstudio while adhering to its coding conventions, thereby enabling the training of any Nerfstudio model in our 3D-aware latent space. We modify the ray batching process to make it correspond to full images, where the default behavior uses randomly sampled pixels. This is necessary be able to decode the rendered images without breaking the gradient flow. Beyond reproducibility purposes for this paper, we hope the proposed generic extension will facilitate research on latent NeRFs and their applications. Our code is open-source and available on the following GitHub repository: https: / ///g/gg/ thub. com Ant neSchnepf attent-nerfstudlo.
B.2 NERFSTUDIO MODELS TRAINING DETAILS
We test the following models from Nerfstudio: Vanilla-NeRF (Mildenhall al., 2020), Instant-NGP (Müller et al., 2022), TensoRF (Chen et al., 2022), and K-Planes (Fridovich-Ke) Keil et al., 2023). For each method, the Nerfstudio framework provides proprietary loss Fs, as well as custom training procedure that includes specific optimizers, schedulers and other method -ethood-peccficc components. To train a latent NeRF in Nerfstudio, we first train the chosen model for 10 000 iterations to minimize St. using the method-specific optimization process. Subsequently, we continue the training with 15 000 iterations of RGB alignment by minimizing Latign- To account for the change of image representations, we modulate the learning rate of each method by a factor of &LS in latent supervision, and a factor Ealign for RGB alignment. Appendix F.2 details the hyper-parameters we used in Nerfstudio, including the values of these factors for each method.
B.3 IG-AE TRAINING DETAILS
We adopt the pre-trained "Ostris KL--f-d166 VAE (Burkett, 2024) from Hugging Face, which has downscale factor / = 8, and c 16 feature channels in the latent space. We apply Total Variation (TV) regularization on our Tri-Planes with a factor X which prevents overfitting on train views (we exclude it from Eq. (8) for clarity). Analogously,
also apply TV regularization on encoded real images ky. A detailed presentation of TV and its benefits for NVS can be found Appendix C. settings can be found Appendix F.I. The training code for IG-AE is soun-s and available on the following
Training IG-AE takes 60 hours 4x NVIDIA L4 GPUs. Our detailed
GitHub repository: https:://// // httg//ttttuubot com k kassabaa / iag..
15

---

Published as a conference paper at ICLR 2025
C TRI-PLANE REPRESENTATIONS
Tri-Plane representations (Chan et al., 2022) are explieit-impliiit scene representations enabling scene modeling in three axis-aligned orthogonal feature planes, each of resolution K K with feature dimension F. To query a 3D point E R it is projected onto each of the three planes to retrieve bilineraly interpolated feature vectors Fzy. Faz and Fyz- These feature vectors are then aggregated via summation and passed into small neural network to retrieve the corresponding color and density, which are then used for volume rendering (Kajiya & Von Herzen, 1984). We adopt Tri-Plane representations for our set of latent scenes for their lightweight architectures and relatively fast training times.
TV regularization. Total variation measures the spatial variation of images. It can act regularisation term to denoise images or ensure spatial smoothness. Total variation on images is expressed as follows:
(TV HW
-11
1-11111++ 11 1
(13)
where 11118 is the Ip norm to the power q.
Total variation regularization was adapted for inverse graphics to promote spatial smoothness in implicit-explici NeRF representation (Fridovich-Keil et al., 2023; 2022; Chen al., 2022). Accord- ingly, we regularize our Tri-Planes with total variation, which we write below:
CTV EET
(14)
where T(c) represents the feature plane of index e in practice, we add the regularization term X E. CTV (T.) our 3D regularization loss LaD (Eq. (8)), where Xap is hyperparameter. While this TV regularization done on plane features, it effectively translates to the latent 3D scene, as latents obtained from decoding these regularized features. This leads to smooth latent renderings and smooth gradients, as we utilize TV regularization here with (p. g) = (2, 2). which discourages high frequency variations in features.
Additionally, we use total variation in our AE preservation loss CAE (Eq. (10) with (p: (2, 1) to discourage autoencoder from producing latents with high frequency variations, while conserving sharp edges (Rudin et al., 1992). In fact, we noticed during our experiments that solely using an reconstructive objective on an autoencoder leads latents with high frequencies. These high frequencies are inconsistent across latent encodings of the same 3D-consistent object. Hence, to ensure the compatibility of Mae on encoded latents.
(<real) with the 3D regularization term LaD, add TV regularization
D SUPPLEMENTARY RESULTS
D.1 IMPORTANCE OF RGB ALIGNMENT
We show in Table 4 the importance the second stage of RGB Alignment in the Latent NeRF Training Pipeline, where NVS performances in the RGB space improves after aligning latent scenes with the RGB views.
D.2 ADDITIONAL QUANTITATIVE RESULTS
Tables 5 and 10 14 illustrate additional quantitative evaluations of various NeRF architectures trained a standard AE latent space and our IG-AE 3D-aware latent space on held-out Objaverse scenes, using latent NeRF training pipeline. These results further support conclusions. Additionally, Table also includes our ablations.
Table 6 compares the NVS performance of latent NeRF methods on latent images. Specifically, it compares NeRF-rendered latent images with encoded latent images. Methods trained in the latent space of our IG-AE exhibit significantly better latent NVS performance, confirming its 3D-awareness.
16

---

Published as a conference paper at ICLR 2025
Table 7 compares the autoencoding performance of our IG-AE and the baseline AE when used on images from held-out Objaverse scenes and OOD Shapenet scenes. Our IG-AE showcases better PSNR and SSIM metrics but slightly worse LPIPS metrics, which indicates minor loss in high frequency details.
Table 8 compares RGB NeRFs and Latent NeRFs using IG-AE, various Nerfstudio methods, in terms of NVS performance as well as training and rendering times.
D.3 ADDITIONAL QUALITATIVE RESULTS
Figs. to 8 compares the renderings of NeRFs respectively trained on the Cake, House and Figurine scene from Objaverse using our ablated models as well our full model. Latent NeRFs trained with IG-AE (no 3D) present artifacts when decoding their renderings to the RGB space, as the latent space not 3D-aware. IG-AE (no Pr) and IG-AE both have 3D-aware latent space and present artifacts, as previously illustrated. IG-AE (no Pr) demonstrates degraded auto-encoding performances.
Fig. 9 illustrates real images that are auto-encoded using all IG-AE ablations.
Figs. to 14 showcase additional qualitative results on held-out objaverse scenes.
Novel view synthesis consistency is better visualized in video format, which showcases frame- to-frame consistency. We provide such visualizations on our project page: https: /atp://ggaaaaaaae ae. github.ic.
E CHOICE OF AUTOENCODER
Table 9 compares autoencoders with various latent resolutions. For this comparison, adopt the autoencoders "kl-f4", "kl-f8", "kl-f16" from (Rombach et al., 2022), exhibiting downscale factors of 4. 8, and 16 respectively. Note that the compared autoencoders are not trained into inverse graphics autoencoders to conduct this study. As the table presents, NVS performances are directly correlated with latent resolution. We speculate that this comes from the fact that higher resolution latent space encourages a more local dependency between the RGB image and its latent representation, which is more advantageous in the context of 3D-awareness. As baseline for IG-AE, select an autoencoder with downscale factor of 8 as compromise between latent resolution and rendering speed.
F HYPERPARAMETERS
This section details our hyperparameter settings for both our IG-AE training and our Latent NeRF training in Nerfstudio.
F. IG-AE TRAINING SETTINGS
Table 15 details the hyperparameters taken to train our IG-AE.
F.2 NERFSTUDIO TRAINING SETTINGS
Table 16 details the hyperparameters taken train NeRF models in Nerfstudio.
17

---

Published as a conference paper at ICLR 2025
Table RGB alignment value. We show our NVS performances after latent supervision and RGB alignment when training NeRF models with IG-AE. NVS has better performances in the RGB space after doing our stage 2 of RGB alignment. All results are obtained on the Cake scene from the Objaverse dataset.
Latent Supervision
RGB Alignment
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
VanillaNNeRF
24.34
0.870
0.175
34.68
0.967
0.050
Instant-NGP
22.08
0.826
0.182
28.41
0.917
0.063
TensoRF
24.12
0.862
0.175
33.06
0.962
0.043
K-Planes
22.68
0.842
0.200
33.20
0.962
0.053
Table 5: Supplementary evaluations. Quantitative NVS results on held-out scenes from Objaverse. This table extends Table by conducting our ablations on all our adopted representations.
Cake
House
Figurine
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
33.18
0.053
31.71
0.031
31.46
0.017
Vanilla NeRF
IG-AE (ou Pr)
33.87
0.966
0.050
34,08
0.961
0,022
33.73
0.9821
0.012
(no 3D)
IG-AE
34.68
0.967
0.050
33.10
0,954
0,027
33.10
0.979
0,013
AE IG-AE (oo Pr)
24.69
0.122
22.67
0.784
0.196
24.62
0.908
0.086
28.04
0.917
0.052
27.22
0,882
0,055
26.46
0.9297
0.043
Instant-NCrI
(no 3D)
27.17
IG-AF
28.41
0.917
0.063
27,04
0,882
0,064
27.20
0.941
0.040
AE
28.79
0.928
0.046
27.31
0.904
0.056
27.60
0,949
0.037
TensoRF
IG-AF (no Pr)
34.30
0.966
0.041
32,92
0,952
0.025
32.86
0.9780
0.012
IG-AE (no 3D)
29.56
0.094
27.99
0,083
28.89
0.034
IG-AE
33.06
0.962
0.043
31,78
0,951
0,026
31.36
0.972
0.018
AE IG-AE (no Pr)
31.82
0.954
0.056
30.06
0.927
0.038
30.45
0.965
0.021
K-Planes
34.75
0.966
0.049
33.22
0,949
0,026
33.67
0.9796
0.011
IG-AB (no 3D)
29.47
0.935
0.103
29.75
0,902
0,088
30.20
0,9602
0.027
IG-AE
33.53
0.963
0.052
31,73
0,940
0,031
32.44
0.9751
0.014
Table Latent NVS performance. Quantitative evaluation of NVS performance of NeRF methods on latent images. The metrics compare NeRF-rendered latent images with encoded latent images. Methods trained in the latent space of our IG-AE exhibit significantly better latent NVS performance, confirming its 3D-awareness.
Objaverse (cake)
Shapenet (vase)
PSNR
SSIM
PSNR
SSIM
AE
38.36
0.854
38.48
0.841
Vanilla-NeRF
IG-AB
50.31
0.989
48.20
0.979
AE 35.87
0.768
35.94
0.734
Instant-NGP
IG-AE
44.11
0.967
43.84
0.955
AE 36.79
0.817
36.56
0.780
TensoRF
IG-AE
47.37
0.983
46.34
0.967
AE 36.85
0.798
36.53
0.755
K- K-Planes
IG-AE
44.74
0.969
44.26
0.949
18

---

Published as a conference paper at ICLR 2025
Table 7: Synthetic reconstruction. Comparison of autoencoding performances of IG-AE and the baseline autoencoder (AE) on images from Objaverse held-out scenes and Shapenet OOD scenes. In each dataset, the metrics are averaged over the images of all the test scenes.
Objaverse
Shapenet
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
36.47
0.975
0.005
34.78
0.985
0.007
IG-AE
37.92
0.980
0.011
37.16
0.989
0.009
Table 8: Comparison with RGB training. Bringing NeRFs to the IG-AE latent space entails significantly lower training and rendering times for most methods. Rendering times represent an average over 1000 image renderings. RGB methods render images at 128 x 128 resolution, whereas latent methods render 16 16. Note that for latent NeRF methods, decoding takes an additional 6.7 ms for each method to obtain the final 128 x 128 images. Training and rendering time is measured using a single NVIDIA L.4 GPU. As for rendering quality (NVS), some methods are more compatible direction of future improvements. NVS metries are averaged over three Objaverse scenes: Cake, House and Figurine.
with training in latent spaces than others, which we
Training Space
Training Time (min)
Rendering Time (ms)
NVS
PSNR
SSIM
LPIPS
RGB
637
1201
40.78
0.993
0.003
Vanilla NeRF
IG-AE
28
19.7
33.60
0.966
0.030
RGB
6
11.3
37.78
0.985
0.006
Instant NGP
IG-AE
19
7.3
27.47
0.913
0.056
RGB
92
101.4
40.65
0.992
0,003
TensoRF
IG-AR
20
10.4
32.04
0.961
0.029
RGB
88
63.3
32.98
0.964
0.027
K-Planes
IG-AE
29
11.3
32.36
0.958
0.033
Table 9: Autoencoder choice. Comparison of the performance of autoencoders with various downscale factors on the task of latent NeRF training. We choose autoencoder with intermediate downscale factor as a compromise between latent NVS performance and training time.
kl-f16
kl-f8
kl-f4
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
Vanilla-NeRF
29.58
0.941
0.047
32.22
0.962
0.027
35.38
0.978
0.013
Instant-NGP
23.54
0.844
0.117
26.16
0.900
0.084
30.03
0.948
0.036
TensoRF
25.91
0.891
0.080
28.46
0.934
0.042
33.56
0.972
0.014
K-Planes
27.59
0.917
0.060
30.06
0.943
0.032
33.42
0.966
0.017
19

---

Published as a conference paper at ICLR 2025
IG-AE (no 3D)
IG-AE (no Pr)
AE
IG-AE
RGB
Vanilla-NoRF
GT
Instant-NGP
TensoRF
K-Planes
Figure 6: Qualitative comparison. NeRF renderings of the Cake scene from Objaverse.
IG-AE (no 3D)
IG-AE (no Pr)
AE
IG-AE
RGB
Vanilla-NeRF
GT
Instant-NGP
TensoRF
K-Planes
Figure Qualitative comparison. NeRF renderings of the House scene from Objaverse.
20

---

Published as a conference paper at ICLR 2025
IG-AE (no 3D)
IG-AE (no Pr)
AE
IG-AE
RGB
Vanilla-NeRF
GT
Instant-NGP
TensoRF
K-Planes
Figure 8: Qualitative comparison. NeRF renderings of the Figurine scene from Objaverse.
IG-AE (no Pr)
IG-AE (no 3D)
IG-AE
AE
GT
Figure 9: Qualitative comparison of reconstruction quality when autoencoding. IG-AE (no Pr) presents artifacts when auto-encoding images. IG-AE (no 3D) omits these artifacts but does not embed a 3D-aware latent space. IG-AE omits the artifacts present in IG-AE (no Pr), while still integrating a 3D-aware latent space.
21

---

Published as a conference paper at ICLR 2025
Table 10: Supplementary evaluations. Quantitative NVS results on held-out scenes from Objaverse.
Helmet
Book
Burger
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
32.42
0.973
0.024
30.55
0.967
0.044
33.21
0.968
0.037
Vanilla-NeRF
IG-AE
33.24
0.977
0.021
32.02
0.973
0.039
34.23
0.972
0.037
24.63
0.918
0.087
23.61
0.904
0.144
23.52
0.862
0.123
Instant-NGP
IG-AE
28.27
0.951
0.035
27.22
0,950
0.058
27.48
0.922
0.044
AE
28.45
0.954
0.040
26.79
0.947
0.070
27.55
0.931
0.138
TensoRF
IG-AE
32.27
0.972
0.021
30.36
0.968
0.041
31.70
0.962
0.022
31.14
0.966
0.027
28.90
0.959
0.051
31.91
0.958
0.040
K-Planes
IG-AE
33.27
0.975
0.019
31.42
0.970
0.039
33.53
0.968
0.037
AE
IG-AE
AE
IG-AE
AE
IG-AE
VanillaNNeRF
INGP
GT
GT
GT
TensoRF
K-Planes
Figure 10: Supplementary evaluations. Qualitative NVS results on held-out scenes from Objaverse.
22

---

Published as a conference paper at ICLR 2025
Table 11: Supplementary evaluations. Quantitative NVS results on held-out scenes from Objaverse.
Cartoon
Dragon
Flower
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
29.54
0.974
0.017
32.20
0.981
0.022
35.59
0.984
0.014
Vanilla-NeRF
IG-AE
30.99
0.981
0.012
33.99
0.987
0.015
37.62
0.989
0.010
22.58
0.906
0.104
25.32
0.932
0.108
27.66
0.925
0.091
Instant-NGP
IG-AE
25.68
0.949
0.038
28.42
0,961
0.044
31.07
0.963
0.035
AE IG-AE
25.44
0.946
0.045
28.53
0.964
0.044
30.62
0.962
0,138
TensoRF
29.29
0.973
0.017
32.65
0.983
0.017
35.44
0.985
0.013
28.22
0.967
0.021
31.09
0.976
0.027
33.77
0.978
0.023
K-Planes
IG-AE
30.57
0,979
0.012
33.24
0,984
0.016
37.26
0,988
0.011
AE
IG-AE
AE
IG-AE
AE
IG-AE
Vanilla-NeRF
INGP
GT
GT
GT
TensoRF
K-Planes
Figure 11: Supplementary evaluations. Qualitative NVS results on held-out scenes from Objaverse.
23

---

Published as a conference paper at ICLR 2025
Table 12: Supplementary evaluations. Quantitative NVS results on held-out scenes from Objaverse.
Gravestone
Hut
Painting
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
31.64
0.973
0.034
30.33
0.963
0.030
36.07
0.984
0.018
Vanilla-NeRF
IG-AE
32.63
0.978
0.029
30.70
0.964
0.027
37.79
0.989
0.013
22.95
0.885
0.169
22.64
0.881
0.154
27.02
0.930
0.089
Instant-NGP
IG-AE
26.68
0.941
0.073
26.04
0.933
0.066
34.34
0.980
0.020
AE
26.65
0.944
0.076
26.89
0,943
0.058
31.51
0.967
0.137
TensoRF
IG-AE
31.15
0.974
0.032
30,00
0.963
0.030
37.75
0.989
0.012
28.72
0.960
0.053
28.68
0.955
0.038
34.79
0.982
0.020
K-Planes
IG-AE
31,25
0.973
0.031
30.19
0.961
0.029
38.28
0.990
0.012
AE
IG-AE
AE
IG-AE
AE
IG-AE
VanillaNNeRF
INGP
GT
GT
GT
TensoRF
K-Planes
Figure 12: Supplementary evaluations. Qualitative NVS results on held-out scenes from Objaverse.
24

---

Published as a conference paper at ICLR 2025
Table 13: Supplementary evaluations. Quantitative NVS results on held-out scenes from Objaverse.
Pig
Ringer
Robut
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
33.21
0.981
0.029
32.14
0.975
0.024
29.94
0.962
0.028
Vanilla-NeRF
IG-AE
34.34
0.987
0.019
33.70
0.982
0.015
31.76
0.973
0.019
24.61
0.913
0.140
24.68
0.901
0.119
23.49
0.884
0.107
Instant-NGP
IG-AE
28.35
0.956
0.067
28.24
0,951
0.042
26.01
0.923
0.049
AE
28.61
0.959
0.060
27.71
0.947
0.057
26.43
0.933
0,142
TensoRF
IG-AE
32.58
0.981
0.027
32.03
0.976
0.019
29.96
0.963
0.024
31.03
0.972
0.044
30.61
0.964
0.032
29.08
0.954
0.031
K-Planes
IG-AE
33.12
0,981
0.026
32.99
0.975
0.020
31.10
0.967
0.022
AE
IG-AE
AE
IG-AE
AE
IG-AE
VanillaNNeRF
INGP
GT
GT
GT
TensoRF
K-Planes
Figure 13: Supplementary evaluations. Qualitative NVS results on held-out scenes from Objaverse.
25

---

Published as a conference paper at ICLR 2025
Table 14: Supplementary evaluations. Quantitative NVS results on held-out scenes from Objaverse.
Collin
Table
Villa
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
AE
30.61
0.968
0.042
33.06
0.978
0.024
30.72
0.970
0.133
Vanilla-NeRF
IG-AE
33.17
0.976
0.032
34.96
0.983
0.019
32.18
0.977
0.024
21.67
0.876
0.211
23.29
0.873
0.161
24.21
0.903
0.138
Instant-NGP
IG-AE
27.25
0.941
0.071
29.19
0,956
0.042
26.18
0.935
0.084
AE
27.35
0.946
0.076
27.16
0.943
0.071
26.86
0.944
0.065
TensoRF
IG-AE
31.87
0.973
0.034
33.20
0.979
0.020
30.59
0.971
0.031
29.30
0.960
0.053
30.72
0.965
0.036
28.86
0.957
0.048
K-Planes
IG-AE
31.56
0.971
0.036
33.25
0.976
0.023
31.66
0,972
0.027
AE
IG-AE
AE
IG-AE
AE
IG-AE
VanillaNNeRF
INGP
GT
GT
GT
TensoRF
K-Planes
Figure 14: Supplementary evaluations. Qualitative NVS results on held-out scenes from Objaverse.
26

---

Published as a conference paper at ICLR 2025
Table 15: IG-AE hyperparameters. Hyperparameters used for our IG-AE training. More detailed information can be found in the configuration files of our open-source code.
Parameter
Value
General
Number of scenes N Pretraining epochs
500
50
Training epochs
75
Loss
Alatent
1
ARGB
1
A&D
1 10-4
(aynth)
Age
0.1
Age
(real)
0.1
Ap
0.1
ATv
10-4
Optimization
Optimizer
Adam
Batch size (scene views) Batch size (real images)
12
3
Learning rate (encoder)
5 10-5 5 x 10 10-5 1 10-4
Learning rate (decoder)
Learning rate (Tri-Plancs)
Scheduler
Exponential Decay
Decay factor
0.988
27

---

Published as a conference paper at ICLR 2025
Table 16: Latent NeRF training hyperparameters. Hyperparameters taken train our Latent
NeRFs in Nerfstudio. More detailed information can be found in the configuration files of our open-source Nerfstudio extension.
NeRF model
Parameter
Value
Latent Supervision iterations
10 000
RGB Alignment iterations
15 000
Batch size
NeRF learning rate
Nerfstudio default Nerfstudio default Nerfstudio default x 10-1 Adam Exponential decay
NeRF optimizer
Any
NeRF scheduler
Decoder learning rate
Decoder optimizer
Decoder scheduler
Decoder decay factor
0.9996
ELS
Vanilla NeRF
Calign
ELS
0.001
Instant-NGP
Salign
0.1
ÊLS
0.01
TensoRF
Calign
0.1
ELS
0.1
K-Planes
Calign
0.1
28


Reviews:
[{"review_id": "nTOfIrBRTR", "reviewer": "Reviewer_RTXt", "summary": "This paper introduces the concept of 3D-awareness into the latent space of autoencoders, through learning a latent NeRF and an Inverse Graphics AutoEncoder (IG-AE). The proposed latent NeRF is general and can be used as standard extension to different previous NeRF architectures, with implementation code in the supplementary. The proposed IG-AE jointly train latent NeRFs and standard AE to achieve 3D awareness in latent space, on both synthetic and real data. The experimental comparisons and analysis are mostly thorough and comprehensive, while still with some issues in justifying its motivations.", "questions": "1. Is there any convergence issue during IG-AE training? Do you simply jointly train all modules with all losses?\n2. I haven't checked the computation of latent NeRF, so I'm not sure if there would be any problems.\n3. Is this paper an improved version of the one in the supplementary? There seems to be quite some similarities.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. 3D-awareness of 2D image generation from autoencoders is an important issue, and this paper seems to be the first work to address it.\n2. The proposed latent NeRF operates fully within the latent space, which is a standardized solution and can work as an open-source extension to established NeRF architectures. The authors submitted the code in the supplementary.\n3. The training framework of latent NeRF and IG-AE is sensible, with detailed ablation study to justify the loss design.\n4. The paper is very well-written and easy to follow, with method design and experiments to support its motivation and arguments.", "weaknesses": "Although I agree with the importance of 3D-awareness of 2D autoencoders and appreciate the authors' efforts, I still have some concerns/questions for the proposed method to address 3D-awareness with latent NeRF:\n1. Is a latent NeRF really necessary? Does it bring more advantage or more damage to the standard 2D AEs, especially when the scenes are quite complicated? Learning a scene with a NeRF model tends to smooth out high-frequency details (easier to be 3D-inconsistent), which is also true for TV loss as discussed in Line 363. Is it possible that the latent NeRF learning and TV loss would force the encoder to remove high-frequency contents during joint training, which can not be recovered by the decoder? One example is the cake in Figure 6, IG-AE is over-smoothed comparing to RGB. Could you conduct quantitative evaluation to show the level of loss in high frequencies?\n2. Additionally, if the intension is to bring 3D-awareness to 2D autoencoders, why not add NeRF constraints on the final RGB images instead of the latents? Maybe this would cause less information loss? Is it possible to compare the trade-off between using latent NeRF and NeRF in final RGB space?\n3. On the condition of the lost high-frequency details, is this level of \"3D-awareness\" still helpful for the 2D autoencoders? The encoding and decoding reconstruction on natural images by 2D encoders seems to be fine for now, as shown in Figure 9 and other literatures. The problem emerges when it comes to novel image generation rather than simple reconstruction, which is untouched in this paper.\n4. The experiments seem to be limited to simple objects? The Objaverse and Shapenet objects should be much simpler than the original domain of the pre-trained \"Ostris KL-f8-d16\" VAE. Is it possible to show the performance on more complicated scenes with rich high-frequency details?\n5. If the focus is to to improve standard 2D autoencoders then the comparisons to AE should be the reconstruction of multiview images?\n6. If the justification is novel view synthesis, the baseline methods should be other (latent) NeRF methods instead of AE? Since AE itself is not for this task. Table 5 does show some advantages in training and rendering time, but with significant sacrifice on NVS performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RLwnt7lWst", "reviewer": "Reviewer_jLB8", "summary": "This paper propose a two-stage pipeline to train a latent space which is suitable for fitting NeRFs. In the first stage, a set of latent tri-planes are individually trained for each scene in the training data. The latent tri-planes are fitted by minimizing both the volumetric rendering error in latent space and the pixel difference in RGB space through a image decoder. In the second stage, an autoencoder is trained on all scenes in the training data, jointly with all individual tri-planes. The fidelity of the autoencoder is additionally regularized by training on additional real images. Experiments demonstrated that the proposed latent space is able to fitting NeRFs better than a image-based latent space.", "questions": "(1) In the first stage each latent tri-plane is trained individually, hence each tri-plane will have its own finetuned version of the image decoder. In section 4.2, when jointly training the IG-AE with the latent triplanes, which initialization is used for the decoder for IG-AE?\n\n(2) Regarding the experiment setup: how many held-out scenes in each category (Cake, Figurine and House) for the Objeverse dataset? Does the same category appears in the training data of IG-AE ?\n\n(2) In Table 5, it is shown that the InstantNGP exhibits both longer training time and rendering time paired with IG-AE (including the RGB decoding time for IG-AE). Given that the quality under IG-AE has no advantages, what are the advantages for the proposed IG-AE method compared with directly training NeRF using InstantNGP?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper attempts to research on an important problem of 3D-aware latent spaces.\n\n- The proposed method overall makes sense: the introduce of individually trained latent tri-planes provides a auxiliary variable which serves as a \"3D-aware\" guidance for auto-encoded latents. \n\n- Experiments show that the proposed IG-AE is better for training NeRFs than vanilla AE.\n\n- The method can easily integrated into NeRFStudio with an open-source extension.", "weaknesses": "My majority concerns are as follows:\n\n(1) While the proposed method is interesting and makes sense, the claimed property of \"3D-aware\" latent space cannot be fully justified from the given experiment results:\n- (a) The proposed method is only tested on dataset with limited variations. For evaluation NeRFs, ShapeNet dataset is not a best choice as it contains simple shapes and textures without any non-Lambertian effects. Additionally, the test dataset contains only three categories for Shapenet and Objeverse dataset. Evaulations on more complicated dataset with non-lambertian effects, such as Realistic Synthetic 360 [1] would help to more thoroughly demonstrate the \"3D-aware\" property of the latent space across a wider range of scenarios and make the results more convincing.\n- (b) Even under the limited test dataset, the performance of the proposed method are not convincing enough. Despite the IG-AE outperforms the vanilla AE counterpart, the quality degrades significantly compares to the RGB version. This raises the question that whether the image fidelity is enough for the IG-AE.\n- (c) The \"3D-aware\" property cannot be easily judged from given metrics and results in the paper. All quantitative results are averaged across views - it is hard to know whether the \"consistency\" between different synthesized views are preserved. One suggestion would be using some perceptal metrics to evaluate the consistency between different generated view, e.g., the CLIP feature similarity used in [2] and [3]. There are also very few qualitative results provided. Providing more qualitative results such as videos showing a rotated object can be also useful to visualize the view consistency.\n- (d) The effect of the original auto-encoder utilized for training IG-AE is not well explored. As discussed in Section 3.2 in [4], a latent space with low channel-wise depth (e.g., down to 4) and a slightly higher (but still significantly lower than RGB input resolution, 32 or 64 for example) will encourage local dependency over the autoencoder’s image and latent spaces. Under such circumstance, the latent representation is a near patch level representation of its corresponding RGB image, making it nearly equivariant to spatial transformations of the scene. In other words, it automatically (at least to some extent) has the property of \"3D-aware\" for training NeRFs. I would suggest the paper add more discussion and/or experiments regarding the 2D autoencoder used.\n\n(2) There has been many attempts for training NeRFs (or similar implicit 3D representations) in a space with reduced resolution, followed with upsampler operations in 2D space. For example:\n- Rodin [5] first generates a low-resolution tri-planes with 3D aware convolutions and an upsampler. \n- StyleNeRF [6] generates a low-resolution latent features using NeRF and then upsample these features in 2D space with a StyleGAN-like structures.\n\nNone of these related methods are discussed and compared in this paper, making the judgement of \"our approach is the first to propose a 3D-aware latent space\" difficult. The suggestion would be add more discussion to clarify how the proposed approach differs from or improves upon these existing methods that used reduced resolution spaces for modeling NeRFs.\n\nReferences:\n\n[1] Mildenhall, Ben, et al. \"Nerf: Representing scenes as neural radiance fields for view synthesis.\" Communications of the ACM 65.1 (2021): 99-106.\n\n[2] Qian, Guocheng, et al. \"Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors.\" The Twelfth International Conference on Learning Representations.\n\n[3] Melas-Kyriazi, Luke, et al. \"Realfusion: 360 reconstruction of any object from a single image. In 2023 IEEE.\" CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vol. 1. 2023.\n\n[4] Metzer, Gal, et al. \"Latent-nerf for shape-guided generation of 3d shapes and textures.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[5] Wang, Tengfei, et al. \"Rodin: A generative model for sculpting 3d digital avatars using diffusion.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[6] Gu, Jiatao, et al. \"Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis.\" arXiv preprint arXiv:2110.08985 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yxLXO1xaNi", "reviewer": "Reviewer_iRnM", "summary": "This paper introduces an Inverse Graphics Autoencoder (IG-AE) to explore the potential of inverse graphics in 2D latent spaces, which is relatively underutilized in computer vision. The authors highlight that inverse graphics applied in latent spaces can reduce both training and rendering complexity while enabling compatibility with other 2D latent-based methods. A key challenge addressed is that traditional image latent spaces lack 3D geometry, preventing direct application of inverse graphics. IG-AE addresses this by aligning the latent space of an image autoencoder with jointly trained latent 3D scenes. This enables a \"latent NeRF\" training pipeline, implemented within the Nerfstudio framework, making latent scene learning accessible for methods supported by the framework. Experiments show that latent NeRFs trained with IG-AE offer improved quality compared to conventional autoencoders, while also providing faster training and rendering compared to NeRFs trained in image space.", "questions": "1. In section 3, is the same VAE used as in section 4? The implementation details have not mentioned this.\n2. How is the \"3D aligned\" auto-encoder useful in the downstream tasks?\n3. In Tab. 2, how is the NVS task performed? Given a view as the input to your encoder, then synthesis novel views directly? Would you include more visual results?\n4. In your AE/VAE design, the latent space is still a \"3D-aware\" image, so why not using a truly 3D-aware latent such as the latent triplane in LN3Diff / Direct3D?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Novel Use of Inverse Graphics in Latent Space**: The authors claim that they are the first to explore this direction, which explores the relatively untapped area of applying inverse graphics in 2D latent spaces, which reduces training and rendering complexity and offers compatibility with other latent-based 2D methods.\n\n2. **Integration of 3D Geometry into Latent Spaces**: The authors address the issue of the lack of 3D geometry in standard image latent spaces by regularizing the autoencoder with 3D information, aligning its latent space with jointly trained 3D scenes. This approach enhances the representation capability of latent spaces for 3D scene understanding. From many perspectives, these efforts will be meaningful in this area.\n\n3. **Latent NeRF Training Pipeline**: Two new training pipelines are proposed. By creating a latent NeRF training pipeline and integrating it within the Nerfstudio framework, the paper unlocks efficient latent scene learning and makes it accessible to a broader set of methods, providing a practical tool for further research.\n\n4. **Open-Source Contribution**: The open-source extension to the Nerfstudio framework adds value to the research community by facilitating reproducibility and further experimentation, and the proposed method can naturally work on any new NeRF representations.", "weaknesses": "1. **The writing is pretty hard to follow.** I tried to understand this paper by reading over and over again, but still find it very hard to follow. Since this direction is relatively new, I strongly suggest the author to revise the writing in the later version for clearer elaboration.\n\n2. **The motivation of the proposed method is unclear.** From my understanding, the proposed method try to align the NeRF rendering to a pretrained autoencoder (Ostris KL-f8-d16 VAE here), but what are the benefits of doing this? Though promising PSNR/LPIPS metrics are achieved as in Tab. 1 and Tab. 2, does the proposed IG-AE / AE supports zero-shot novel view synthesis on the imagenet dataset? Regarding Tab. 2, I don't quite understand how the author evaluate the novel view synthesis pipeline here, since no visual results are included in the main paper. All the visual results are just reconstructing the input view.\n\n3. **Lack of comparison with existing methods**. Though the author claims this idea is quite novel and they are among the first to propose this solution, many existing methods have already shown similar spirits, including \n\n* SRT: Scene Representation Transformer.\n* NeRF-VAE: A Geometry Aware 3D Scene Generative Model\n* LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation\n\nTheir 3D (V)AE models support 3D view synthesis on the open vocabulary single-view inputs. The discussions / comparisons with these methods are needed to demonstrate the soundness of the proposed pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)