Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder

## Abstract

Abstract While pre-trained image autoencoders are increasingly utilized in computer vision, the application of inverse graphics in 2D latent spaces has been under-explored.
Yet, besides reducing the training and rendering complexity, applying inverse graphics in the latent space enables a valuable interoperability with other latent-based 2D methods.
The major challenge is that inverse graphics cannot be directly applied to such image latent spaces because they lack an underlying 3D geometry.
In this paper, we propose an Inverse Graphics Autoencoder (IG-AE) that specifically addresses this issue.
To this end, we regularize an image autoencoder with 3D-geometry by aligning its latent space with jointly trained latent 3D scenes.
We utilize the trained IG-AE to bring NeRFs to the latent space with a latent NeRF training pipeline, which we implement in an open-source extension of the Nerfstudio framework, thereby unlocking latent scene learning for its supported methods.
We experimentally confirm that Latent NeRFs trained with IG-AE present an improved quality compared to a standard autoencoder, all while exhibiting training and rendering accelerations with respect to NeRFs trained in the image space.
Our project page can be found at https://ig-ae.github.io .

## 1 Introduction

Figure: (a) Image space analogy.
Refer to caption: x1.png

Latent image representations are increasingly used in computer vision tasks.
Most recent methods utilize latent representations for various tasks such as image generation (Rombach et al., ; Ramesh et al., ; Esser et al., ), and image segmentation (Long et al., ; Ohgushi et al., ), as they allow to represent images in a compact form which reduces their representation complexity.
Nevertheless, their adoption for 3D tasks, such as Novel View Synthesis (NVS), remains limited due to the inherent incompatibilities between image latent spaces and 3D modeling methods.
These incompatibilities stem from the lack of an underlying 3D geometry in latent spaces on the one hand, and the necessity of 3D-consistency among images for solving the inverse graphics problem on the other hand.
In practice, encoding two 3D-consistent views of an object does not lead to 3D-consistent latent representations; instead, it produces latent images with features that vary inconsistently across views.
This prohibits the direct application of scene learning methods such as Neural Radiance Fields (Mildenhall et al., , NeRF) on such representations.

Furthermore, recent methods in 3D reconstruction have attempted to enable certain applications of latent spaces for NeRFs but also encountered latent 3D-inconsistency issues induced by Autoencoders (AE).
They sidestepped this issue by implementing custom, scene-dependent “adapters” (Khalid et al., ) or “refinement layers” (Park et al., ) to correct renderings of neural scene representations into standard latent image representations.
However, as these adapters are scene-dependent, they only enable a limited subset of applications.
Thus, a universal “3D-aware” image latent space, analogous to the 2D image space, remains an under-explored area of research with promising potential, as it would enable interoperability with latent-based 2D methods.

In this paper, we start by proposing a technique that enables learning NeRF models in the latent space.
We present a training pipeline comprising two stages: Latent Supervision supervises NeRFs with latent image representations, and RGB Alignment aligns the learned latent scene with the RGB space.
Yet, its application in a standard latent space results in sub-optimal latent NeRFs, due to their aforementioned incompatibilities.
To address this, we draw inspiration from the relationship between the image space and 3D space (Fig. 1a) and introduce 3D-aware latent spaces, which augment regular image latent spaces by incorporating an underlying 3D geometry.
We present an Inverse Graphics Autoencoder (IG-AE) that embeds a 3D-aware latent space (Fig. 1b).
To achieve this, we apply 3D regularization on the latent space of an autoencoder by jointly training it with a synthetic set of latent 3D scenes.
During this process, we align the encoded views with the renderings of the latent 3D scenes, which enforces 3D-consistency.
Additionally, we propose an autoencoder preservation process that simultaneously autoencodes real and synthetic data while performing our training, which allows us to conserve the reconstructive performance of the autoencoder.

We propose an open-source extension of Nerfstudio (Tancik et al., ) that enables learning its supported NeRF models in latent spaces, thereby unlocking a streamlined approach for latent NeRF learning.
This enables us to evaluate various current NeRF architectures on the task of learning scenes in latent spaces.
Our results highlight the effectiveness of our IG-AE in latent scene learning as compared to a standard AE.
We consider this work to be the first milestone towards foundation inverse graphics autoencoders, and aspire that our open-source Nerfstudio extension promotes further research in this direction.

An overview of our contributions is given below.

- •
We introduce the concept of 3D-aware latent spaces that are compatible with 3D tasks,
- •
We present an Inverse Graphics Autoencoder (IG-AE) that maps images to a 3D-aware latent space, all while preserving autoencoder performances,
- •
We propose a standardized method to train NeRF architectures in latent spaces,
- •
We propose an open-source extension of Nerfstudio that support training supported NeRF models in the latent space, in an effort to streamline future work involving latent NeRFs.
- •
We experimentally show that IG-AE enables improved latent NeRF quality with respect to the baseline AE and decreased training times with respect to regular NeRFs.

## 2 Related Work

#### Latent NeRFs.

While NeRFs (Mildenhall et al., ) were originally conceived to work in the image space, they have also been extended to work in other feature spaces (Tschernezki et al., ; Kobayashi et al., ).
Notably, Latent NeRFs are extensions of NeRFs to the latent spaces of image autoencoders.
They have been particularly explored in works targeting applications such as scene editing (Park et al., ; Khalid et al., ) and text-to-3D generation (Metzer et al., ), which are not directly feasible with regular NeRFs.
To circumvent the incompatibilities between NeRFs and latent spaces, these works have resorted to special adapter layers that correct NeRF renderings into standard latent image representations.
For novel view synthesis, Aumentado-Armstrong et al. () employ hybrid NeRFs that are trained to simultaneously render both RGB and latent components.
This is done to supervise the NeRF geometry during training, while only keeping latent components at inference, which enables good NVS performances and rendering speed-ups.
In this work, we aim to train NeRFs that operate fully within the latent space.
We address the incompatibility between NeRFs and latent spaces by tackling its root cause: the need for a inverse graphics autoencoders that preserves 3D consistency, or in other words, the need for 3D-aware latent spaces.

#### Nerfstudio.

Subsequent to the introduction of NeRF (Mildenhall et al., ), a multitude of NeRF models emerged to improve upon the original architecture by accelerating training times (Müller et al., ; Kerbl et al., ), improving rendering quality (Barron et al., ; Chen et al., ), as well as targeting other limitations (Fridovich-Keil et al., ; Martin-Brualla et al., ; Yu et al., ).
With the escalation of NeRF methods, Nerfstudio (Tancik et al., ) emerged as a unified PyTorch (Paszke et al., ) framework in which NeRF models are implemented using standardized implementations, making it straightforward for researchers and practitioners to integrate various NeRF models into their projects.
Today, most well-known NeRF techniques are implemented in Nerfstudio (Müller et al., ; Barron et al., ; Mildenhall et al., ; Kerbl et al., ), thus allowing for a friction-free experience when testing and comparing novel techniques.
While Nerfstudio currently provides a streamlined approach for NeRFs, it limits the representation of 3D scenes to the natural color space, hence limiting both the research and development of Latent NeRFs.
In this work, we propose an open-source extension of the Nerfstudio framework that supports training any Nerfstudio architecture in a custom latent space, thus facilitating our current as well as future research in this area.
Furthermore, we integrate our IG-AE into this extension, which enables us to evaluate various current NeRF architectures on the task of latent scene learning.

## 3 Latent NeRF

Figure: (a) Latent images.
Refer to caption: x3.png

In this section, we start by presenting latent NeRFs.
Then, we propose a general latent NeRF training method, which will later serve to train NeRFs in the latent space of our IG-AE.
Provided an encoder $E_{\phi}$ and decoder $D_{\psi}$, our method can train any NeRF architecture in a latent space via two stages.
Latent Supervision is an adaptation of the standard NeRF training framework to the latent space which replaces RGB images with latent images.
RGB Alignment fine-tunes the decoder to align the learned latent scene with the RGB space, which our experiments proved to be essential for strong NVS performances in the RGB space.

### 3.1 Definition

Latent NeRFs are conceptually similar to standard NeRFs, with the primary difference being that they model scenes in the latent space of an autoencoder as opposed to the RGB space.
As such, latent NeRFs are simple extensions of standard NeRF methods, where the rendering resolution and the number of output channels are modified in accordance with the latent space dimensions.
Let $F_{\theta}$ be a latent NeRF method with trainable parameters $\theta$, where $F_{\theta}\in$ {Vanilla-NeRF (Mildenhall et al., ), Instant-NGP (Müller et al., ), TensoRF (Chen et al., ), K-Planes (Fridovich-Keil et al., )}.
For conciseness, we consider $F_{\theta}$ to be a generic NeRF method and abstract from method-specific nuances.
Given a camera pose $p$, one can render a novel view as such:

$$ $\displaystyle\tilde{z}_{p}=F_{\theta}(p)~{},$ $\displaystyle\tilde{x}_{p}=D_{\psi}(\tilde{z}_{p})~{},$ (1) $$

where $\tilde{z}_{p}$ is the rendered latent image of shape $(h,w,c)$, and $\tilde{x}_{p}$ is the decoded RGB image of shape $(H,W,3)$.
We define $l>1$ as the resolution downscale factor when going from the RGB space to the latent space: $(h,w)=(\nicefrac{{H}}{{l}},\nicefrac{{W}}{{l}})$.
As NeRF rendering pipelines implement classic volume rendering (Kajiya & Von Herzen, ) where pixels are independently rendered, latent NeRFs reduce the rendering complexity by a factor of $l^{2}$ as compared to standard NeRFs.
This makes latent NeRFs highly attractive, as they alleviate a key bottleneck in NeRF training, while maintaining the same target resolution after decoding.

### 3.2 Training

Our latent NeRF training scheme consists of two stages, as illustrated in Fig. 3.

Figure: Figure 3: Latent NeRF Training. We train a Latent NeRF in two stages. First, we train the chosen NeRF method $F_{\theta}$ on posed encoded latent images using its proprietary loss $\mathcal{L}_{F_{\theta}}$ that matches rendered latents $\tilde{z}_{p}$ and encoded latents $z_{p}$. Subsequently, we align with the scene in the RGB space by adding decoder fine-tuning via $\mathcal{L}_{\mathrm{align}}$ that matches ground truth images $x_{p}$ and decoded renderings $\tilde{x}_{p}$.
Refer to caption: x5.png

#### Latent Supervision.

In this stage, we consider the direct adaptation of NeRF training in the latent space, i.e. training NeRF on latent image representations rather than RGB images.
Let $\mathcal{L}_{F_{\theta}}$ be the training loss corresponding to the NeRF method $F_{\theta}$.
$\mathcal{L}_{F_{\theta}}$ comprises of a pixel-level reconstructive objective that matches the NeRF renderings to ground truth views, as well as other method-specific regularization terms.
Latent Supervision consists of minimizing the following objective:

$$ $\mathfrak{L}_{\mathrm{LS}}(\theta)=\sum_{p\in\mathcal{P}}\mathcal{L}_{F_{ \theta}}(\theta;z_{p},\tilde{z}_{p})~{},$ (2) $$

where $z_{p}$ and $\tilde{z}_{p}$ are respectively the encoded latent representation of the RGB ground truth and the rendered latent image, with pose $p$.
$\mathcal{P}$ denotes the set of training camera poses.
Note that, to avoid redundantly encoding training views, we start this stage by encoding all the training images $\mathcal{X}=\{x_{p}\}_{p\in\mathcal{P}}$ into latent image representations $\mathcal{Z}=\{z_{p}\}_{p\in\mathcal{P}}$, and then caching them.

#### RGB alignment.

While Latent Supervision effectively captures a 3D structure in the latent space, even minor inaccuracies in latent NeRF renderings can be magnified during the decoding process.
These inaccuracies stem from various sources of error within the pipeline, mainly: errors originating from the latent space and autoencoding performances (i.e. imperfect AE), and errors associated with 3D modeling (i.e. imperfect NeRF).
Hence, in order to alleviate the effect of these errors, we finish our latent NeRF training by an RGB alignment process, where we fine-tune the decoder $D_{\psi}$ and the latent scene $F_{\theta}$ to align with the RGB space.
In practice, we minimize the following objective:

$$ $\mathfrak{L}_{\mathrm{align}}(\theta,\psi)=\sum_{p\in\mathcal{P}}\lVert x_{p}- \tilde{x}_{p}\rVert_{2}^{2}~{},$ (3) $$

where $x_{p}$ is the RGB ground truth, and $\tilde{x}_{p}=D_{\psi}(\tilde{z}_{p})$ is the decoded latent NeRF rendering.
Consequently, the latent NeRF not only exhibits good NVS performances in the latent space, but also when decoding its renderings to the RGB space.

Overall, we divide latent NeRF training into Latent Supervision and RGB Alignment.
Latent Supervision is an accelerated approach that learns 3D structures in the latent space thanks to reduced image resolutions, and RGB alignment enhances NVS quality in the RGB space by using RGB supervision and a decoder fine-tuning, while still rendering the NeRF in the latent space.
Fig. 2 (AE columns) illustrates the results of our latent NeRF training method when applied in a standard latent space.
Latent NeRFs trained in a standard latent space with our method learn a coarse geometry.
However, due to the aforementioned incompatibilities between latent spaces and NeRFs, the decoded renderings present artifacts in the RGB space.
To address this, we shift focus to resolving these incompatibilities in the next section, and present an inverse graphics autoencoder embedding a 3D-aware latent that is compatible with learning latent NeRFs.

## 4 Inverse Graphics Autoencoder

Figure: Figure 4: IG-AE Training. We jointly learn a set of latent synthetic scenes $\mathcal{T}_{\tau}$ and supervise the latent images $z_{s,p}$ of an autoencoder with rendered 3D-consistent latents $\tilde{z}_{s,p}$ using $\mathcal{L}_{\mathrm{latent}}$. We match decoded latent renderings $\tilde{x}_{s,p}$ with the ground truth scene renderings $x_{s,p}$ using $\mathcal{L}_{\mathrm{RGB}}$. We preserve autoencoder performances on synthetic and real data respectively through $\mathcal{L}_{\mathrm{ae}}^{\mathrm{(synth)}}$ and $\mathcal{L}_{\mathrm{ae}}^{\mathrm{(real)}}$.
Refer to caption: x6.png

We present IG-AE, an autoencoder that encodes 3D-consistent images into 3D-consistent latent representations.
To attain such an autoencoder, it is necessary that its latent space encodes the RGB space while also retaining an underlying 3D geometry.
In this section, we start by defining 3D-consistency (Section 4.1), and then elaborate on how we train an IG-AE: we utilize synthetic data to construct a learnable set of latent scenes which aims to supervise the latent space of an AE with 3D-consistent latents (Section 4.2), all while preserving autoencoding performances (Section 4.3).
Fig. 4 presents an overview of our method.

### 4.1 3D-Consistency

The notion of 3D consistency ensures that corresponding points or features in different images represent the same point or object in the scene, despite variations in viewpoint, lighting or occlusion.
We denote the posed images as $\mathcal{X}=\{x_{p}\mid p\in\mathrm{SE}(3)\}$, where $\mathrm{SE}(3)$ is the Special Euclidean group containing all camera poses in 3D.
$\mathcal{X}$ are posed 3D-consistent images if and only if there exists a 3D model $M$ such that:

$$ $\forall p\in\mathrm{SE}(3),~{}\mathrm{Render}(M,p)=x_{p}~{}.$ (4) $$

Note that a NeRF model inherently produces 3D-consistent images as it is an implicit model $M$ rendered via classic volume rendering (Kajiya & Von Herzen, ).
Also note that while 3D consistency is natural for posed images $\mathcal{X}=\{x_{p}\}_{p\in\mathcal{P}}$ obtained from a scene in the image space, it does not naturally extend to the latent space, as latent representations of two 3D-consistent images are not necessarily 3D consistent.
Our 3D-aware latent space is designed to mitigate this discrepancy.

### 4.2 3D-Regularization

To achieve 3D-consistency in the latent space, we learn an IG-AE by aligning the latent encodings of 3D-consistent images with reference 3D-consistent latent images.
However, this cannot be directly achieved as such 3D-consistent latent images are not available.
To this end, we learn a set of 3D latent scenes with NeRFs from which 3D-consistent latents can be easily rendered.
In fact, while NeRFs cannot replicate 3D-inconsistent latents from a given autoencoder, training them via $\mathcal{L}_{F_{\theta}}$ (which typically employs an MSE reconstructive objective) leads to a convergence towards a common coarse geometry that most satisfies the inputs, while maintaining 3D-consistency.
We utilize this property in our approach to obtain 3D-consistent latent images with which we supervise our autoencoder.
To learn our latent scenes in practice, we adopt Tri-Plane representations (Chan et al., ), as their simple architecture ensures a low memory footprint and a fast training.

Let $\mathcal{M}=\{M_{1},...,M_{N}\}$ be a dataset of 3D scenes, and $\mathcal{X}_{s}=\{x_{s,p}\}_{p\in\mathcal{P}}$ be the set of renderings of a scene $M_{s}\in\mathcal{M}$ from an array of training views $\mathcal{P}$.
We denote the set of latent scenes as the set of Tri-Planes $\mathcal{T}_{\tau}=\{T_{1},...,T_{N}\}$, where $\tau$ comprises both scene-specific trainable parameters, as well as shared parameters present in the feature decoder of our common Tri-Plane renderer $\mathcal{R}_{\mathcal{T}}$.

Before training our IG-AE, we start by encoding views of our scenes $\mathcal{M}$ into standard (i.e. 3D-inconsistent) latent views, and training our Tri-Planes on these latents.
Subsequently, we proceed to train our IG-AE while continuing to jointly train our Tri-Planes.
For each scene $M_{s}$, we encode each view $x_{s,p}$ into a latent image $z_{s,p}$, and render the corresponding 3D-aware latent image $\tilde{z}_{s,p}$ from the latent scene $T_{s}$ as follows:

$$ $\displaystyle z_{s,p}=E_{\phi}(x_{s,p})~{},$ $\displaystyle\tilde{z}_{s,p}=\mathcal{R}_{\mathcal{T}}(T_{s},p)~{},$ (5) $$

where $E_{\phi}$ is the encoder with trainable parameters $\phi$.
We then define a loss function $\mathcal{L}_{\mathrm{latent}}$ that aligns these latent representations:

$$ $\mathcal{L}_{\mathrm{latent}}(\phi,\tau;z_{s,p},\tilde{z}_{s,p})=\lVert z_{s,p }-\tilde{z}_{s,p}\rVert_{2}^{2}~{}.$ (6) $$

This loss function updates both the encoder parameters $\phi$ and the latent scene parameters $\tau$ to minimize the distance between the latent representations.
This means that, on the one hand, the latent scene $T_{s}$ is updated to align with the 3D-inconsistent latent images $z_{s,p}$, and hence learn a coarse geometry common among all latent images of $M_{s}$.
On the other hand, the encoder parameters $\phi$ are updated to produce latent images that more closely reassemble the latent scene renderings $\tilde{z}_{s,p}$, or in other words, produce 3D-consistent images.

We additionally define $\mathcal{L}_{\mathrm{RGB}}$ as a loss function that mirrors $\mathcal{L}_{\mathrm{latent}}$ in the RGB space:

$$ $\mathcal{L}_{\mathrm{RGB}}(\psi,\tau;x_{s,p},\tilde{x}_{s,p})=\lVert x_{s,p}- \tilde{x}_{s,p}\rVert_{2}^{2}~{},$ (7) $$

where $\tilde{x}_{s,p}=D_{\psi}(\tilde{z}_{s,p})$ is the decoded latent scene rendering. $\mathcal{L}_{\mathrm{RGB}}$ updates both the decoder parameters $\psi$ and the latent scene parameters $\tau$ so that the latent scene aligns with the RGB scene when decoded.
This is to ensure the optimal decoding of 3D-consistent latents as well as NVS performances in the RGB space.

Therefore, our 3D regularization objective minimizes the following loss:

$$ $\mathfrak{L}_{\mathrm{3D}}(\phi,\psi,\tau)=\sum_{s,p}\left[\lambda_{\mathrm{ latent}}\mathcal{L}_{\mathrm{latent}}(\phi,\tau;z_{s,p},\tilde{z}_{s,p})+ \lambda_{\mathrm{RGB}}\mathcal{L}_{\mathrm{RGB}}(\psi,\tau;x_{s,p},\tilde{x}_{ s,p})\right]~{},$ (8) $$

where $\lambda_{\mathrm{latent}}$ and $\lambda_{\mathrm{RGB}}$ are hyper-parameters.

Overall, our IG-AE is trained by aligning the latent space of an autoencoder with the space of latent images that is inferable via Tri-Planes, while also ensuring the proper mapping from the latent space to the RGB space.
In practice, we use synthetic scenes from Objaverse (Deitke et al., ) to learn 3D structure in the latent space, as they present well-defined geometry and error-free camera parameters.
Fig. 2 illustrates a comparison between a standard AE latent space and our 3D-aware latent space.
While latent NeRFs trained with an AE present artifacts when decoding their renderings, those trained in the latent space of our IG-AE do not exhibit such artifacts.
Yet, our experiments show that solely applying our 3D-regularization optimization objective leads to a degradation of the AE’s generalization capabilities and autoencoding performances.
On that account, we present in the next section additional components of our training that target a reconstructive objective.

### 4.3 Autoencoder Preservation

As 3D regularization does not incorporate autoencoder reconstruction, this section focuses on preserving the autoencoding performance of the AE on both synthetic and real data.
To this end, we additionally jointly learn a reconstructive objective.
First, we add an autoencoder loss to align the ground truth RGB views with the reconstructed views after encoding and decoding:

$$ $\mathcal{L}_{\mathrm{ae}}^{\mathrm{(synth)}}(\phi,\psi;x_{s,p},\hat{x}_{s,p})= \lVert x_{s,p}-\hat{x}_{s,p}\rVert_{2}^{2}~{},$ (9) $$

where $\hat{x}_{s,p}=D_{\psi}(z_{s,p})$ is the reconstruction of the ground truth image. This ensures that the autoencoder still reconstructs images from the scenes.

To avoid overfitting on synthetic data, we additionally inject real images from Imagenet (Deng et al., ) into our training pipeline, on which we also define a reconstructive objective.
We denote $\mathcal{I}=\{I_{1},...,I_{L}\}$ the dataset of real images.
We define the reconstructive loss on an image $I_{j}\in\mathcal{I}$ as follows:

$$ $\mathcal{L}_{\mathrm{ae}}^{\mathrm{(real)}}(\phi,\psi;I_{j},\hat{I}_{j})= \lVert I_{j}-\hat{I}_{j}\rVert_{2}^{2}+\lambda_{\mathrm{p}}\mathcal{L}_{ \mathrm{p}}(I_{j},\hat{I}_{j})+\lambda_{\mathrm{TV}}\mathcal{L}_{\mathrm{TV}}( k_{j})~{},$ (10) $$

where $\lambda_{\mathrm{p}}$ and $\lambda_{\mathrm{TV}}$ are hyper-parameters, $\hat{I}_{j}=D_{\psi}(k_{j})$ is the reconstruction of $I_{j}$, and $k_{j}=E_{\phi}(I_{j})$ is the encoding of $I_{j}$.
$\mathcal{L}_{\mathrm{p}}$ is a perceptual loss that refines the autoencoder reconstructions by aligning the reconstructed and original images in the feature space of a pre-trained network.
$\mathcal{L}_{\mathrm{TV}}$ is a total variation loss that acts as a regularization term to encourage spatial smoothness by penalizing high-frequency variations in the latent images.
We found that it leads to latent images that more closely reassemble our 3D-consistent latents.
More details about TV are present in Appendix A.
This is done to prevent the IG-AE from converging towards a dual-mode solution, where it is only 3D-consistent for synthetic scenes while functioning as a normal AE for real scenes.

Therefore, our autoencoder preservation loss is defined as follows:

$$ $\mathfrak{L}_{\mathrm{ae}}(\phi,\psi)=\lambda_{\mathrm{ae}}^{\mathrm{(synth)}} \sum_{s,p}\mathcal{L}_{\mathrm{ae}}^{\mathrm{(synth)}}(\phi,\psi;x_{s,p},\hat{ x}_{s,p})+\lambda_{\mathrm{ae}}^{\mathrm{(real)}}\sum_{j}\mathcal{L}_{\mathrm{ ae}}^{\mathrm{(real)}}(\phi,\psi;I_{j},\hat{I}_{j})~{},$ (11) $$

where $\lambda_{\mathrm{ae}}^{\mathrm{(synth)}}$ and $\lambda_{\mathrm{ae}}^{\mathrm{(real)}}$ are hyper-parameters.

### 4.4 IG-AE Training Objective

Overall, we present a joint training objective for our IG-AE that regularizes its latent space with 3D geometry while preserving its autoencoding performances.
Our IG-AE training objective minimizes the following loss:

$$ $\mathfrak{L}_{\mathrm{IG-AE}}(\phi,\psi,\tau)=\mathfrak{L}_{\mathrm{3D}}(\phi, \psi,\tau)+\mathfrak{L}_{\mathrm{ae}}(\phi,\psi)~{}.$ (12) $$

In summary, our training strategy exploits the well-defined geometry of synthetic data and utilizes $\mathfrak{L}_{\mathrm{3D}}$ to learn latent scenes and regularize the latent space of an AE with 3D geometry, all while maintaining the reconstruction performance of the AE via $\mathfrak{L}_{\mathrm{ae}}$ and real data.

## 5 Experiments

We evaluate our method by first training an IG-AE on synthetic scenes from Objaverse, as described in Section 4.
Subsequently, we train Nerfstudio models in our 3D-aware latent space on scenes from *out-of-distribution* datasets.
We also train Nerfstudio models on held-out scenes from Objaverse (results in Section B.3).
We compare the novel view synthesis performance of the Nerfstudio models when trained in our 3D-aware latent space (IG-AE), a standard latent space (AE), and the RGB space.
Moreover, we evaluate the auto-encoding performances of our IG-AE on held-out real images.
Finally, we assess our design choices by presenting an ablation study of our method, where we omit either our 3D-regularization components or our AE preservation components.

### 5.1 Nerfstudio Extension

We extend Nerfstudio to support latent scene learning in a code which we include in the supplementary material and will publicly release as open-source.
Our extension omits the constraint of training models in the RGB space and allows to train Nerfstudio models in any feature space.
Concurrently, we incorporate our latent NeRF training pipeline as well as IG-AE into Nerfstudio while adhering to its coding conventions, thereby enabling the training of any Nerfstudio model in our 3D-aware latent space.
We modify the ray batching process to make it correspond to full images, where the default behavior uses randomly sampled pixels.
This is necessary to be able to decode the rendered images without breaking the gradient flow.
Beyond reproducibility purposes for this paper, we hope the proposed generic extension will facilitate research on latent NeRFs and their applications.

### 5.2 Datasets

For 3D-regularization, we adopt Objaverse (Deitke et al., ), a large dataset of diverse synthetic 3D objects.
Objaverse provides meshes of synthetic objects, and thus presents no artifact in its scenes and no approximation errors in camera parameters.
We utilize $N=500$ objects from Objaverse.
Each object is rendered from $V=300$ views at a $128\times 128$ resolution.
Each view is rendered from a randomly sampled camera pose on the sphere surrounding the object.

For AE preservation, we adopt Imagenet (Deng et al., ), a large dataset of diverse real images.
We utilize $L=$40\,000$$ images, which we pre-process by doing a square cropping followed by a downscaling to a $128\times 128$ resolution that matches our rendered scenes.

For NeRF evaluations, we train NeRFs on held-out scenes from Objaverse, and on scenes from three out-of-distribution datasets: Shapenet Hats, Bags, and Vases (Chang et al., ).
All scenes are rendered with the same resolution and number of views as Objaverse.

Figure: Figure 5: Qualitative results. Visualization of decoded latent NeRF renderings trained with a standard AE and an IG-AE on scenes from three *out-of-distribution* datasets. Latent NeRFs trained with an AE exhibit artifacts in decoded renderings that are not present in those trained with IG-AE.
Refer to caption: x7.png

Table: Table 1: Main Results on ShapeNet datasets. All results are obtained by training NeRF models in Nerfstudio using our Latent NeRF Training Pipeline, and are averaged over $4$ scenes from each dataset. Our 3D-aware latent space generalizes to *out-of-distribution* datasets and is more suited for latent NeRF training.

### 5.3 Training Details

#### IG-AE training details.

We adopt the pre-trained “Ostris KL-f8-d16” VAE (Burkett, ) from Hugging Face, which has a downscale factor $l=8$, and $c=16$ feature channels in the latent space.
We apply a Total Variation (TV) regularization on our Tri-Planes with a factor $\lambda_{\mathrm{TV}}^{\mathrm{3D}}$ which prevents overfitting on train views (we exclude it from Eq. 8 for clarity).
Analogously, we also apply TV regularization on encoded real images $k_{j}$.
A detailed presentation of TV and its benefits for NVS can be found in Appendix A.
Training IG-AE takes $60$ hours on $4\times$ NVIDIA L4 GPUs. Our detailed hyper-parameter settings can be found in Section C.1.

#### Nerfstudio models training details.

We test the following models from Nerfstudio: Vanilla-NeRF (Mildenhall et al., ), Instant-NGP (Müller et al., ), TensoRF (Chen et al., ), and K-Planes (Fridovich-Keil et al., ).
For each method, the Nerfstudio framework provides a proprietary loss $F_{\theta}$, as well as a custom training procedure that includes specific optimizers, schedulers and other method-specific components.
To train a latent NeRF in Nerfstudio, we first train the chosen model for $10\,000$ iterations to minimize $\mathfrak{L}_{\mathrm{LS}}$ using the method-specific optimization process.
Subsequently, we continue the training with $15\,000$ iterations of RGB alignment by minimizing $\mathfrak{L}_{\mathrm{align}}$.
To account for the change of image representations, we modulate the learning rate of each method by a factor of $\xi_{\mathrm{LS}}$ in latent supervision, and a factor $\xi_{\mathrm{align}}$ for RGB alignment.
Section C.2 details the hyper-parameters we used in Nerfstudio, including the values of these factors for each method.

### 5.4 Results

We report for each experiment the Peak Signal-to-Noise Ratio (PSNR $\uparrow$) for pixel-level similarity, the Structural Similarity Index Measure (SSIM $\uparrow$) for structural-level similarity, and the Learned Perceptual Image Patch Similarity (Zhang et al., , LPIPS $\downarrow$) for perceptual similarity.
Quantitative and qualitative results can be found in Tables 1 and 5, where we utilize our latent NeRF training pipeline to train various Nerfstudio models on out-of-distribution datasets from ShapeNet, both using our IG-AE and a standard AE.
As illustrated, models trained with our IG-AE showcase a superior NVS performance compared to those trained with a standard AE.
This proves that our IG-AE embeds a 3D-aware latent space that is more compatible with NeRF training than a standard latent space, and that is able to generalize to novel datasets.

Section B.1 presents the evolution of the NVS performance throughout our two Latent NeRF training stages.
In Section B.2, we provide a comparison of latent NeRF training with classical RGB training.
Latent NeRF training showcases significant speedups both in terms of training and rendering times, but lower NVS performance.
This is due to a limitation in the representation of high frequencies when decoding latent NeRF renderings, as it can be seen in Fig. 5.
We leave this to future work, as our approach is the first to propose a 3D-aware latent space and to assess its performance with various NeRF architectures.

### 5.5 Ablations

To justify our design choices, we conduct an ablation study in which we alternatively omit our 3D regularization pipeline and our AE preservation components.
Ablation results on held-out Objaverse scenes for Vanilla NeRF can be found in Table 2.
“IG-AE (no Pr)” removes the AE preservation components from our training by deactivating $\mathfrak{L}_{\mathrm{ae}}$.
“IG-AE (no 3D)” omits our 3D regularizatiom by deactivating $\mathfrak{L}_{\mathrm{3D}}$.
As illustrated, IG-AE (no Pr) presents deteriorated autoencoding performance due to overfitting on the task of 3D-regularization.
IG-AE (no 3D) showcases good autoencoding performances, but lower NVS performances, as the latent space here is not 3D-aware due to the lack of 3D-regularization.
Our method (IG-AE) demonstrates both good NVS and autoencoding reconstruction performances as compared to its ablations, thereby justifying the components of our pipeline.
Note that ablating both our 3D regularization components and our AE preservation components leads back to the case of a standard autoencoder (AE).
Additional ablations done with other NeRF models can be found in Table 4 in Section B.2.

Table: Table 2: Ablations. A comparison between our proposed IG-AE and the baseline autoencoder (AE) on two tasks. First, we compare their NVS performances when used to train a latent Vanilla-NeRF. Second, we compare their reconstruction performances when auto-encoding held-out images from ImageNet. Our method presents both strong NVS and reconstruction performances as compared to its ablations, thus validating our design choices. The bold and underlined entries respectively indicate the best and second-best results.

## 6 Conclusion

In this paper, we introduce 3D-aware latent spaces and propose IG-AE, an autoencoder compatible with latent NeRF training.
Moreover, we present a latent NeRF training pipeline that brings NeRF architectures to the latent space.
We integrate our pipeline into an open-source extension of Nerfstudio, thereby enabling latent NeRF training for its supported architectures.
Extensive experiments show the notable improvements our 3D-aware latent space brings as compared to training NeRFs in a standard latent space, as well as the efficiency improvements it brings with respect to training NeRFs in the RGB space.
In concluding this paper, several directions of future work arise, as we consider this work to the first milestone towards foundation inverse graphics autoencoders.
This includes, for instance, the exploration of approaches improving the representation of high-frequency details when decoding latent NeRFs.
We hope that our proposed Nerfstudio extension, as well as our open-source codebase, will promote further research in this direction.
In this regard, Kassab et al. () propose a novel method tackling “scaled inverse graphics”, in which our 3D-aware latent space serves as a key component.

## Appendix A Tri-Plane Representations

Tri-Plane representations (Chan et al., ) are explicit-implicit scene representations enabling scene modeling in three axis-aligned orthogonal feature planes, each of resolution $K\times K$ with feature dimension $F$.
To query a 3D point $x\in\mathbb{R}^{3}$, it is projected onto each of the three planes to retrieve bilineraly interpolated feature vectors $F_{xy}$, $F_{xz}$ and $F_{yz}$.
These feature vectors are then aggregated via summation and passed into a small neural network to retrieve the corresponding color and density, which are then used for volume rendering (Kajiya & Von Herzen, ).
We adopt Tri-Plane representations for our set of latent scenes for their lightweight architectures and relatively fast training times.

#### TV regularization.

Total variation measures the spatial variation of images. It can act as a regularisation term to denoise images or ensure spatial smoothness.
Total variation on images is expressed as follows:

$$ $\mathcal{L}_{TV}(I)=\frac{1}{HW}\sum_{i,j}[\lVert I_{i,j}-I_{i-1,j}\rVert_{p}^ {q}+\lVert I_{i,j}-I_{i,j-1}\rVert_{p}^{q}]~{},$ (13) $$

where $\left\lVert\cdot\right\rVert_{p}^{q}$ is the $l_{p}$ norm to the power $q$.

Total variation regularization was adapted for inverse graphics to promote spatial smoothness in implicit-explicit NeRF representation (Fridovich-Keil et al., ; ; Chen et al., ).
Accordingly, we regularize our Tri-Planes with total variation, which we write below:

$$ $\mathcal{L}_{TV}^{\mathrm{3D}}(T)=\sum_{c}\mathcal{L}_{TV}(T^{(c)})~{},$ (14) $$

where $T^{(c)}$ represents the feature plane of index $c$ in $T$.
In practice, we add the regularization term $\lambda_{\mathrm{TV}}^{\mathrm{3D}}\sum_{i}\mathcal{L}_{TV}^{\mathrm{3D}}(T_{i})$ to our 3D regularization loss $\mathfrak{L}_{\mathrm{3D}}$ (Eq. 8), where $\lambda_{\mathrm{TV}}^{\mathrm{3D}}$ is a hyperparameter.
While this TV regularization is done on plane features, it effectively translates to the latent 3D scene, as latents are obtained from decoding these regularized features.
This leads to smooth latent renderings and smooth gradients, as we utilize TV regularization here with $(p,q)=(2,2)$, which discourages high frequency variations in features.

Additionally, we use total variation in our AE preservation loss $\mathfrak{L}_{\mathrm{AE}}$ (Eq. 10) with $(p,q)=(2,1)$ to discourage our autoencoder from producing latents with high frequency variations, while conserving sharp edges (Rudin et al., ).
In fact, we noticed during our experiments that solely using an $l_{2}$ reconstructive objective on an autoencoder leads to latents with high frequencies.
These high frequencies are inconsistent across latent encodings of the same 3D-consistent object.
Hence, to ensure the compatibility of $\mathfrak{L}_{\mathrm{ae}}^{\mathrm{(real)}}$ with the 3D regularization term $\mathfrak{L}_{\mathrm{3D}}$, we add a TV regularization on encoded latents.

## Appendix B Supplementary Results

### B.1 RGB Alignment Value

We show in Table 3 the importance of the second stage of RGB Alignment in the Latent NeRF Training Pipeline, where NVS performances in the RGB space improves after aligning latent scenes with the RGB views.

### B.2 Quantitative Results

Table 4 includes additional results on held-out scenes from Objaverse.
Table 5 compares IG-AE training and RGB training of Nerfstudio methods, in terms of NVS performance as well as training and rendering times.

### B.3 Qualitative Results

Figs. 6, 7 and 8 compares the renderings of NeRFs respectively trained on the Cake, House and Figurine scene from Objaverse using our ablated models as well as our full model.
Latent NeRFs trained with IG-AE (no 3D) present artifacts when decoding their renderings to the RGB space, as the latent space is not 3D-aware.
IG-AE (no Pr) and IG-AE both have 3D-aware latent space and present no artifacts, as previously illustrated, IG-AE (no Pr) demonstrates degraded auto-encoding performances.

Fig. 9 illustrates real images that are auto-encoded using all our models.

## Appendix C Hyperparameters

This section is dedicated to illustrate our hyperparameter settings for both our IG-AE training and our Latent NeRF training in Nerfstudio.

### C.1 IG-AE Training Settings

Table 6 details the hyperparameters taken to train our IG-AE.

### C.2 Nerfstudio Training Settings

Table 7 details the hyperparameters taken to train NeRF models in Nerfstudio.

**Table 3: RGB alignment value. We show our NVS performances after latent supervision and RGB Alignment when training NeRF models with IG-AE. NVS has better performances in the RGB space after doing our stage 2 of RGB alignment. All results are obtained on the Cake scene from the Objaverse dataset.**
|  | Latent Supervision |  | RGB Alignment |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS |  | PSNR | SSIM | LPIPS |
| Vanilla-NeRF | 24.34 | 0.870 | 0.175 |  | 34.68 | 0.967 | 0.050 |
| Instant-NGP | 22.08 | 0.826 | 0.182 |  | 28.41 | 0.917 | 0.063 |
| TensoRF | 24.12 | 0.862 | 0.175 |  | 33.06 | 0.962 | 0.043 |
| K-Planes | 22.68 | 0.842 | 0.200 |  | 33.20 | 0.962 | 0.053 |

**Table 4: Results on Objaverse scenes. All results are obtained by training the NeRF models using our Latent NeRF Training Pipeline on held-out Objaverse scenes. The IG-AE latent space is more suited for latent NeRF training compared to a standard latent space. This table also extends Table 2 by conducting our ablations on all our adopted representations.**
|  |  | Cake |  | House |  | Figurine |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | PSNR | SSIM | LPIPS |  | PSNR | SSIM | LPIPS |  | PSNR | SSIM | LPIPS |
| Vanilla-NeRF | AE | 33.18 | 0.962 | 0.053 |  | 31.71 | 0.946 | 0.031 |  | 31.46 | 0.972 | 0.017 |
| IG-AE (no Pr) | 33.87 | 0.966 | 0.050 |  | 34,08 | 0,961 | 0,022 |  | 33.73 | 0.9821 | 0.012 |  |
| IG-AE (no 3D) | 32.09 | 0.951 | 0.063 |  | 30,56 | 0,923 | 0,046 |  | 30.94 | 0.9650 | 0.021 |  |
| IG-AE | 34.68 | 0.967 | 0.050 |  | 33,10 | 0,954 | 0,027 |  | 33.10 | 0.979 | 0.013 |  |
| Instant-NGP | AE | 24.69 | 0.860 | 0.122 |  | 22.67 | 0.784 | 0.196 |  | 24.62 | 0.908 | 0.086 |
| IG-AE (no Pr) | 28.04 | 0.917 | 0.052 |  | 27,22 | 0,882 | 0,055 |  | 26.46 | 0.9297 | 0.043 |  |
| IG-AE (no 3D) | 27.17 | 0.904 | 0.087 |  | 23,34 | 0,805 | 0,189 |  | 24.48 | 0.9078 | 0.091 |  |
| IG-AE | 28.41 | 0.917 | 0.063 |  | 27,04 | 0,882 | 0,064 |  | 27.20 | 0.941 | 0.040 |  |
| TensoRF | AE | 28.79 | 0.928 | 0.046 |  | 27.31 | 0.904 | 0.066 |  | 27.60 | 0.949 | 0.037 |
| IG-AE (no Pr) | 34.30 | 0.966 | 0.041 |  | 32,92 | 0,952 | 0,025 |  | 32.86 | 0.9780 | 0.012 |  |
| IG-AE (no 3D) | 29.56 | 0.936 | 0.094 |  | 27,99 | 0,891 | 0,083 |  | 28.89 | 0.9515 | 0.034 |  |
| IG-AE | 33.06 | 0.962 | 0.043 |  | 31,78 | 0,951 | 0,026 |  | 31.36 | 0.972 | 0.018 |  |
| K-Planes | AE | 31.82 | 0.954 | 0.056 |  | 30.06 | 0.927 | 0.038 |  | 30.45 | 0.965 | 0.021 |
| IG-AE (no Pr) | 34.75 | 0.966 | 0.049 |  | 33,22 | 0,949 | 0,026 |  | 33.67 | 0.9796 | 0.011 |  |
| IG-AE (no 3D) | 29.47 | 0.935 | 0.103 |  | 29,75 | 0,902 | 0,088 |  | 30.20 | 0.9602 | 0.027 |  |
| IG-AE | 33.53 | 0.963 | 0.052 |  | 31,73 | 0,940 | 0,031 |  | 32.44 | 0.9751 | 0.014 |  |

**Table 5: Comparison with RGB training. Bringing NeRFs to the IG-AE latent space entails significantly lower training and rendering times for most methods. Rendering times represent an average over 1000 image renderings. RGB methods render images at a $128\times 128$ resolution, whereas latent methods render at $16\times 16$. Note that for latent NeRF methods, decoding takes an additional 6.7 ms for each method to obtain the final $128\times 128$ images. Training and rendering time is measured using a single NVIDIA L4 GPU. As for rendering quality (NVS), some methods are more compatible with training in latent spaces than others, which we see as a direction of future improvements. NVS metrics are averaged over three Objaverse scenes: Cake, House and Figurine.**
|  | Training<br>Space | Training<br>Time (min) | Rendering<br>Time (ms) | NVS |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS |  |  |  |
| Vanilla NeRF | RGB | 637 | 1201 | 40.78 | 0.993 | 0.003 |
| IG-AE | 28 | 19.7 | 33.60 | 0.966 | 0.030 |  |
| Instant NGP | RGB | 6 | 11.3 | 37.78 | 0.985 | 0.006 |
| IG-AE | 19 | 7.3 | 27.47 | 0.913 | 0.056 |  |
| TensoRF | RGB | 92 | 101.4 | 40.65 | 0.992 | 0.003 |
| IG-AE | 20 | 10.4 | 32.04 | 0.961 | 0.029 |  |
| K-Planes | RGB | 88 | 63.3 | 32.98 | 0.964 | 0.027 |
| IG-AE | 29 | 11.3 | 32.36 | 0.958 | 0.033 |  |

Figure: Figure 6: Qualitative comparison. NeRF renderings of the Cake scene from Objaverse.
Refer to caption: x8.png

Figure: Figure 7: Qualitative comparison. NeRF renderings of the House scene from Objaverse.
Refer to caption: x9.png

Figure: Figure 8: Qualitative comparison. NeRF renderings of the Figurine scene from Objaverse.
Refer to caption: x10.png

Figure: Figure 9: Qualitative comparison of reconstruction quality when autoencoding. IG-AE (no Pr) presents artifacts when auto-encoding images. IG-AE (no 3D) omits these artifacts but does not embed a 3D-aware latent space. IG-AE omits the artifacts present in IG-AE (no Pr), while still integrating a 3D-aware latent space.
Refer to caption: x11.png

**Table 6: IG-AE hyperparameters. Hyperparameters used for our IG-AE training. More detailed information can be found in the configuration files of our open-source code.**
| Parameter | Value |
| --- | --- |
| General |  |
| Number of scenes $N$ | $500$ |
| Pretraining epochs | $50$ |
| Training epochs | $75$ |
| Loss |  |
| $\lambda_{\mathrm{latent}}$ | $1$ |
| $\lambda_{\mathrm{RGB}}$ | $1$ |
| $\lambda_{\mathrm{TV}}^{\mathrm{3D}}$ | $1\times 10^{-4}$ |
| $\lambda_{\mathrm{ae}}^{\mathrm{(synth)}}$ | $0.1$ |
| $\lambda_{\mathrm{ae}}^{\mathrm{(real)}}$ | $0.1$ |
| $\lambda_{\mathrm{p}}$ | $0.1$ |
| $\lambda_{\mathrm{TV}}$ | $1\times 10^{-4}$ |
| Optimization |  |
| Optimizer | Adam |
| Batch size (scene views) | $12$ |
| Batch size (real images) | $3$ |
| Learning rate (encoder) | $5\times 10^{-5}$ |
| Learning rate (decoder) | $5\times 10^{-5}$ |
| Learning rate (Tri-Planes) | $1\times 10^{-4}$ |
| Scheduler | Exponential Decay |
| Decay factor | $0.988$ |

**Table 7: Latent NeRF training hyperparameters. Hyperparameters taken to train our Latent NeRFs in Nerfstudio. More detailed information can be found in the configuration files of our open-source Nerfstudio extension.**
| NeRF model | Parameter | Value |
| --- | --- | --- |
| Any | Latent Supervision iterations | $10\,000$ |
| RGB Alignment iterations | $15\,000$ |  |
| Batch size | 4 |  |
| NeRF learning rate | Nerfstudio default |  |
| NeRF optimizer | Nerfstudio default |  |
| NeRF scheduler | Nerfstudio default |  |
| Decoder learning rate | $1\times 10^{-4}$ |  |
| Decoder optimizer | Adam |  |
| Decoder scheduler | Exponential decay |  |
| Decoder decay factor | 0.9996 |  |
| Vanilla NeRF | $\xi_{\mathrm{LS}}$ | $0.1$ |
| $\xi_{\mathrm{align}}$ | $1$ |  |
| Instant-NGP | $\xi_{\mathrm{LS}}$ | $0.001$ |
| $\xi_{\mathrm{align}}$ | $0.1$ |  |
| TensoRF | $\xi_{\mathrm{LS}}$ | $0.01$ |
| $\xi_{\mathrm{align}}$ | $0.1$ |  |
| K-Planes | $\xi_{\mathrm{LS}}$ | $0.1$ |
| $\xi_{\mathrm{align}}$ | $0.1$ |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. Is there any convergence issue during IG-AE training? Do you simply jointly train all modules with all losses?
2. I haven't checked the computation of latent NeRF, so I'm not sure if there would be any problems.
3. Is this paper an improved version of the one in the supplementary? There seems to be quite some similarities.
soundness: 3
strengths: 1. 3D-awareness of 2D image generation from autoencoders is an important issue, and this paper seems to be the first work to address it.
2. The proposed latent NeRF operates fully within the latent space, which is a standardized solution and can work as an open-source extension to established NeRF architectures. The authors submitted the code in the supplementary.
3. The training framework of latent NeRF and IG-AE is sensible, with detailed ablation study to justify the loss design.
4. The paper is very well-written and easy to follow, with method design and experiments to support its motivation and arguments.
summary: This paper introduces the concept of 3D-awareness into the latent space of autoencoders, through learning a latent NeRF and an Inverse Graphics AutoEncoder (IG-AE). The proposed latent NeRF is general and can be used as standard extension to different previous NeRF architectures, with implementation code in the supplementary. The proposed IG-AE jointly train latent NeRFs and standard AE to achieve 3D awareness in latent space, on both synthetic and real data. The experimental comparisons and analysis are mostly thorough and comprehensive, while still with some issues in justifying its motivations.
weaknesses: Although I agree with the importance of 3D-awareness of 2D autoencoders and appreciate the authors' efforts, I still have some concerns/questions for the proposed method to address 3D-awareness with latent NeRF:
1. Is a latent NeRF really necessary? Does it bring more advantage or more damage to the standard 2D AEs, especially when the scenes are quite complicated? Learning a scene with a NeRF model tends to smooth out high-frequency details (easier to be 3D-inconsistent), which is also true for TV loss as discussed in Line 363. Is it possible that the latent NeRF learning and TV loss would force the encoder to remove high-frequency contents during joint training, which can not be recovered by the decoder? One example is the cake in Figure 6, IG-AE is over-smoothed comparing to RGB. Could you conduct quantitative evaluation to show the level of loss in high frequencies?
2. Additionally, if the intension is to bring 3D-awareness to 2D autoencoders, why not add NeRF constraints on the final RGB images instead of the latents? Maybe this would cause less information loss? Is it possible to compare the trade-off between using latent NeRF and NeRF in final RGB space?
3. On the condition of the lost high-frequency details, is this level of "3D-awareness" still helpful for the 2D autoencoders? The encoding and decoding reconstruction on natural images by 2D encoders seems to be fine for now, as shown in Figure 9 and other literatures. The problem emerges when it comes to novel image generation rather than simple reconstruction, which is untouched in this paper.
4. The experiments seem to be limited to simple objects? The Objaverse and Shapenet objects should be much simpler than the original domain of the pre-trained "Ostris KL-f8-d16" VAE. Is it possible to show the performance on more complicated scenes with rich high-frequency details?
5. If the focus is to to improve standard 2D autoencoders then the comparisons to AE should be the reconstruction of multiview images?
6. If the justification is novel view synthesis, the baseline methods should be other (latent) NeRF methods instead of AE? Since AE itself is not for this task. Table 5 does show some advantages in training and rendering time, but with significant sacrifice on NVS performance.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: (1) In the first stage each latent tri-plane is trained individually, hence each tri-plane will have its own finetuned version of the image decoder. In section 4.2, when jointly training the IG-AE with the latent triplanes, which initialization is used for the decoder for IG-AE?

(2) Regarding the experiment setup: how many held-out scenes in each category (Cake, Figurine and House) for the Objeverse dataset? Does the same category appears in the training data of IG-AE ?

(2) In Table 5, it is shown that the InstantNGP exhibits both longer training time and rendering time paired with IG-AE (including the RGB decoding time for IG-AE). Given that the quality under IG-AE has no advantages, what are the advantages for the proposed IG-AE method compared with directly training NeRF using InstantNGP?
soundness: 3
strengths: - The paper attempts to research on an important problem of 3D-aware latent spaces.

- The proposed method overall makes sense: the introduce of individually trained latent tri-planes provides a auxiliary variable which serves as a "3D-aware" guidance for auto-encoded latents. 

- Experiments show that the proposed IG-AE is better for training NeRFs than vanilla AE.

- The method can easily integrated into NeRFStudio with an open-source extension.
summary: This paper propose a two-stage pipeline to train a latent space which is suitable for fitting NeRFs. In the first stage, a set of latent tri-planes are individually trained for each scene in the training data. The latent tri-planes are fitted by minimizing both the volumetric rendering error in latent space and the pixel difference in RGB space through a image decoder. In the second stage, an autoencoder is trained on all scenes in the training data, jointly with all individual tri-planes. The fidelity of the autoencoder is additionally regularized by training on additional real images. Experiments demonstrated that the proposed latent space is able to fitting NeRFs better than a image-based latent space.
weaknesses: My majority concerns are as follows:

(1) While the proposed method is interesting and makes sense, the claimed property of "3D-aware" latent space cannot be fully justified from the given experiment results:
- (a) The proposed method is only tested on dataset with limited variations. For evaluation NeRFs, ShapeNet dataset is not a best choice as it contains simple shapes and textures without any non-Lambertian effects. Additionally, the test dataset contains only three categories for Shapenet and Objeverse dataset. Evaulations on more complicated dataset with non-lambertian effects, such as Realistic Synthetic 360 [1] would help to more thoroughly demonstrate the "3D-aware" property of the latent space across a wider range of scenarios and make the results more convincing.
- (b) Even under the limited test dataset, the performance of the proposed method are not convincing enough. Despite the IG-AE outperforms the vanilla AE counterpart, the quality degrades significantly compares to the RGB version. This raises the question that whether the image fidelity is enough for the IG-AE.
- (c) The "3D-aware" property cannot be easily judged from given metrics and results in the paper. All quantitative results are averaged across views - it is hard to know whether the "consistency" between different synthesized views are preserved. One suggestion would be using some perceptal metrics to evaluate the consistency between different generated view, e.g., the CLIP feature similarity used in [2] and [3]. There are also very few qualitative results provided. Providing more qualitative results such as videos showing a rotated object can be also useful to visualize the view consistency.
- (d) The effect of the original auto-encoder utilized for training IG-AE is not well explored. As discussed in Section 3.2 in [4], a latent space with low channel-wise depth (e.g., down to 4) and a slightly higher (but still significantly lower than RGB input resolution, 32 or 64 for example) will encourage local dependency over the autoencoder’s image and latent spaces. Under such circumstance, the latent representation is a near patch level representation of its corresponding RGB image, making it nearly equivariant to spatial transformations of the scene. In other words, it automatically (at least to some extent) has the property of "3D-aware" for training NeRFs. I would suggest the paper add more discussion and/or experiments regarding the 2D autoencoder used.

(2) There has been many attempts for training NeRFs (or similar implicit 3D representations) in a space with reduced resolution, followed with upsampler operations in 2D space. For example:
- Rodin [5] first generates a low-resolution tri-planes with 3D aware convolutions and an upsampler. 
- StyleNeRF [6] generates a low-resolution latent features using NeRF and then upsample these features in 2D space with a StyleGAN-like structures.

None of these related methods are discussed and compared in this paper, making the judgement of "our approach is the first to propose a 3D-aware latent space" difficult. The suggestion would be add more discussion to clarify how the proposed approach differs from or improves upon these existing methods that used reduced resolution spaces for modeling NeRFs.

References:

[1] Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[2] Qian, Guocheng, et al. "Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors." The Twelfth International Conference on Learning Representations.

[3] Melas-Kyriazi, Luke, et al. "Realfusion: 360 reconstruction of any object from a single image. In 2023 IEEE." CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vol. 1. 2023.

[4] Metzer, Gal, et al. "Latent-nerf for shape-guided generation of 3d shapes and textures." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[5] Wang, Tengfei, et al. "Rodin: A generative model for sculpting 3d digital avatars using diffusion." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[6] Gu, Jiatao, et al. "Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis." arXiv preprint arXiv:2110.08985 (2021).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In section 3, is the same VAE used as in section 4? The implementation details have not mentioned this.
2. How is the "3D aligned" auto-encoder useful in the downstream tasks?
3. In Tab. 2, how is the NVS task performed? Given a view as the input to your encoder, then synthesis novel views directly? Would you include more visual results?
4. In your AE/VAE design, the latent space is still a "3D-aware" image, so why not using a truly 3D-aware latent such as the latent triplane in LN3Diff / Direct3D?
soundness: 2
strengths: 1. **Novel Use of Inverse Graphics in Latent Space**: The authors claim that they are the first to explore this direction, which explores the relatively untapped area of applying inverse graphics in 2D latent spaces, which reduces training and rendering complexity and offers compatibility with other latent-based 2D methods.

2. **Integration of 3D Geometry into Latent Spaces**: The authors address the issue of the lack of 3D geometry in standard image latent spaces by regularizing the autoencoder with 3D information, aligning its latent space with jointly trained 3D scenes. This approach enhances the representation capability of latent spaces for 3D scene understanding. From many perspectives, these efforts will be meaningful in this area.

3. **Latent NeRF Training Pipeline**: Two new training pipelines are proposed. By creating a latent NeRF training pipeline and integrating it within the Nerfstudio framework, the paper unlocks efficient latent scene learning and makes it accessible to a broader set of methods, providing a practical tool for further research.

4. **Open-Source Contribution**: The open-source extension to the Nerfstudio framework adds value to the research community by facilitating reproducibility and further experimentation, and the proposed method can naturally work on any new NeRF representations.
summary: This paper introduces an Inverse Graphics Autoencoder (IG-AE) to explore the potential of inverse graphics in 2D latent spaces, which is relatively underutilized in computer vision. The authors highlight that inverse graphics applied in latent spaces can reduce both training and rendering complexity while enabling compatibility with other 2D latent-based methods. A key challenge addressed is that traditional image latent spaces lack 3D geometry, preventing direct application of inverse graphics. IG-AE addresses this by aligning the latent space of an image autoencoder with jointly trained latent 3D scenes. This enables a "latent NeRF" training pipeline, implemented within the Nerfstudio framework, making latent scene learning accessible for methods supported by the framework. Experiments show that latent NeRFs trained with IG-AE offer improved quality compared to conventional autoencoders, while also providing faster training and rendering compared to NeRFs trained in image space.
weaknesses: 1. **The writing is pretty hard to follow.** I tried to understand this paper by reading over and over again, but still find it very hard to follow. Since this direction is relatively new, I strongly suggest the author to revise the writing in the later version for clearer elaboration.

2. **The motivation of the proposed method is unclear.** From my understanding, the proposed method try to align the NeRF rendering to a pretrained autoencoder (Ostris KL-f8-d16 VAE here), but what are the benefits of doing this? Though promising PSNR/LPIPS metrics are achieved as in Tab. 1 and Tab. 2, does the proposed IG-AE / AE supports zero-shot novel view synthesis on the imagenet dataset? Regarding Tab. 2, I don't quite understand how the author evaluate the novel view synthesis pipeline here, since no visual results are included in the main paper. All the visual results are just reconstructing the input view.

3. **Lack of comparison with existing methods**. Though the author claims this idea is quite novel and they are among the first to propose this solution, many existing methods have already shown similar spirits, including 

* SRT: Scene Representation Transformer.
* NeRF-VAE: A Geometry Aware 3D Scene Generative Model
* LN3Diff: Scalable Latent Neural Fields Diffusion for Speedy 3D Generation

Their 3D (V)AE models support 3D view synthesis on the open vocabulary single-view inputs. The discussions / comparisons with these methods are needed to demonstrate the soundness of the proposed pipeline.

Meta-review: The authors present a method to align a 2D image autoencoder with a 3D latent space. The authors demonstrate that the resulting latent space leads to better NVS quality with less 2D artifacts.

The reviewers converged to positively-leaning scores of 6/6/6 over the course of discussion, appreciating the interesting new idea and novelty of this work and acknowledging the clear shown benefit. At the same time, questions regarding relevancy remain.

I decided to follow the reviewers recommendation and propose to accept the paper. I think this is an interesting paper that can spark some novel ideas.

Decision: Accept (Poster)