# Fused-Planes: Improving Planar Representations for Learning Large Sets of 3D Scenes

Karim Kassab, Antoine Schnepf, Jean-Yves Franceschi, Laurent Caraffa, Flavian Vasile, Jeremie Mary, Andrew I. Comport, Valérie Gouet-Brunet

► To cite this version:

Karim Kassab, Antoine Schnepf, Jean-Yves Franceschi, Laurent Caraffa, Flavian Vasile, et al., Fused-Planes: Improving Planar Representations for Learning Large Sets of 3D Scenes. 2025. [hal-04761470v2](https://hal.science/hal-04761470v2)

HAL Id: hal-04761470

https://hal.science/hal-04761470v2

Preprint submitted on 31 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Fused-Planes: Improving Planar Representations
for Learning Large Sets of 3D Scenes

Karim Kassab¹²* Antoine Schnepf¹³* Jean-Yves Franceschi¹ Laurent Caraffa²

Flavian Vasile¹ Jeremie Mary¹ Andrew Comport³ Valérie Gouet-Brunet²

## Abstract

To learn large sets of scenes, Tri-Planes are commonly employed for their planar structure that enables an interoperability with image models, and thus diverse 3D applications. However, this advantage comes at the cost of resource efficiency, as Tri-Planes are not the most computationally efficient option. In this paper, we introduce *Fused-Planes*, a new planar architecture that improves Tri-Planes resource-efficiency in the framework of learning large sets of scenes, which we call “*multi-scene inverse graphics*”. To learn a large set of scenes, our method divides it into two subsets and operates as follows: (i) we train the first subset of scenes jointly with a compression model, (ii) we use that compression model to learn the remaining scenes. This compression model consists of a 3D-aware latent space in which Fused-Planes are learned, enabling a reduced rendering resolution, and shared structures across scenes that reduce scene representation complexity. Fused-Planes present competitive resource costs in multi-scene inverse graphics, while preserving Tri-Planes rendering quality, and maintaining their widely favored planar structure. Our codebase is publicly available as open-source. Our project page can be found at https://fused-planes.github.io.

## 1. Introduction

Learning large sets of scenes is essential for various 3D applications. Recent works in 3D generative models (Müller et al., 2023; Erkoç et al., 2023; Shue et al., 2023; Liu et al., 2024) commonly involve a preliminary stage where large sets of scene representations (Mildenhall et al., 2020; Chan et al., 2022b) are trained. These sets are then used to train

the generative model. Similarly, works in feed-forward 3D reconstruction (Hamdi et al., 2023; Wang et al., 2023) also require using large sets of scenes in order to train models that infer scene representations from captured images. Moreover, recent works training scene representations to learn large-scale scenes (Tancik et al., 2022) typically decompose them into individually trained representations, hence also requiring the simultaneous training of many scene representations. On a different note, this problem emerges in real-world use cases such as modeling product inventories or creating 3D assets for virtual reality applications. Yet, despite its prevalence, the process of learning such large sets of scenes, which we call “*multi-scene inverse graphics*” (MSIG), is widely recognized as computationally demanding and poses significant challenges in terms of resource requirements (Liu et al., 2024; Wang et al., 2023, Sections 4.1 and 5).

To learn large sets of scenes, Tri-Planes (Chan et al., 2022b) have been the preferred representations in many recent works (Shue et al., 2023; Liu et al., 2024; Chen et al., 2023; Chan et al., 2022b; Gupta et al., 2023). This preference stems from the planar structure of Tri-Planes, which closely resembles 2D images, hence providing an inter-operability with existing image-based methods. This inter-operability enables diverse applications, such as scene generation using image models (Shue et al., 2023; Chen et al., 2023) and feed-forward scene reconstruction (Wang et al., 2023; Hamdi et al., 2023). Computationally, Tri-Planes present a trade-off between explicit scene representations (Sun et al., 2022a;b) that offer fast training times but high dimensionality and memory usage, and purely implicit scene representations (Mildenhall et al., 2020) (MLPs) which require fewer trainable parameters but an expensive training. However, this trade-off remains insufficient, as the resource costs required by Tri-Planes remain prohibitive for the framework of MSIG.

In this paper, we present *Fused-Planes*, an adapted planar representation that alleviates Tri-Planes resource costs for multi-scene inverse graphics. Our primary strategy is to compact the information required for individual scenes when learning large sets of scenes. To this end, our Fused-Planes representation splits a scene into two separate compo-

<footnote>*Equal contribution †Criteo AI Lab, Paris, France ‡LASTIG, Université Gustave Eiffel, IGN-ENSG, F-94160 Saint-Mandé §Université Côte d'Azur, CNRS, I3S, France.</footnote>
<figure><img src="image_1.png" /><figcaption>**Figure 1.** Overview: NeRF methods for MSIG. Comparison of resource costs and rendering quality across recent works when training a scene. Circle sizes represent the NVS quality. Our method presents the lowest training time and memory footprint among all planar representations, while maintaining a similar rendering quality. Fused-Planes-ULW presents the lowest memory requirement. Additional details can be found in Section 4.1.</figcaption></figure>

nents: the first “Micro” component learns features specific to the scene at hand; the second “Macro” component is a learned decomposition over a set of base planes, where each plane encapsulates general information about the dataset. Concurrently, we learn our scenes in a 3D-aware latent space (Anonymous, 2025), thereby alleviating the NeRF rendering bottleneck, and accelerating our training. Practically, our method uses these components for learning a large set of scenes by dividing it into two subsets: one that is first learned jointly with the 3D-aware latent space and the shared base features, and the other that is then learned with these trained components.

We conduct extensive experiments to evaluate our method in terms of both resource costs and rendering quality when learning large sets of scenes, and compare it under the same conditions against current methods. We further provide an ablation study of our method, hence justifying our design choices. As illustrated in Figure 1, our method achieves state-of-the-art efficiency in resource costs for planar scene representations while maintaining similar rendering quality.

## 2. Related Work

This section elaborates on existing literature related to our proposed method, Fused-Planes (Section 2.1), and more generally the framework of MSIG (Section 2.2).

### 2.1. Fused-Planes

Fused-Planes reduces the *resource costs* to learn scenes by leveraging *shared representations* and learning scenes in a *3D-aware latent space*. This subsection discusses related works pertinent to these concepts.

**NeRF resource reduction.** Neural Radiance Fields (Mildenhall et al., 2020, NeRFs) achieve notable performances on the task of Novel View Synthesis (NVS) by adopting a purely implicit representation to model scenes. This requires little memory to store the scene, but high training times. Some later works have traded off compute time for memory usage by explicitly storing proxy features for the emitted radiances and densities in voxel-based representations (Sun et al., 2022a; Chen et al., 2022; Yu et al., 2021; Müller et al., 2022; Fridovich-Keil et al., 2022)), gaussian splats (Kerbl et al., 2023), or plane-based representations (Fridovich-Keil et al., 2023; Cao & Johnson, 2023). Fused-Planes continues in this direction and improves upon plane-based representations in terms of resource costs when learning numerous scenes. Figure 1 provides an overview of the resource costs of current NeRF methods.

**Shared representations.** Fused-Planes utilize representations shared across scenes as a means to compact information. This approach is reminiscent of previous work learning a single implicit representation to model several scenes. Singh et al. (2024) model multiple scenes in a single NeRF by learning pseudo-labels for each scene and training it while preserving the information of previously learned scenes. Their approach however does not scale well beyond 20 scenes. Jang & Agapito (2021) represent multiple scenes in a single NeRF by conditioning it with two latent codes. Schwarz et al. (2021) and Niemeyer & Geiger (2021) condition a shared NeRF in a similar manner but apply it instead for scene generation. In contrast to previous works, we represent common structure through an *explicit* decomposition over *multiple* planar representations, enabling higher model expressivity and faster training times. To the best of our knowledge, our approach is the first to utilize shared representations with planar structures.
**Latent NeRFs.** We train Fused-Planes in the latent space of an image autoencoder, which makes them latent scene representations. Several recent work have utilized Latent NeRFs for 3D generation (Metzer et al., 2023; Seo et al., 2023; Ye et al., 2023; Chan et al., 2023), scene editing (Khalid et al., 2023; Park et al., 2024), and scene modeling (Aumentado-Armstrong et al., 2023). As latent spaces are not directly compatible with NeRF learning, these works have resorted to special scene-dependent adaptations. To counter this, Anonymous (2025) learn a universal 3D-aware latent space compatible with latent NeRF training. This enables the modeling of numerous scenes within a common latent space. We thus choose to build a 3D-aware latent space in a similar fashion to Anonymous (2025), in which we train our Fused-Planes.

## 2.2. Multi-Scene Inverse Graphics

Multi-scene Inverse Graphics extends traditional inverse graphics to the context of learning many scenes. To tackle MSIG, many existing approaches could potentially be applied, including some related to Fused-Planes mentioned in Section 2.1. We elaborate on such approaches in this section. Appendix A provides an overview of these works, highlighting the strengths and limitations of each approach, and comparing their experimental setting.

**Single-scene methods.** The most straightforward approach to learn a set of scenes is to repeatedly and independently train single-scene representations (Mildenhall et al., 2020; Müller et al., 2022; Chen et al., 2022; Fridovich-Keil et al., 2023; Kerbl et al., 2023; Chan et al., 2022b). This approach is viable thanks to the aforementioned advancements in NeRF resource reduction, but is usually only advantageous when the number of scenes to learn is relatively small. To accelerate training in the context of multiple scenes, works like Tancik et al. (2021) and Chen & Wang (2022) obtain enhanced NeRF initializations with meta-learning, enabling them to train a scene with fewer optimization steps (Tancik et al., 2021) in a few-view setting.

**Feed-forward methods.** Feed-forward approaches could also potentially be used to tackle MSIG (Wang et al., 2023; Anciukevičius et al., 2023; Szymanowicz et al., 2023; Hamdi et al., 2023), as they side-step the need for per-scene optimizations. They train a single neural network to produce scene representations from captured scene views in a single forward pass. This allows for inferring new scenes orders of magnitude faster than optimization-based single-scene methods. However, these approaches showcase three limitations for MSIG: (i) the initial cost of training the feed-forward model is particularly high, making them only competitive when the number of scenes to learn is very large, (ii) they are limited to few-view settings which leads to lower quality

scenes, (iii) they may require the construction of a large set of scenes to train the feed-forward model (Hamdi et al., 2023), which would require other MSIG methods.

**Many-scene methods.** Although the aforementioned works could be applied for MSIG, the methods discussed in this paragraph were specifically proposed for this purpose. As opposed to feed-forward methods, many-scene methods directly optimize their neural scene representations. The aforementioned CodeNeRF (Jang & Agapito, 2021) uses a common NeRF model to learn a large set of scenes. This enables the efficient representation of scene information by disentangling the common structure shared across scenes and scene-specific features, a strategy we also employ in Fused-Planes. While this makes CodeNeRF memory efficient, its quality remains limited. On a different note, Hamdi et al. (2023, SPARF) construct a dataset of NeRFs by employing a sparse representation based on Plenoxels (Fridovich-Keil et al., 2022), which leads to a fast and memory-efficient voxel grid representation. To train a 3D generative model, Chan et al. (2022a) propose a planar representation (Tri-Planes) which is lightweight, expressive, and compatible with off-the-shelf 2D CNNs. As discussed in Section 1, Tri-Planes have become widely popular thanks to their planar structure. Nevertheless, they demonstrate significantly higher resource costs as compared to current state-of-the-art approaches. Fused-Planes improves upon Tri-Planes by reducing resource costs, matching recent methods in training time and SPARF in memory efficiency, while retaining a planar structure (Figure 1).

## 3. Method

In this section, we present our method for multi-scene inverse graphics. Section 3.1 presents our Fused-Planes architecture, which compacts information by splitting a scene representation into a scene-specific “micro” component and a shared “macro” component derived from base representations shared across scenes. This architecture is trained in a 3D-aware latent space. Section 3.2 presents our full training strategy to tackle MSIG. Our approach consists of dividing the large set of scenes into two subsets. The first subset is learned jointly with the 3D-aware latent space, and benefits from light training costs. Subsequently, we leverage the trained components to learn the second subset with even lighter training costs.

We denote $S = \{S_1, ..., S_N\}$ a large set of $N$ scenes drawn from a common distribution. Each scene $S_i = \{(x_{i,j}, p_{i,j})\}_{j=1}^V$ consists of $V$ posed views. Here, $x_{i,j}$ and $p_{i,j}$ respectively denote the $j$-th view and pose of the $i$-th scene $S_i$. We denote $\mathcal{T} = \{T_1, ..., T_N\}$ the set of scene representations modeling the scenes in $S$. We subdivide $S$ and $\mathcal{T}$ into two subsets $(S_1, S_2)$ and $(\mathcal{T}_1, \mathcal{T}_2)$ at random,
<figure><img src="image_2.png" /><figcaption>**Figure 2. Fused-Planes architecture and training framework.** We learn a set of Fused-Planes $\mathcal{T} = \{T_i\}$ in the latent space of an autoencoder, denoted by the encoder $E_\phi$ and the decoder $D_\psi$. Hence, Fused-Planes render latent images $\tilde{z}_{i,j}$ with reduced resolution, enabling faster rendering and training. Each Fused-Plane $T_i$ is split into a micro plane $T_i^{\text{mic}}$ which captures scene specific information, and a macro plane $T_i^{\text{mac}}$ computed via a weighted summation over $M$ shared base planes $\mathcal{B}$, with weights $W_i$. The shared planes $\mathcal{B}$ capture common structure across scenes. To learn our set of Fused-Planes, we start by training a first subset of micro planes $T_1^{\text{mic}}$, their corresponding weights $W_i$ and the base planes $\mathcal{B}$, jointly with the encoder $E_\phi$ and decoder $D_\psi$. Subsequently, we learn the remaining scenes by training the micro planes $T_2^{\text{mic}}$ and their corresponding weights $W_i$ while fine-tuning $\mathcal{B}$ and $D_\psi$.</figcaption></figure>

respectively containing $N_1$ and $N_2$ scenes, with $N_1 < N_2$.

### 3.1. Fused-Planes architecture

**Pre-requisite: Tri-Planes.** Tri-Plane representations (Chan et al., 2022a) are explicit-implicit scene representations enabling scene modeling in three axis-aligned orthogonal feature planes, each of resolution $K \times K$ with feature dimension $F$. To query a 3D point $x \in \mathbb{R}^3$, it is projected onto each of the three planes to retrieve bilinearly interpolated feature vectors. These feature vectors are then aggregated via summation and passed into a small neural network with parameters $\alpha$ to retrieve the corresponding color and density, which are then used for volume rendering (Kajiya & Von Herzen, 1984).

**Fused-Planes architecture.** We introduce Fused-Planes, a novel planar representation that builds upon Tri-Planes. Fused-Planes splits planar representations into scene-specific features, and features representing global structures,

which allows to learn a common structure across a large set of scenes. In particular, Fused-Planes split a planar representation $T_i$ into “micro” planes $T_i^{\text{mic}}$ integrating scene specific information, and “macro” planes $T_i^{\text{mac}}$ that encompass global information, as follows:

$$T_i = T_i^{\text{mic}} \oplus T_i^{\text{mac}}, \qquad (1)$$

where $\oplus$ concatenates two planar structures along the feature dimension. We denote by $F^{\text{mic}}$ the number of local features in $T_i^{\text{mic}}$ and by $F^{\text{mac}}$ the number of global features in $T_i^{\text{mac}}$, with the total number of features $F = F^{\text{mic}} + F^{\text{mac}}$.

The micro planes $T_i^{\text{mic}}$ are scene-specific, and are hence independently learned for every scene. The macro planes $T_i^{\text{mac}}$ represent globally captured information that is relevant for the current scene. They are computed for each scene from globally shared planar structures $\mathcal{B} = \{B_k\}_{k=1}^M$ by the weighted sum:

$$T_i^{\text{mac}} = W_i \mathcal{B} = \sum_{k=1}^{M} w_i^k B_k, \qquad (2)$$
where $W_i$ are learned coefficients for scene $S_i$, and $\{B_k\}_{k=1}^M$ are jointly trained with every scene. With this approach, the number of micro planes $N$ scales directly with the number of scenes, while the number of macro planes $M$ is a chosen hyper-parameter. We take $M > 1$ in order to capture diverse information, which our experiments showed to be beneficial for maintaining rendering quality. We also propose an ultra-lightweight (ULW) variant of our method with $F^{mic} = 0$ (only macro planes), where we achieve substantial memory savings at the expense of a slight reduction in rendering quality. Overall, decomposing our Fused-Planes into micro and macro components accelerates training and reduces its memory footprint, as we divide the number of trainable features relative to traditional Tri-Planes by a factor of $\frac{F}{F^{mic}}$, asymptotically.

While Tri-Planes are traditionally used to model scenes in the RGB space, we train our Fused-Planes in the latent space of an image autoencoder, defined by an encoder $E_\phi$ and a decoder $D_\psi$. This reduces the cost associated with volume rendering. Given a camera pose $p_j$, we render a latent Fused-Plane $T_i$ as follows:

$$ \tilde{z}_{i,j} = R_{\alpha}(T_i, p_j), \quad \tilde{x}_{i,j} = D_{\psi}(\tilde{z}_{i,j}), \quad (3) $$

where $R_\alpha$ is the Tri-Plane renderer with trainable parameters $\alpha$, $\tilde{z}_{i,j}$ is the rendered latent image, and $\tilde{x}_{i,j}$ is the corresponding decoded rendering.

## 3.2. Learning a Large Set of Fused-Planes

This section outlines our training strategy to learn a large set of scenes. Figure 2 provides an overview of our training pipeline. For clarity, the corresponding detailed algorithm is written in Appendix E.

**Learning the first subset $\mathcal{T}_1$ (including $\mathcal{B}$), and the latent space.** We learn the representations $\mathcal{T}_1$ modeling the first subset of scenes $\mathcal{S}_1$ jointly with our auto-encoder. It is important to note that training the representations $\mathcal{T}_1$ implies training both their scene-specific micro planes, and the globally shared base planes that will also be utilized in the next stage. To learn our latent space and $\mathcal{T}_1$, we adopt the 3D regularization losses from Anonymous (2025) recalled below. We supervise a Fused-Plane $T_i$ and the encoder $E_\phi$ in the latent space with the loss $L^{(latent)}$:

$$ L_{i,j}^{(latent)}(\phi, T_i, \alpha) = \|z_{i,j} - \tilde{z}_{i,j}\|_2^2, \quad (4) $$

where $z_{i,j} = E_\phi(x_{i,j})$ is the encoded ground truth image, and $\tilde{z}_{i,j} = R_\alpha(T_i, p_{i,j})$ is the rendered latent image. This loss optimizes the encoder parameters and the micro-macro Fused-Planes parameters to align the encoded latent images $z_{i,j}$ and the Fused-Planes rendering $\tilde{z}_{i,j}$. We also supervise $T_i$ and the decoder $D_\psi$ in the RGB space via $L^{(RGB)}$:

$$ L_{i,j}^{(RGB)}(\psi, T_i, \alpha) = \|x_{i,j} - \tilde{x}_{i,j}\|_2^2, \quad (5) $$

where $x_{i,j}$ is the ground truth image, and $\tilde{x}_{i,j} = D_\psi(\tilde{z}_{i,j})$ is the decoded rendering. This loss ensures a good Fused-Planes rendering quality when decoded to the RGB space, and finds the optimal decoder for this task. Finally, we adopt the reconstructive objective $L^{(ae)}$ supervising the auto-encoder:

$$ L_{i,j}^{(ae)}(\phi, \psi) = \|x_{i,j} - \tilde{x}_{i,j}\|_2^2, \quad (6) $$

where $\tilde{x}_{i,j} = D_\psi(E_\psi(x_{i,j}))$ is the reconstructed ground truth image.

Overall, our full training objective is composed of the three previous losses summed over $\mathcal{S}_1$ to optimize the Fused-Planes $\mathcal{T}_1$, the encoder $E_\phi$ and the decoder $D_\psi$:

$$ \min_{\mathcal{T}_1, \alpha, \phi, \psi} \sum_{i=1}^{N_1} \sum_{j=1}^{V} \lambda^{(latent)} L_{i,j}^{(latent)}(\phi, T_i, \alpha) \\ + \lambda^{(RGB)} L_{i,j}^{(RGB)}(\psi, T_i, \alpha) \\ + \lambda^{(ae)} L_{i,j}^{(ae)}(\phi, \psi), \quad (7) $$

where $\lambda^{(latent)}$, $\lambda^{(RGB)}$, and $\lambda^{(ae)}$ are hyper-parameters. In practice, we start this optimization process with a warm-up stage where the auto-encoder is frozen and only $L^{(latent)}$ is activated. This is done to warm-up the Fused-Planes $\mathcal{T}_1$ and avoid backpropagating irrelevant gradients from random initialization into the auto-encoder.

By the end of this stage, we obtain a custom 3D-aware latent space as well as shared base planes $\mathcal{B}$ that are specialized on the dataset at hand. These components are passed to train the next subset $\mathcal{T}_2$ with an even faster training time.

**Learning the remaining scenes $\mathcal{T}_2$.** We train the remaining scenes $\mathcal{T}_2$ with even lighter resource costs thanks to the optimized components obtained previously. To do so, we adapt the Latent NeRF Training Pipeline from Anonymous (2025) to Fused-Planes and scale it via our micro-macro decomposition. We first optimize the representations $\mathcal{T}_2$ via a Latent Supervision objective as follows:

$$ \min_{\mathcal{T}_2, \alpha} \sum_{i=N_1+1}^{N} \sum_{j=1}^{V} L_{i,j}^{(latent)}(\phi, T_i, \alpha). \quad (8) $$

This objective optimizes the representations in $\mathcal{T}_2$ to reproduce the latent images. Subsequently, we continue with an RGB Alignment which also fine-tunes the decoder for the current scenes:

$$ \min_{\mathcal{T}_2, \alpha, \psi} \sum_{i=N_1+1}^{N} \sum_{j=1}^{V} L_{i,j}^{(RGB)}(\psi, T_i, \alpha). \quad (9) $$

The end of this stage marks the end of our training where all the scenes in $\mathcal{T} = \mathcal{T}_1 \cup \mathcal{T}_2$ are now learned. Note that the trained components of our pipeline in $\mathcal{T}_1$ can still be utilized after this training to learn additional scenes in $\mathcal{T}_2$ with alleviated resource costs.
<figure><img src="image_3.png" /><figcaption>**Figure 3.** **Qualitative comparison** Comparison of NVS quality between Fused-Planes and other many-scene methods, on Basel Faces and four ShapeNet categories. NVS is performed on never-seen test views. Fused-Planes (planar) and SPARF-128 (non-planar) showcase superior rendering quality compared to other methods.</figcaption></figure>

# 4. Experiments

In this section, we evaluate Fused-Planes in the framework of multi-scene inverse graphics. We compare our method with several baselines, focusing on NVS rendering quality, training time, and memory footprint.

**Baselines.** As portrayed in Section 2 and Appendix A, various methods could be utilized in the context of multi-scene inverse graphics. To choose our baselines, we first select the methods that target **many-view NVS**. Few-view methods lie in a different setting and typically produce lower-quality scenes due to the under-constrained task. Subsequently, we identify from our related work two categories of baselines. **The first category – single-scene methods –**, includes works aimed at reducing the resource costs of NeRFs when learning a single scene, and by extension, many scenes. From this category, we compare Fused-Planes with Vanilla-NeRF (Mildenhall et al., 2020), Instant-NGP (Müller et al., 2022), TensoRF (Chen et al., 2022), Gaussian Splatting (Kerbl et al., 2023, 3DGS), and K-Planes (Fridovich-Keil et al., 2023), using their Nerfstudio (Tancik et al., 2023) implementations. Among these, K-Planes is a planar representation with multi-resolution structures, resulting in higher quality but higher memory usage than Tri-Planes and reduced compatibility with 2D backbones. **The second category – many-scene methods –**, includes works that were specifically designed for the context of learning large sets of scenes. This includes Tri-Planes (Chan et al., 2022a) and SPARF (Hamdi et al., 2023) as they were initially introduced as representations used to train generative models (3D GAN and SuRFNet, respectively). This category additionally includes CodeNeRF (Jang & Agapito, 2021), which

is a scalable method that uses a single neural representation to represent multiple scenes. Furthermore, we compare our method with “CodeNeRF (MSIG)”, a variant of CodeNeRF which we adapted for MSIG by adopting a similar approach to our method. We first train their common neural representation on the first subset of scenes $S_1$, and then utilize it to learn the remaining scenes $S_2$ by employing their proposed test-time optimization. More details about CodeNeRF (MSIG) can be found in Appendix C.

**Datasets.** Evaluating our method requires large-scale 3D data, which is currently only available as synthetic scenes. Hence, and consistently with our multi-scene baselines (Tri-Planes, SPARF, and CodeNeRF), we use ShapeNet (Chang et al., 2015), from which we take four categories: Cars, Furniture, Speakers and Sofas. Additionally, we adopt the large-scale front-facing Basel-Face dataset (Walker et al., 2018). For ShapeNet, we render $V = 160$ views, sampled from the upper hemisphere surrounding the object. For faces, we take $V = 50$ front-facing views. All views are rendered at a resolution of $128 \times 128$. In all our experiments, we use 90% of the views for training and 10% for evaluation.

**Implementation details.** For all Fused-Planes experiments, we take $N_1 = 500$ scenes for the first subset. Then, we utilize the trained components to learn $N_2 = 1500$ scenes from the second subset. Note that we take $N_1 = 1000$ for CodeNeRF (MSIG), which we discuss in Appendix C. For Fused-Planes, we take $F^{\text{mic}} = 10$, $F^{\text{mac}} = 22$, and $M = 50$. For Fused-Planes-ULW, we take $F^{\text{mic}} = 0$, $F^{\text{mac}} = 32$, and $M = 50$. We detail our hyper-parameters in Appendix F. We adopt the pre-trained VAE
<table><caption>**Table 1.** Quantitative comparison. Comparison of NVS quality and resources costs between Fused-Planes and other methods. For resource costs, $\tau$ and $\mu$ respectively denote the training time and memory requirement per scene. The training time is measured using a single NVIDIA L4 GPU. Fused-Planes presents comparable rendering quality to other planar representations while significantly reducing their resource costs. This brings the resource requirements for learning a planar representation closer to recent non-planar methods. Fused-Planes-ULW has the lowest per-scene memory footprint.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Planar</th><th colspan="3">ShapeNet datasets</th><th colspan="3">Basel Faces</th><th rowspan="2">τ (min)</th><th rowspan="2">μ (MB)</th></tr><tr><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th></tr></thead><tbody><tr><td rowspan="5">single-scene approaches</td><td>Vanilla-NeRF</td><td>✗</td><td>35.85</td><td>0.977</td><td>0.027</td><td>42.91</td><td>0.996</td><td>0.001</td><td>636.8</td><td>22.00</td></tr><tr><td>Instant-NGP</td><td>✗</td><td>34.06</td><td>0.973</td><td>0.022</td><td>36.54</td><td>0.981</td><td>0.009</td><td>7.52</td><td>189.13</td></tr><tr><td>TensoRF</td><td>✗</td><td>36.74</td><td>0.985</td><td>0.013</td><td>40.66</td><td>0.991</td><td>0.004</td><td>68.93</td><td>208.32</td></tr><tr><td>3DGS</td><td>✗</td><td>32.95</td><td>0.975</td><td>0.043</td><td>43.06</td><td>0.995</td><td>0.002</td><td>9.37</td><td>27.66</td></tr><tr><td>K-Planes</td><td>✓</td><td>30.88</td><td>0.956</td><td>0.043</td><td>40.23</td><td>0.991</td><td>0.005</td><td>75.35</td><td>410.17</td></tr><tr><td rowspan="5">many-scene approaches</td><td>CodeNeRF</td><td>✗</td><td>28.34</td><td>0.930</td><td>0.121</td><td>35.46</td><td>0.972</td><td>0.010</td><td>14.96</td><td>0.0026</td></tr><tr><td>CodeNeRF (MSIG)</td><td>✗</td><td>26.99</td><td>0.915</td><td>0.126</td><td>35.44</td><td>0.971</td><td>0.010</td><td>9.54</td><td>0.0026</td></tr><tr><td>SPARF-32</td><td>✗</td><td>26.12</td><td>0.910</td><td>0.152</td><td>28.53</td><td>0.921</td><td>0.116</td><td>0.69</td><td>0.04</td></tr><tr><td>SPARF-128</td><td>✗</td><td>31.07</td><td>0.966</td><td>0.064</td><td>38.18</td><td>0.986</td><td>0.016</td><td>0.82</td><td>0.74</td></tr><tr><td>Tri-Planes</td><td>✓</td><td>28.15</td><td>0.919</td><td>0.121</td><td>36.47</td><td>0.980</td><td>0.013</td><td>64.32</td><td>1.50</td></tr><tr><td rowspan="2">ours</td><td>Fused-Planes-ULW</td><td>✓</td><td>29.02</td><td>0.937</td><td>0.092</td><td>33.96</td><td>0.955</td><td>0.010</td><td>7.16</td><td>0.0008</td></tr><tr><td>Fused-Planes</td><td>✓</td><td>30.47</td><td>0.957</td><td>0.042</td><td>37.24</td><td>0.973</td><td>0.006</td><td>8.96</td><td>0.48</td></tr></tbody></table>

from Stable Diffusion (Rombach et al., 2022) as initialization for our VAE. Our code is available in the supplementary material and will be available as open-source.

## 4.1. Evaluations

**NVS quality.** To evaluate the NVS quality of the learned scenes, we compute the Peak Signal-to-Noise Ratio (PSNR ↑), the Structural Similarity Index Measure (SSIM ↑) and the Learned Perceptual Image Patch Similarity (Zhang et al., 2018, LPIPS ↓) between reference views and corresponding rendered NVS views. The reference views have never been seen during training. Note that to evaluate the NVS quality of all methods in a consistent manner, we randomly sample 4 evaluation scenes from each dataset, leading to 16 scenes for ShapeNet (four categories) and 4 scenes for Basel Faces. This sampling approach is necessary due to the high computational cost associated with evaluating single-scene methods on large sets of scenes.

**Resource costs.** We denote $t_{\text{tot}}(N)$ the total training time required to train a particular method on $N$ scenes. For single-scene methods, it is written as:

$$t_{\text{tot}}(N) = N\tau, \qquad (10)$$

where $\tau$ represents the time required to train a single scene.

Fused-Planes and CodeNeRF (MSIG) showcase a different regime, with two different training speeds depending on which subset $S_1$ or $S_2$ is being trained. We express the total training time required to train $N > N_1$ scenes as follows:

$$t_{\text{tot}}(N) = N_1\tau_1 + (N - N_1)\tau, \qquad (11)$$

where $\tau_1$ and $\tau$ respectively represent the training time per-scene for the first and second subsets.

We also report the total memory footprint $m_{\text{tot}}(N)$ required to store the $N$ scene representations, written as:

$$m_{\text{tot}}(N) = m_0 + N\mu, \qquad (12)$$

where $m_0$ denotes the memory required to store all the components that are shared across scenes (e.g. encoder, decoder), and $\mu$ represents the memory required to store a single scene representation.

**Results.** We present our main quantitative and qualitative results in Table 1 and Figure 3. Per-scene NVS results and visualizations are available in Tables 8 to 12 and Figures 7 to 11. Fused-Planes demonstrates a significant reduction in resource costs compared to all other planar methods. Additionally, it also improves upon rendering quality compared to Tri-Planes, its base representation. Note that Fused-Planes also exhibits 2.3× faster rendering than Tri-Planes (0.36+9.71 ms of decoding versus 23.30 ms). In a broader context, Fused-Planes exhibits competitive resource costs, outperforming all current methods except SPARF. It also showcases the best rendering quality among many-scene methods, excluding SPARF-128. It is important to note that SPARF is a non planar method for which we observe faulty textures similar to those depicted by Hamdi et al. (2023), which are reflected in the its LPIPS metrics. We illustrate these SPARF failure cases in Figure 6. Finally, our Fused-Planes-ULW variant trades minor rendering quality for exceptional memory efficiency, achieving the lowest memory usage among all methods.
<figure><img src="image_4.png" /><figcaption>**Figure 4.** Scaling the number of scenes. Evolution of the total training time $t_{tot}$ (left) and total memory footprint $m_{tot}$ (right) when scaling the number of scenes ($N$), for Fused-Planes and other methods. Solid lines depict planar representation, while dashed and dotted lines denote non-planar representation. Fused-Planes presents better scaling in terms of resources cost as compared to other planar representations. For clarity, we present in Figure 5 a magnified version of the memory cost plot focusing on the range $N \in [0, 100]$.</figcaption></figure>

<table><caption>**Table 2.** Ablation Study. Comparison of NVS quality and resource costs across ablated versions of our method. NVS metrics are computed and averaged over the same 50 randomly sampled scenes from the ShapeNet Cars dataset. For resource costs, $\tau$ and $\mu$ respectively denote the training time and memory requirement per scene. Fused-Planes outperforms its ablations. Fused-Planes-ULW trades off minor NVS quality for substantial memory saving.</caption><thead><tr><th></th><th>Latent<br>Space</th><th>Micro<br>Planes</th><th>Macro<br>Planes</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>τ (min)</th><th>μ (MB)</th></tr></thead><tbody><tr><td>Fused-Planes (M = 1)</td><td>✓</td><td>✓</td><td>✓</td><td>27.69</td><td>0.942</td><td>0.042</td><td>8.48</td><td>0.48</td></tr><tr><td>Fused-Planes (Micro)</td><td>✓</td><td>✓</td><td>✗</td><td>27.64</td><td>0.941</td><td>0.040</td><td>12.84</td><td>1.50</td></tr><tr><td>Fused-Planes (RGB)</td><td>✗</td><td>✓</td><td>✓</td><td>27.71</td><td>0.942</td><td>0.044</td><td>63.52</td><td>0.48</td></tr><tr><td>Tri-Planes</td><td>✗</td><td>✓</td><td>✗</td><td>28.56</td><td>0.953</td><td>0.035</td><td>64.08</td><td>1.50</td></tr><tr><td>Fused-Planes-ULW</td><td>✓</td><td>✗</td><td>✓</td><td>27.51</td><td>0.935</td><td>0.063</td><td>7.16</td><td>0.0008</td></tr><tr><td>Fused-Planes</td><td>✓</td><td>✓</td><td>✓</td><td>28.64</td><td>0.950</td><td>0.037</td><td>8.92</td><td>0.48</td></tr></tbody></table>

An overview of the resource costs and ShapeNet rendering quality can be found in Figure 1, illustrating the evaluations in Table 1. Moreover, we present in Figure 4 the resource costs required by each method when scaling the number of scenes, as described by Equations (10) to (12). Fused-Planes significantly reduces the resource requirements for planar scene representations in MSIG, and becomes more advantageous as the number of scenes grows. For clarity, the costs coefficients $\tau_1$, $\tau$, $m_0$ and $\mu$ are reported in Table 6 when relevant (i.e. methods training two subsets, or utilizing shared components with $m_0 > 0$).

In Appendix B.1, Tables 4 and 5 compare for Fused-Planes the NVS metrics of scenes in $S_1$ with those in $S_2$. Fused-Planes showcases similar rendering quality in both subsets.

## 4.2. Ablations

To justify our design choices, we present an ablation study of our method, for which the results are presented in Table 2.

The first ablation, “Fused-Planes (M = 1)” reduces the set of shared planes $B$ to a single base plane. The second ablation, “Fused-Planes (RGB)” ablates the latent space and trains Fused-Planes in the RGB space. The third ablation, “Fused-Planes (Micro)”, eliminates the Macro component

of Fused-Planes, and consequently global information sharing (i.e. $F^{mac} = 0, F^{mic} = 32$). These ablations demonstrate a slight degradation of quality as compared to Fused-Planes, highlighting the necessity for a set of global planes and for training Fused-Planes in the latent space.

Finally, ablating both the latent space and macro planes is equivalent to our baseline representation “Tri-Planes”, which presents significantly higher resource costs.

## 5. Conclusion

In this paper, we introduce Fused-Planes, a novel planar scene representation that is specifically tailored for the framework of learning large-sets of scenes, which we call “multi-scene inverse graphics”. Fused-Planes learn shared structures across scenes which reduces per-scene representation complexity, and are trained in a 3D-aware latent space. Our approach demonstrates a substantial improvement in resource costs in multi-scene inverse graphics compared to previous planar scene representations, all while maintaining comparable rendering quality. It also showcases competitive resource costs in MSIG when compared to state-of-the-art methods. We consider this work to be a pivotal milestone for methods targeting multi-scene inverse graphics, and hope it inspires further work in this direction.
# Impact Statement

This paper contributes to advancing the field of inverse graphics, with a focus on accelerating techniques for 3D reconstruction from images in large-scale settings. As such, risks associated with our work parallel those of other inverse graphics papers. This includes but is not limited to potential privacy concerns when applied to sensitive datasets, and the environmental impact of computationally intensive training and inference, which we aim to alleviate. Furthermore, our work utilizes the Variational Auto-Encoder from Stable Diffusion (Rombach et al., 2022) as a pre-trained model, and thus has the potential of inheriting any problematic biases and limitations this model may have.

# References

Anciukevičius, T., Xu, Z., Fisher, M., Henderson, P., Bilen, H., Mitra, N. J., and Guerrero, P. RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 12608–12618, June 2023.

Anonymous. Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder. In *The Thirteenth International Conference on Learning Representations*, 2025. To appear at ICLR 2025.

Aumentado-Armstrong, T., Mirzaei, A., Brubaker, M. A., Kelly, J., Levinshtein, A., Derpanis, K. G., and Gilitschenski, I. Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations. *arXiv preprint arXiv:2310.17880*, 2023.

Cao, A. and Johnson, J. HexPlane: A Fast Representation for Dynamic Scenes. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 130–141, June 2023.

Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L. J., Tremblay, J., Khamis, S., Karras, T., and Wetzstein, G. Efficient Geometry-Aware 3D Generative Adversarial Networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 16123–16133, June 2022a.

Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L. J., Tremblay, J., Khamis, S., Karras, T., and Wetzstein, G. Efficient Geometry-Aware 3D Generative Adversarial Networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 16123–16133, June 2022b.

Chan, E. R., Nagano, K., Chan, M. A., Bergman, A. W., Park, J., Levy, A., Aittala, M., Mello, S. D., Karras, T., and Wetzstein, G. Generative Novel View Synthesis with 3D-Aware Diffusion Models. In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 4194–4206, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society. doi: 10.1109/ICCV51070.2023.00389.

Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. ShapeNet: An Information-Rich 3D Model Repository. *arXiv preprint arXiv:1512.03012*, 2015.

Chen, A., Xu, Z., Geiger, A., Yu, J., and Su, H. TensoRF: Tensorial Radiance Fields. In *European Conference on Computer Vision (ECCV)*, 2022.

Chen, H., Gu, J., Chen, A., Tian, W., Tu, Z., Liu, L., and Su, H. Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 2416–2425, October 2023.

Chen, Y. and Wang, X. Transformers as Meta-learners for Implicit Neural Representations. In Avidan, S., Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), *Computer Vision – ECCV 2022*, pp. 170–187, Cham, 2022. Springer Nature Switzerland.

Erkoç, Z., Ma, F., Shan, Q., Nießner, M., and Dai, A. HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 14300–14310, October 2023.

Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., and Kanazawa, A. Plenoxels: Radiance Fields Without Neural Networks. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 5501–5510, June 2022.

Fridovich-Keil, S., Meanti, G., Warburg, F. R., Recht, B., and Kanazawa, A. K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 12479–12488, June 2023.

Gupta, A., Xiong, W., Nie, Y., Jones, I., and Oğuz, B. 3dgen: Triplane latent diffusion for textured mesh generation. *arXiv preprint arXiv:2303.05371*, 2023.

Hamdi, A., Ghanem, B., and Nießsner, M. SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops*, pp. 2930–2940, October 2023.
Jang, W. and Agapito, L. CodeNeRF: Disentangled Neural Radiance Fields for Object Categories. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 12949–12958, October 2021.

Kajiya, J. T. and Von Herzen, B. P. Ray Tracing Volume Densities. *SIGGRAPH Comput. Graph.*, 18(3):165–174, January 1984. doi: 10.1145/964965.808594.

Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. *ACM Transactions on Graphics*, 42(4), July 2023.

Khalid, U., Iqbal, H., Karim, N., Hua, J., and Chen, C. LatentEditor: Text Driven Local Editing of 3D Scenes, 2023.

Liu, Y.-T., Guo, Y.-C., Luo, G., Sun, H., Yin, W., and Zhang, S.-H. PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 19915–19924, June 2024.

Metzer, G., Richardson, E., Patashnik, O., Giryes, R., and Cohen-Or, D. Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 12663–12673, June 2023.

Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In *ECCV*, 2020.

Müller, N., Siddiqui, Y., Porzi, L., Bulò, S. R., Kontschieder, P., and Nießner, M. DiffRF: Rendering-Guided 3D Radiance Field Diffusion. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 4328–4338, June 2023.

Müller, T., Evans, A., Schied, C., and Keller, A. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. *ACM Trans. Graph.*, 41(4):102:1–102:15, July 2022. doi: 10.1145/3528223.3530127.

Niemeyer, M. and Geiger, A. Giraffe: Representing scenes as compositional generative neural feature fields. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 11453–11464, June 2021.

Park, J., Kwon, G., and Ye, J. C. ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF. In *International Conference on Learning Representations*, 2024.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-Resolution Image Synthesis With Latent Diffusion Models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 10684–10695, June 2022.

Schwarz, K., Liao, Y., Niemeyer, M., and Geiger, A. Graf: Generative radiance fields for 3d-aware image synthesis, 2021.

Seo, H., Kim, H., Kim, G., and Chun, S. Y. Ditto-nerf: Diffusion-based iterative text to omni-directional 3d model. *arXiv preprint arXiv:2304.02827*, 2023.

Shue, J. R., Chan, E. R., Po, R., Ankner, Z., Wu, J., and Wetzstein, G. 3D Neural Field Generation Using Triplane Diffusion. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 20875–20886, June 2023.

Singh, P., Tiwari, A., Vashishtha, G., and Raman, S. C3-NeRF: Modeling Multiple Scenes via Conditional-cum-Continual Neural Radiance Fields. *arXiv preprint arXiv:2411.19903*, 2024.

Sun, C., Sun, M., and Chen, H.-T. Direct Voxel Grid Optimization: Super-Fast Convergence for Radiance Fields Reconstruction. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 5459–5469, June 2022a.

Sun, C., Sun, M., and Chen, H.-T. Improved direct voxel grid optimization for radiance fields reconstruction. *arXiv preprint arXiv:2206.05085*, 2022b.

Szymanowicz, S., Rupprecht, C., and Vedaldi, A. Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 8863–8873, October 2023.

Tancik, M., Mildenhall, B., Wang, T., Schmidt, D., Srinivasan, P. P., Barron, J. T., and Ng, R. Learned Initializations for Optimizing Coordinate-Based Neural Representations. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 2846–2855, June 2021.

Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P. P., Barron, J. T., and Kretzschmar, H. Block-NeRF: Scalable Large Scene Neural View Synthesis. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 8248–8258, June 2022.

Tancik, M., Weber, E., Ng, E., Li, R., Yi, B., Kerr, J., Wang, T., Kristoffersen, A., Austin, J., Salahi, K., Ahuja, A., McAllister, D., and Kanazawa, A. Nerfstudio: A modular
framework for neural radiance field development. In *ACM SIGGRAPH 2023 Conference Proceedings*, SIGGRAPH '23, 2023.

Walker, M., Schönborn, S., Greifeneder, R., and Vetter, T. *The Basel Face Database: A validated set of photographs reflecting systematic differences in Big Two and Big Five personality dimensions*. *PLOS ONE*, 13(3):1–20, 03 2018. doi: 10.1371/journal.pone.0193190.

Wang, T., Zhang, B., Zhang, T., Gu, S., Bao, J., Baltrusaitis, T., Shen, J., Chen, D., Wen, F., Chen, Q., and Guo, B. *RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion*. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 4563–4573, June 2023.

Ye, J., Wang, N., and Wang, X. *FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models*. In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 8928–8939, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society. doi: 10.1109/ICCV51070.2023.00823.

Yu, A., Li, R., Tancik, M., Li, H., Ng, R., and Kanazawa, A. *PlenOctrees for Real-Time Rendering of Neural Radiance Fields*. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 5752–5761, October 2021.

Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. *The Unreasonable Effectiveness of Deep Features as a Perceptual Metric*. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2018.
## A. Comparison with related work

We present in Table 3 an overview of the related work that could be employed for the framework of multi-scene inverse graphics, following the taxonomy of Section 2.2.

**Table 3. Related work overview.** This table provides an overview of various approaches discussed in Section 2.2 that could be utilized for addressing multi-scene inverse graphics, highlighting the strengths and limitations of each approach. *Preliminary steps* indicates the steps taken before being able to leverage a model for multi-scene learning (e.g. pre-training, creating a dataset of scene representations). *NVS setting* indicates whether the method performs novel view synthesis (NVS) from few or many training views. *Shared parameters* indicates the presence of common parameters that contribute to the computation of multiple scene representations (e.g. a common model, parameter sharing). *Planar structure* indicates whether the approach adopts scene architectures with planar structures. We compare our work with all presented methods lying in a similar NVS setting as our method (many views).

### Approaches for Multi-Scene Inverse Graphics

**Single-scene methods.** Single-scene methods can be used for MSIG by repeated and independent applications. It is usually only advantageous when the number of scenes to learn is relatively small.

<table><thead><tr><th></th><th>Preliminary steps</th><th>NVS setting</th><th>Shared parameters</th><th>Planar structure</th></tr></thead><tbody><tr><td>Mildenhall et al. (2020)</td><td>None</td><td>Many views</td><td>✗</td><td>✗</td></tr><tr><td>Müller et al. (2022)</td><td>None</td><td>Many views</td><td>✗</td><td>✗</td></tr><tr><td>Chen et al. (2022)</td><td>None</td><td>Many views</td><td>✗</td><td>✗</td></tr><tr><td>Kerbl et al. (2023)</td><td>None</td><td>Many views</td><td>✗</td><td>✗</td></tr><tr><td>Fridovich-Keil et al. (2023)</td><td>None</td><td>Many views</td><td>✗</td><td>✓</td></tr><tr><td>Tancik et al. (2021)</td><td>Meta-learning<br>initialization</td><td>Few views</td><td>✗</td><td>✗</td></tr><tr><td>Chen & Wang (2022)</td><td>Meta-learning<br>initialization</td><td>Few views</td><td>✗</td><td>✗</td></tr></tbody></table>

**Feed-forward methods.** These methods benefit from a fast inference time for scenes, but only after an expensive preliminary training. Additionally, they are currently limited to the setting of few-view NVS, leading to lower quality scenes.

<table><thead><tr><th></th><th>Preliminary steps</th><th>NVS setting</th><th>Shared parameters</th><th>Planar structure</th></tr></thead><tbody><tr><td>Wang et al. (2023)</td><td>Model training</td><td>Few views</td><td>✓</td><td>✓</td></tr><tr><td>Anciukevičius et al. (2023)</td><td>Model training</td><td>Few views</td><td>✓</td><td>✓</td></tr><tr><td>Szymanowicz et al. (2023)</td><td>Model training</td><td>Few views</td><td>✓</td><td>✗</td></tr><tr><td>Hamdi et al. (2023, SuRFNet)</td><td>Dataset training<br>& model training</td><td>Few views</td><td>✓</td><td>✗</td></tr></tbody></table>

**Many-scene methods.** These approaches have been specifically introduced to learn large sets of scenes. We experimentally compare our work with all these methods.

<table><thead><tr><th></th><th>Preliminary steps</th><th>NVS setting</th><th>Shared parameters</th><th>Planar structure</th></tr></thead><tbody><tr><td>Jang & Agapito (2021)</td><td>Model training</td><td>Many views</td><td>✓</td><td>✗</td></tr><tr><td>Hamdi et al. (2023, SPARF)</td><td>None</td><td>Many views</td><td>✗</td><td>✗</td></tr><tr><td>Chan et al. (2022b)</td><td>None</td><td>Many views</td><td>✗</td><td>✓</td></tr><tr><td><strong>Fused-Planes (ours)</strong></td><td>None</td><td>Many views</td><td>✓</td><td>✓</td></tr></tbody></table>
# B. Supplementary Results

## B.1. Quantitative results

Tables 4 and 5 present the NVS metrics for both subsets of our training, for ShapeNet Cars and Basel Faces, respectively. All metrics are computed on never-seen test views from 50 randomly sampled scenes from each dataset. Both subsets demonstrate similar NVS performances.

Table 6 presents resource costs for MSIG methods that subdivide a large set of scenes into two subsets. For improved clarity, Figure 5 provides a magnified view of Figure 4, focusing on the range [0,100].

We present per-scene NVS metrics in Tables 8 to 12.

<table><caption>Table 4. Quantitative comparison. NVS performances on ShapeNet Cars in both subsets of our training.</caption><thead><tr><th rowspan="3"></th><th colspan="6">ShapeNet Cars</th></tr><tr><th colspan="3">S<sub>1</sub></th><th colspan="3">S<sub>2</sub></th></tr><tr><th>PSNR↑</th><th>SSIM↑</th><th>LPIPS↓</th><th>PSNR↑</th><th>SSIM↑</th><th>LPIPS↓</th></tr></thead><tbody><tr><th>Tri-Planes (RGB)</th><td>28.49</td><td>0.9539</td><td>0.0291</td><td>28.58</td><td>0.9505</td><td>0.0360</td></tr><tr><th>Fused-Planes</th><td>28.14</td><td>0.9505</td><td>0.0301</td><td>28.77</td><td>0.9496</td><td>0.0383</td></tr></tbody></table>

<table><caption>Table 5. Quantitative comparison. NVS performances on Basel Faces in both subsets of our training.</caption><thead><tr><th rowspan="3"></th><th colspan="6">Basel Faces</th></tr><tr><th colspan="3">S<sub>1</sub></th><th colspan="3">S<sub>2</sub></th></tr><tr><th>PSNR↑</th><th>SSIM↑</th><th>LPIPS↓</th><th>PSNR↑</th><th>SSIM↑</th><th>LPIPS↓</th></tr></thead><tbody><tr><th>Tri-Planes (RGB)</th><td>36.82</td><td>0.9807</td><td>0.0122</td><td>36.35</td><td>0.9787</td><td>0.0129</td></tr><tr><th>Fused-Planes</th><td>36.17</td><td>0.9678</td><td>0.0062</td><td>36.99</td><td>0.9712</td><td>0.0056</td></tr></tbody></table>

<table><caption>Table 6. Resource costs for MSIG methods with two subsets. We present the resource costs of MSIG methods that subdivide large sets of scenes into two subsets, as depicted in Equations (11) and (12).</caption><thead><tr><th></th><th>N<sub>1</sub></th><th>τ<sub>1</sub> (min)</th><th>τ (min)</th><th>m<sub>0</sub> (MB)</th><th>μ (MB)</th></tr></thead><tbody><tr><th>CodeNeRF (MSIG)</th><td>1000</td><td>14.96</td><td>9.54</td><td>3.71</td><td>0.0026</td></tr><tr><th>Fused-Planes-ULW</th><td>500</td><td>13.72</td><td>7.16</td><td>384.19</td><td>0.0008</td></tr><tr><th>Fused-Planes</th><td>500</td><td>14.96</td><td>8.92</td><td>360.75</td><td>0.48</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 5. Memory costs. This figure presents the memory costs depicted in Figure 4 within the range $N \in [0, 100]$.</figcaption></figure>
## B.2. Qualitative results

We present in Figures 7 to 11 additional qualitative comparisons across all the methods discussed in our experiments (Section 4). Fused-Plane demonstrates similar visual quality to state-of-the-art methods.

## C. CodeNeRF (MSIG)

**CodeNeRF.** CodeNeRF (Jang & Agapito, 2021) learns a set of scenes with a single neural representation $f_\theta$ which is conditioned on scene-specific latent codes. Specifically, for each scene, a shape code $z_s$ and an appearance code $z_\alpha$ is learned, such that $f_\theta(z_s, z_\alpha)$ models the current scene. Once the conditional NeRF $f_\theta$ is trained on a large set of scenes, it can learn new scenes using test-time optimization. This test-time optimization consists of learning a new scene by optimizing only the codes ($z_s, z_\alpha$), while keeping $f_\theta$ fixed. By reducing the number of trainable parameters, test-time-optimization offers increased training speed. Furthermore, the memory required to store an additional scene on disk is very low, since only ($z_s, z_\alpha$) need to be stored.

**CodeNeRF (MSIG).** In our experiments, we introduce CodeNeRF (MSIG) as a new comparative baseline. CodeNeRF (MSIG) employs a novel training procedure inspired by ours, which leverages the test-time optimization method originally proposed by CodeNeRF to improve efficiency in multi-scene inverse graphics. Specifically, we first train the shared neural representation $f_\theta$ of CodeNeRF on a subset $S_1$ composed of $N_1$ scenes. Subsequently, we employ test-time-optimization with the previously trained representation to learn the remaining scenes $S_2$, with lowered training times.

We present in Table 7 a comparison of CodeNeRF (MSIG) performances when taking $N_1 = 500$ (like Fused-Planes) and $N_1 = 1000$. CodeNeRF (MSIG) showcases better performances with $N_1 = 1000$, which we set throughout the paper for this method.

<table><caption>Table 7. Choice of $N_1$ for CodeNeRF (MSIG). CodeNeRF (MSIG) showcases better performances when taking $N_1 = 1000$, which we set throughout the paper for this method. Metrics are averaged over 16 scenes for ShapeNet, and 4 scenes for Basel Faces.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">N<sub>1</sub></th><th colspan="3">ShapeNet datasets</th><th colspan="3">Basel Faces</th></tr><tr><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th></tr></thead><tbody><tr><td>CodeNeRF (MSIG)</td><td>500</td><td>26.81</td><td>0.9108</td><td>0.1281</td><td>34.15</td><td>0.964</td><td>0.011</td></tr><tr><td>CodeNeRF (MSIG)</td><td>1000</td><td>26.99</td><td>0.9154</td><td>0.1257</td><td>35.44</td><td>0.971</td><td>0.010</td></tr></tbody></table>

## D. SPARF failure cases

Hamdi et al. (2023) showcase that SPARF demonstrates failure cases when processing certain textures. Our experimental results corroborate these findings, as illustrated in Figure 6.

## E. Training Algorithm

For clarity, Algorithm 1 provides a detailed outline of our training algorithm on the two subsets.

## F. Hyperparameters

For reproducibility purposes, Tables 13 and 14 expose our hyperparameter settings respectively for the first and second subsets of our training. A more detailed list of our hyperparameters can be found in the configuration files of our open-source code.
<figure><img src="image_6.png" /><figcaption>**Figure 6.** **SPARF failure cases.** As demonstrated by Hamdi et al. (2023), SPARF exhibits faulty texture representations when learning certain scenes. Fused-Planes does not exhibit such failure cases.</figcaption></figure>
<table><caption>Table 8. Per-scene quantitative comparison on Basel Faces.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Planar</th><th colspan="3">Face 1</th><th colspan="3">Face 2</th><th colspan="3">Face 3</th><th colspan="3">Face 4</th></tr><tr><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th></tr></thead><tbody><tr><td>Vanilla-NeRF</td><td>✗</td><td>43.44</td><td>0.996</td><td>0.001</td><td>43.90</td><td>0.997</td><td>0.001</td><td>42.65</td><td>0.996</td><td>0.001</td><td>41.64</td><td>0.994</td><td>0.003</td></tr><tr><td>Instant-NGP</td><td>✗</td><td>37.79</td><td>0.987</td><td>0.004</td><td>40.01</td><td>0.990</td><td>0.002</td><td>35.38</td><td>0.977</td><td>0.013</td><td>32.96</td><td>0.969</td><td>0.016</td></tr><tr><td>TensoRF</td><td>✗</td><td>40.80</td><td>0.993</td><td>0.003</td><td>42.72</td><td>0.995</td><td>0.001</td><td>40.96</td><td>0.993</td><td>0.003</td><td>38.16</td><td>0.982</td><td>0.011</td></tr><tr><td>3DGS</td><td>✗</td><td>43.69</td><td>0.998</td><td>0.001</td><td>45.41</td><td>0.998</td><td>0.001</td><td>43.22</td><td>0.997</td><td>0.001</td><td>39.93</td><td>0.986</td><td>0.007</td></tr><tr><td>K-Planes</td><td>✓</td><td>40.68</td><td>0.993</td><td>0.003</td><td>39.46</td><td>0.988</td><td>0.010</td><td>41.11</td><td>0.993</td><td>0.004</td><td>39.68</td><td>0.990</td><td>0.004</td></tr><tr><td>CodeNeRF</td><td>✗</td><td>35.49</td><td>0.974</td><td>0.009</td><td>36.35</td><td>0.974</td><td>0.006</td><td>34.42</td><td>0.970</td><td>0.012</td><td>35.60</td><td>0.971</td><td>0.012</td></tr><tr><td>CodeNeRF (MSIG)</td><td>✗</td><td>36.25</td><td>0.974</td><td>0.008</td><td>37.14</td><td>0.977</td><td>0.005</td><td>32.49</td><td>0.961</td><td>0.015</td><td>35.87</td><td>0.972</td><td>0.013</td></tr><tr><td>SPARF-32</td><td>✗</td><td>28.35</td><td>0.921</td><td>0.110</td><td>29.18</td><td>0.926</td><td>0.118</td><td>28.78</td><td>0.925</td><td>0.119</td><td>27.81</td><td>0.911</td><td>0.118</td></tr><tr><td>SPARF-128</td><td>✗</td><td>36.51</td><td>0.984</td><td>0.021</td><td>39.39</td><td>0.989</td><td>0.010</td><td>38.77</td><td>0.987</td><td>0.016</td><td>38.07</td><td>0.984</td><td>0.016</td></tr><tr><td>Tri-Planes</td><td>✓</td><td>36.05</td><td>0.978</td><td>0.015</td><td>37.46</td><td>0.982</td><td>0.011</td><td>36.78</td><td>0.980</td><td>0.014</td><td>35.59</td><td>0.977</td><td>0.012</td></tr><tr><td>Fused-Planes-ULW</td><td>✓</td><td>33.84</td><td>0.950</td><td>0.013</td><td>35.00</td><td>0.958</td><td>0.007</td><td>33.41</td><td>0.959</td><td>0.011</td><td>33.58</td><td>0.954</td><td>0.010</td></tr><tr><td>Fused-Planes</td><td>✓</td><td>36.24</td><td>0.970</td><td>0.007</td><td>38.63</td><td>0.975</td><td>0.004</td><td>37.04</td><td>0.975</td><td>0.006</td><td>37.04</td><td>0.971</td><td>0.006</td></tr></tbody></table>

<table><caption>Table 9. Per-scene quantitative comparison on ShapeNet Cars.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Planar</th><th colspan="3">Car 1</th><th colspan="3">Car 2</th><th colspan="3">Car 3</th><th colspan="3">Car 4</th></tr><tr><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th></tr></thead><tbody><tr><td>Vanilla-NeRF</td><td>✗</td><td>38.43</td><td>0.995</td><td>0.003</td><td>41.20</td><td>0.995</td><td>0.005</td><td>37.43</td><td>0.994</td><td>0.005</td><td>39.53</td><td>0.995</td><td>0.003</td></tr><tr><td>Instant-NGP</td><td>✗</td><td>35.31</td><td>0.986</td><td>0.008</td><td>37.88</td><td>0.990</td><td>0.010</td><td>34.06</td><td>0.987</td><td>0.013</td><td>36.33</td><td>0.989</td><td>0.007</td></tr><tr><td>TensoRF</td><td>✗</td><td>38.66</td><td>0.994</td><td>0.003</td><td>40.55</td><td>0.995</td><td>0.005</td><td>37.80</td><td>0.995</td><td>0.004</td><td>40.00</td><td>0.995</td><td>0.003</td></tr><tr><td>3DGS</td><td>✗</td><td>32.00</td><td>0.966</td><td>0.057</td><td>38.74</td><td>0.993</td><td>0.010</td><td>35.41</td><td>0.985</td><td>0.024</td><td>37.51</td><td>0.994</td><td>0.006</td></tr><tr><td>K-Planes</td><td>✓</td><td>30.51</td><td>0.966</td><td>0.029</td><td>33.84</td><td>0.976</td><td>0.027</td><td>29.73</td><td>0.968</td><td>0.037</td><td>30.57</td><td>0.967</td><td>0.031</td></tr><tr><td>CodeNeRF</td><td>✗</td><td>27.87</td><td>0.950</td><td>0.055</td><td>28.05</td><td>0.937</td><td>0.097</td><td>26.19</td><td>0.929</td><td>0.088</td><td>27.07</td><td>0.930</td><td>0.075</td></tr><tr><td>CodeNeRF (MSIG)</td><td>✗</td><td>27.10</td><td>0.946</td><td>0.055</td><td>26.86</td><td>0.929</td><td>0.103</td><td>25.25</td><td>0.921</td><td>0.092</td><td>27.10</td><td>0.932</td><td>0.074</td></tr><tr><td>SPARF-32</td><td>✗</td><td>25.35</td><td>0.931</td><td>0.126</td><td>26.43</td><td>0.920</td><td>0.135</td><td>22.98</td><td>0.892</td><td>0.148</td><td>24.05</td><td>0.896</td><td>0.146</td></tr><tr><td>SPARF-128</td><td>✗</td><td>32.97</td><td>0.986</td><td>0.028</td><td>31.69</td><td>0.972</td><td>0.054</td><td>30.89</td><td>0.978</td><td>0.044</td><td>32.26</td><td>0.981</td><td>0.040</td></tr><tr><td>Tri-Planes</td><td>✓</td><td>30.11</td><td>0.962</td><td>0.024</td><td>30.13</td><td>0.949</td><td>0.043</td><td>28.86</td><td>0.949</td><td>0.040</td><td>29.67</td><td>0.950</td><td>0.039</td></tr><tr><td>Fused-Planes-ULW</td><td>✓</td><td>27.60</td><td>0.938</td><td>0.054</td><td>29.91</td><td>0.948</td><td>0.064</td><td>28.44</td><td>0.945</td><td>0.050</td><td>28.87</td><td>0.942</td><td>0.051</td></tr><tr><td>Fused-Planes</td><td>✓</td><td>30.15</td><td>0.964</td><td>0.021</td><td>31.20</td><td>0.961</td><td>0.043</td><td>29.69</td><td>0.958</td><td>0.035</td><td>30.05</td><td>0.954</td><td>0.033</td></tr></tbody></table>

<table><caption>Table 10. Per-scene quantitative comparison on ShapeNet Sofas.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Planar</th><th colspan="3">Sofa 1</th><th colspan="3">Sofa 2</th><th colspan="3">Sofa 3</th><th colspan="3">Sofa 4</th></tr><tr><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th></tr></thead><tbody><tr><td>Vanilla-NeRF</td><td>✗</td><td>31.06</td><td>0.966</td><td>0.034</td><td>31.83</td><td>0.965</td><td>0.032</td><td>33.58</td><td>0.940</td><td>0.122</td><td>36.82</td><td>0.984</td><td>0.013</td></tr><tr><td>Instant-NGP</td><td>✗</td><td>29.91</td><td>0.969</td><td>0.031</td><td>33.60</td><td>0.975</td><td>0.027</td><td>35.42</td><td>0.974</td><td>0.013</td><td>35.54</td><td>0.977</td><td>0.016</td></tr><tr><td>TensoRF</td><td>✗</td><td>32.92</td><td>0.987</td><td>0.011</td><td>37.17</td><td>0.992</td><td>0.010</td><td>37.47</td><td>0.987</td><td>0.009</td><td>37.98</td><td>0.987</td><td>0.013</td></tr><tr><td>3DGS</td><td>✗</td><td>30.85</td><td>0.986</td><td>0.020</td><td>33.95</td><td>0.989</td><td>0.025</td><td>34.60</td><td>0.982</td><td>0.023</td><td>33.46</td><td>0.984</td><td>0.047</td></tr><tr><td>K-Planes</td><td>✓</td><td>25.84</td><td>0.947</td><td>0.054</td><td>32.28</td><td>0.974</td><td>0.028</td><td>32.90</td><td>0.968</td><td>0.028</td><td>32.59</td><td>0.964</td><td>0.037</td></tr><tr><td>CodeNeRF</td><td>✗</td><td>25.47</td><td>0.938</td><td>0.113</td><td>29.80</td><td>0.938</td><td>0.068</td><td>29.18</td><td>0.919</td><td>0.139</td><td>30.14</td><td>0.944</td><td>0.100</td></tr><tr><td>CodeNeRF (MSIG)</td><td>✗</td><td>24.61</td><td>0.928</td><td>0.121</td><td>29.67</td><td>0.936</td><td>0.067</td><td>28.05</td><td>0.899</td><td>0.130</td><td>29.39</td><td>0.938</td><td>0.092</td></tr><tr><td>SPARF-32</td><td>✗</td><td>23.97</td><td>0.929</td><td>0.135</td><td>27.81</td><td>0.935</td><td>0.103</td><td>27.42</td><td>0.914</td><td>0.122</td><td>28.48</td><td>0.926</td><td>0.133</td></tr><tr><td>SPARF-128</td><td>✗</td><td>30.54</td><td>0.985</td><td>0.033</td><td>33.30</td><td>0.983</td><td>0.034</td><td>34.13</td><td>0.979</td><td>0.035</td><td>33.52</td><td>0.973</td><td>0.054</td></tr><tr><td>Tri-Planes</td><td>✓</td><td>26.34</td><td>0.929</td><td>0.082</td><td>29.24</td><td>0.930</td><td>0.091</td><td>28.89</td><td>0.903</td><td>0.121</td><td>29.43</td><td>0.922</td><td>0.118</td></tr><tr><td>Fused-Planes-ULW</td><td>✓</td><td>24.72</td><td>0.921</td><td>0.130</td><td>30.29</td><td>0.938</td><td>0.047</td><td>29.99</td><td>0.917</td><td>0.091</td><td>31.06</td><td>0.947</td><td>0.069</td></tr><tr><td>Fused-Planes</td><td>✓</td><td>27.83</td><td>0.964</td><td>0.020</td><td>31.75</td><td>0.958</td><td>0.024</td><td>31.71</td><td>0.945</td><td>0.032</td><td>32.39</td><td>0.963</td><td>0.038</td></tr></tbody></table>
<table><caption>Table 11. Per-scene quantitative comparison on ShapeNet Speakers.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Planar</th><th colspan="3">Speaker 1</th><th colspan="3">Speaker 2</th><th colspan="3">Speaker 3</th><th colspan="3">Speaker 4</th></tr><tr><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th></tr></thead><tbody><tr><td>Vanilla-NeRF</td><td>✗</td><td>35.95</td><td>0.962</td><td>0.065</td><td>30.97</td><td>0.980</td><td>0.021</td><td>35.41</td><td>0.970</td><td>0.032</td><td>33.65</td><td>0.977</td><td>0.015</td></tr><tr><td>Instant-NGP</td><td>✗</td><td>36.56</td><td>0.983</td><td>0.016</td><td>27.31</td><td>0.955</td><td>0.043</td><td>34.75</td><td>0.980</td><td>0.024</td><td>31.16</td><td>0.966</td><td>0.022</td></tr><tr><td>TensoRF</td><td>✗</td><td>38.73</td><td>0.989</td><td>0.012</td><td>29.74</td><td>0.978</td><td>0.024</td><td>37.57</td><td>0.988</td><td>0.017</td><td>33.60</td><td>0.976</td><td>0.016</td></tr><tr><td>3DGS</td><td>✗</td><td>34.45</td><td>0.981</td><td>0.059</td><td>24.21</td><td>0.870</td><td>0.095</td><td>31.70</td><td>0.979</td><td>0.071</td><td>28.11</td><td>0.940</td><td>0.071</td></tr><tr><td>K-Planes</td><td>✓</td><td>33.80</td><td>0.969</td><td>0.034</td><td>21.84</td><td>0.905</td><td>0.102</td><td>32.33</td><td>0.963</td><td>0.046</td><td>27.19</td><td>0.923</td><td>0.069</td></tr><tr><td>CodeNeRF</td><td>✗</td><td>29.81</td><td>0.894</td><td>0.133</td><td>24.91</td><td>0.931</td><td>0.106</td><td>28.85</td><td>0.925</td><td>0.178</td><td>28.26</td><td>0.935</td><td>0.126</td></tr><tr><td>CodeNeRF (MSIG)</td><td>✗</td><td>23.73</td><td>0.875</td><td>0.167</td><td>24.32</td><td>0.922</td><td>0.114</td><td>27.31</td><td>0.905</td><td>0.184</td><td>26.11</td><td>0.918</td><td>0.129</td></tr><tr><td>SPARF-32</td><td>✗</td><td>28.26</td><td>0.915</td><td>0.156</td><td>22.10</td><td>0.916</td><td>0.130</td><td>27.77</td><td>0.905</td><td>0.179</td><td>24.94</td><td>0.908</td><td>0.163</td></tr><tr><td>SPARF-128</td><td>✗</td><td>30.38</td><td>0.955</td><td>0.084</td><td>27.06</td><td>0.958</td><td>0.061</td><td>29.81</td><td>0.947</td><td>0.093</td><td>30.39</td><td>0.965</td><td>0.060</td></tr><tr><td>Tri-Planes</td><td>✓</td><td>29.25</td><td>0.911</td><td>0.147</td><td>23.11</td><td>0.903</td><td>0.098</td><td>29.30</td><td>0.914</td><td>0.160</td><td>26.41</td><td>0.907</td><td>0.132</td></tr><tr><td>Fused-Planes-ULW</td><td>✓</td><td>30.38</td><td>0.932</td><td>0.134</td><td>26.75</td><td>0.948</td><td>0.049</td><td>29.93</td><td>0.940</td><td>0.104</td><td>29.83</td><td>0.942</td><td>0.063</td></tr><tr><td>Fused-Planes</td><td>✓</td><td>32.89</td><td>0.966</td><td>0.042</td><td>26.40</td><td>0.949</td><td>0.046</td><td>30.63</td><td>0.951</td><td>0.066</td><td>30.04</td><td>0.947</td><td>0.057</td></tr></tbody></table>

<table><caption>Table 12. Per-scene quantitative comparison on ShapeNet Furnitures.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Planar</th><th colspan="3">Furniture 1</th><th colspan="3">Furniture 2</th><th colspan="3">Furniture 3</th><th colspan="3">Furniture 4</th></tr><tr><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th><th>PSNR</th><th>SSIM</th><th>LPIPS</th></tr></thead><tbody><tr><td>Vanilla-NeRF</td><td>✗</td><td>38.42</td><td>0.976</td><td>0.014</td><td>35.56</td><td>0.985</td><td>0.015</td><td>38.01</td><td>0.978</td><td>0.018</td><td>35.75</td><td>0.965</td><td>0.028</td></tr><tr><td>Instant-NGP</td><td>✗</td><td>35.12</td><td>0.951</td><td>0.032</td><td>33.64</td><td>0.977</td><td>0.023</td><td>33.91</td><td>0.954</td><td>0.035</td><td>34.49</td><td>0.959</td><td>0.031</td></tr><tr><td>TensoRF</td><td>✗</td><td>38.03</td><td>0.974</td><td>0.018</td><td>36.23</td><td>0.989</td><td>0.013</td><td>36.22</td><td>0.973</td><td>0.021</td><td>35.23</td><td>0.964</td><td>0.031</td></tr><tr><td>3DGS</td><td>✗</td><td>34.49</td><td>0.991</td><td>0.033</td><td>31.14</td><td>0.989</td><td>0.051</td><td>33.15</td><td>0.975</td><td>0.050</td><td>33.49</td><td>0.998</td><td>0.052</td></tr><tr><td>K-Planes</td><td>✓</td><td>34.41</td><td>0.951</td><td>0.028</td><td>30.72</td><td>0.960</td><td>0.044</td><td>33.01</td><td>0.949</td><td>0.045</td><td>32.51</td><td>0.941</td><td>0.055</td></tr><tr><td>CodeNeRF</td><td>✗</td><td>29.82</td><td>0.909</td><td>0.157</td><td>27.22</td><td>0.928</td><td>0.139</td><td>30.20</td><td>0.932</td><td>0.224</td><td>30.68</td><td>0.944</td><td>0.139</td></tr><tr><td>CodeNeRF (MSIG)</td><td>✗</td><td>28.65</td><td>0.868</td><td>0.156</td><td>26.18</td><td>0.914</td><td>0.148</td><td>28.47</td><td>0.899</td><td>0.234</td><td>29.11</td><td>0.918</td><td>0.146</td></tr><tr><td>SPARF-32</td><td>✗</td><td>27.30</td><td>0.882</td><td>0.200</td><td>24.60</td><td>0.889</td><td>0.170</td><td>28.63</td><td>0.904</td><td>0.194</td><td>27.84</td><td>0.891</td><td>0.189</td></tr><tr><td>SPARF-128</td><td>✗</td><td>29.89</td><td>0.936</td><td>0.116</td><td>28.68</td><td>0.965</td><td>0.070</td><td>31.51</td><td>0.953</td><td>0.094</td><td>30.02</td><td>0.940</td><td>0.122</td></tr><tr><td>Tri-Planes</td><td>✓</td><td>26.88</td><td>0.875</td><td>0.175</td><td>27.17</td><td>0.913</td><td>0.180</td><td>28.05</td><td>0.902</td><td>0.237</td><td>27.58</td><td>0.886</td><td>0.250</td></tr><tr><td>Fused-Planes-ULW</td><td>✓</td><td>25.80</td><td>0.891</td><td>0.219</td><td>29.91</td><td>0.947</td><td>0.073</td><td>29.67</td><td>0.938</td><td>0.173</td><td>31.20</td><td>0.958</td><td>0.102</td></tr><tr><td>Fused-Planes</td><td>✓</td><td>30.19</td><td>0.962</td><td>0.030</td><td>30.54</td><td>0.957</td><td>0.039</td><td>29.81</td><td>0.947</td><td>0.102</td><td>32.34</td><td>0.972</td><td>0.042</td></tr></tbody></table>
<figure><img src="image_7.png" /><figcaption>Figure 7. **Qualitative comparison.** Comparison of NVS quality on test views of four scenes from Basel Faces.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 8. **Qualitative comparison.** Comparison of NVS quality on test views of four scenes from the Cars category of ShapeNet.</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Figure 9. **Qualitative comparison.** Comparison of NVS quality on test views of four scenes from the Sofas category of ShapeNet.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>Figure 10. **Qualitative comparison.** Comparison of NVS quality on test views of four scenes from the Speakers category of ShapeNet.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Figure 11. **Qualitative comparison.** Comparison of NVS quality on test views of four scenes from the Furniture category of ShapeNet.</figcaption></figure>
**Algorithm 1** Training a large set of scenes.

1: **Input:** $S_1, S_2, N_1, N_2, V, E_\phi, D_\psi, \mathcal{R}_\alpha, N_{\text{epoch}}^{(1)}, N_{\text{epoch}}^{(2)}, N_{\text{epoch}}^{(\text{LS})}, \lambda^{(\text{latent})}, \lambda^{(\text{RGB})}, \lambda^{(\text{ae})}$, optimizer
2: **Random initialization:** $\mathcal{T}_1^{\text{mic}}, \mathcal{T}_2^{\text{mic}}, W, \mathcal{B}$
3:
4: // Subset 1
5: **for** $N_{\text{epoch}}^{(1)}$ **steps do**
6:     **for** $(i, j)$ in shuffle($[1, N_1] \times [1, V]$) **do**
7:         // Compute Micro-Macro Planes
8:         $T_i^{(\text{mic})}, T_i^{(\text{mac})} \leftarrow \mathcal{T}_1^{(\text{mic})}[i], W_i\mathcal{B}$
9:         $T_i \leftarrow T_i^{(\text{mic})} \oplus T_i^{(\text{mac})}$
10:     // Encode, Render & Decode
11:     $x_{i,j}, p_{i,j} \leftarrow S_1[i][j]$
12:     $z_{i,j} \leftarrow E_\phi(x_{i,j})$
13:     $\tilde{z}_{i,j} \leftarrow \mathcal{R}_\alpha(T_i, p_{i,j})$
14:     $\hat{x}_{i,j}, \tilde{x}_{i,j} \leftarrow D_\psi(z_{i,j}), D_\psi(\tilde{z}_{i,j})$
15:     // Compute losses
16:     $L_{i,j}^{(\text{latent})} \leftarrow \|z_{i,j} - \tilde{z}_{i,j}\|_2^2$
17:     $L_{i,j}^{(\text{RGB})} \leftarrow \|x_{i,j} - \tilde{x}_{i,j}\|_2^2$
18:     $L_{i,j}^{(\text{ae})} \leftarrow \|x_{i,j} - \hat{x}_{i,j}\|_2^2$
19:     $L_{i,j} \leftarrow \lambda^{(\text{latent})} L_{i,j}^{(\text{latent})} + \lambda^{(\text{RGB})} L_{i,j}^{(\text{RGB})} + \lambda^{(\text{ae})} L_{i,j}^{(\text{ae})}$
20:     // Backpropagate
21:     $T_i^{(\text{mic})}, W_i, \mathcal{B}, \alpha, \phi, \psi \leftarrow \text{optimizer.step}(L_{i,j})$
22:     **end for**
23:     **end for**
24:
25: // Subset 2
26: epoch=1
27: **for** $N_{\text{epoch}}^{(2)}$ **steps do**
28:     **for** $(i, j)$ in shuffle($[N_1 + 1, N_1 + N_2] \times [1, V]$) **do**
29:         // Compute Micro-Macro Planes
30:         $T_i^{(\text{mic})}, T_i^{(\text{mac})} \leftarrow \mathcal{T}_2^{(\text{mic})}[i], W_i\mathcal{B}$
31:         $T_i \leftarrow T_i^{(\text{mic})} \oplus T_i^{(\text{mac})}$
32:         // Encode, Render & Decode
33:         $x_{i,j}, p_{i,j} \leftarrow S_2[i][j]$
34:         $z_{i,j} \leftarrow E_\phi(x_{i,j})$
35:         $\tilde{z}_{i,j} \leftarrow \mathcal{R}_\alpha(T_i, p_{i,j})$
36:         $\tilde{x}_{i,j} \leftarrow D_\psi(\tilde{z}_{i,j})$
37:
38:     **if** $\text{epoch} \le N_{\text{epoch}}^{(\text{LS})}$ **then**
39:         // Latent Supervision
40:         $L_{i,j}^{(\text{latent})} \leftarrow \|z_{i,j} - \tilde{z}_{i,j}\|_2^2$
41:         $T_i^{(\text{mic})}, W_i, \mathcal{B}, \alpha \leftarrow \text{optimizer.step}(L_{i,j}^{(\text{latent})})$
42:     **else**
43:         // RGB Alignment
44:         $L_{i,j}^{(\text{RGB})} \leftarrow \|x_{i,j} - \tilde{x}_{i,j}\|_2^2$
45:         $T_i^{(\text{mic})}, W_i, \mathcal{B}, \alpha, \psi \leftarrow \text{optimizer.step}(L_{i,j}^{(\text{RGB})})$
46:     **end if**
47:     **end for**
48:     epoch $\leftarrow$ epoch + 1
49: **end for**
<table><caption>Table 13. Fused-Planes subset 1 hyperparameters.</caption><thead><tr><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td colspan="2"><strong>General</strong></td></tr><tr><td>Number of scenes N<sub>1</sub></td><td>500</td></tr><tr><td>Pretraining epochs</td><td>50</td></tr><tr><td>Training epochs</td><td>50</td></tr><tr><td colspan="2"><strong>Tri-Planes</strong></td></tr><tr><td>Number of micro feature F<sub>mic</sub></td><td>10</td></tr><tr><td>Number of macro feature F<sub>mac</sub></td><td>22</td></tr><tr><td>Number of base plane M</td><td>50</td></tr><tr><td>Tri-Planes resolution</td><td>64</td></tr><tr><td colspan="2"><strong>Loss</strong></td></tr><tr><td>λ<sup>(latent)</sup></td><td>1</td></tr><tr><td>λ<sup>(RGB)</sup></td><td>1</td></tr><tr><td>λ<sup>(ae)</sup></td><td>0.1</td></tr><tr><td colspan="2"><strong>Optimization (warm-up)</strong></td></tr><tr><td>Optimizer</td><td>Adam</td></tr><tr><td>Batch size</td><td>512</td></tr><tr><td>Learning rate (Tri-Planes T<sub>i</sub><sup>(mic)</sup>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (Triplane renderer R<sub>α</sub>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (Coefficients w<sub>i</sub><sup>k</sup>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (Base planes B<sub>k</sub>)</td><td>10<sup>-2</sup></td></tr><tr><td>Scheduler</td><td>Multistep</td></tr><tr><td>Decay factor</td><td>0.3</td></tr><tr><td>Decay milestones</td><td>[20, 40]</td></tr><tr><td colspan="2"><strong>Optimization (training)</strong></td></tr><tr><td>Optimizer</td><td>Adam</td></tr><tr><td>Batch size</td><td>32</td></tr><tr><td>Learning rate (encoder)</td><td>10<sup>-4</sup></td></tr><tr><td>Learning rate (decoder)</td><td>10<sup>-4</sup></td></tr><tr><td>Learning rate (Tri-Planes T<sub>i</sub><sup>(mic)</sup>)</td><td>10<sup>-4</sup></td></tr><tr><td>Learning rate (Triplane renderer R<sub>α</sub>)</td><td>10<sup>-4</sup></td></tr><tr><td>Learning rate (Coefficients w<sub>i</sub><sup>k</sup>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (Base planes B<sub>k</sub>)</td><td>10<sup>-2</sup></td></tr><tr><td>Scheduler</td><td>Multistep</td></tr><tr><td>Decay factor</td><td>0.3</td></tr><tr><td>Decay milestones</td><td>[20, 40]</td></tr></tbody></table>
<table><caption>Table 14. Fused-Planes subset 2 hyperparameters.</caption><thead><tr><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td colspan="2"><strong>General</strong></td></tr><tr><td>Number of scenes N<sub>2</sub></td><td>1500</td></tr><tr><td>Number of Latent Supervision epochs N<sub>epoch</sub><sup>(LS)</sup></td><td>30</td></tr><tr><td>Number of RGB Alignment epochs N<sub>epoch</sub><sup>(RA)</sup></td><td>50</td></tr><tr><td colspan="2"><strong>Tri-Planes</strong></td></tr><tr><td>Number of micro feature F<sub>mic</sub></td><td>10</td></tr><tr><td>Number of macro feature F<sub>mac</sub></td><td>22</td></tr><tr><td>Number of base plane M</td><td>50</td></tr><tr><td>Tri-Planes resolution</td><td>64</td></tr><tr><td colspan="2"><strong>Loss</strong></td></tr><tr><td>λ<sup>(latent)</sup></td><td>1</td></tr><tr><td>λ<sup>(RGB)</sup></td><td>1</td></tr><tr><td colspan="2"><strong>Optimization (Latent Supervision)</strong></td></tr><tr><td>Optimizer</td><td>Adam</td></tr><tr><td>Batch size</td><td>32</td></tr><tr><td>Learning rate (Tri-Planes T<sub>i</sub><sup>(mic)</sup>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (Triplane renderer R<sub>α</sub>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (Coefficients w<sub>i</sub><sup>k</sup>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (Base planes B<sub>k</sub>)</td><td>10<sup>-2</sup></td></tr><tr><td>Scheduler</td><td>Exponential decay</td></tr><tr><td>Decay factor</td><td>0.941</td></tr><tr><td colspan="2"><strong>Optimization (RGB Alignment)</strong></td></tr><tr><td>Optimizer</td><td>Adam</td></tr><tr><td>Batch size</td><td>32</td></tr><tr><td>Learning rate (decoder)</td><td>10<sup>-4</sup></td></tr><tr><td>Learning rate (Tri-Planes T<sub>i</sub><sup>(mic)</sup>)</td><td>10<sup>-3</sup></td></tr><tr><td>Learning rate (Triplane renderer R<sub>α</sub>)</td><td>10<sup>-3</sup></td></tr><tr><td>Learning rate (coefficients w<sub>i</sub><sup>k</sup>)</td><td>10<sup>-2</sup></td></tr><tr><td>Learning rate (nase planes B<sub>k</sub>)</td><td>10<sup>-2</sup></td></tr><tr><td>Scheduler</td><td>Exponential decay</td></tr><tr><td>Decay factor</td><td>0.941</td></tr></tbody></table>