Attention Beats Concatenation for Conditioning Neural Fields
Authors: Daniel Rebain
Mark J. Matthews
Kwang Moo Yi
Gopal Sharma
Dmitry Lagun
Andrea Tagliasacchi
Venue: Accepted by TMLR ()

Abstract: Neural fields model signals by mapping coordinate inputs to sampled values. They are becoming an increasingly important backbone architecture across many fields from vision and graphics to biology and astronomy. In this paper, we explore the differences between common conditioning mechanisms within these networks, an essential ingredient in shifting neural fields from memorization of signals to generalization, where the set of signals lying on a manifold is modelled jointly. In particular, we are interested in the scaling behaviour of these mechanisms to increasingly high-dimensional conditioning variables. As we show in our experiments, high-dimensional conditioning is key to modelling complex data distributions, thus it is important to determine what architecture choices best enable this when working on such problems. To this end, we run experiments modelling 2D, 3D, and 4D signals with neural fields, employing concatenation, hyper-network, and attention-based conditioning strategies -- a necessary but laborious effort that has not been performed in the literature.
We find that attention-based conditioning outperforms other approaches in a variety of settings.

Published in Transactions on Machine Learning Research (11/2022)
Attention Beats Concatenation for Conditioning Neural Fields
Daniel Rebain
drebain @igoollli@@ooggccccoo
University of British Columbia Google Research
Mark J. Matthews
mjmattheus Sranutthews@gooll...cc com
Google Research
Kwang Moo Yi University of British Columbia
kmyi@cs caap:dsssubbc.oo ubc.ca ca
Gopal Sharma University of British Columbia
Dmitry Lagun
dlagun@google.com
Google Research
Andrea Tagliasacchi
taglia Siliiaagoogee..o
Google Research
Simon Fraser University
Reviewed on OpenReview: https: // openreview. net/ formm? id- GaqdMrFQQE
Abstract
Neural fields model signals by mapping coordinate inputs sampled values. They are becoming an increasingly important backbone architecture across many fields from vision and graphics to biology and astronomy. In this paper, we explore the differences between common conditioning mechanisms within these networks, essential ingredient in shifting neural fields from memorization of signals to generalization, where the set of signals lying on a manifold is modelled jointly. In particular, we are interested in the scaling behaviour of these mechanisms to increasingly high-dimensional conditioning variables. As we show our experiments, high-dimensional conditioning key to modelling complex data distributions, thus it is important to determine what architecture choices best enable this when working such problems. To this end, we experiments modelling 2D, 3D, and 4D signals with neural fields, employing concatenation, hepeer,,etorrk, and attention-based conditioning strategies necessary but laborious effort that has not been performed in the literature. We find that attention-basel conditioning outperforms other approaches in a variety of settings.
1 Introduction
Neural fields, also called coordinate-based neural networks, have demonstrated excellent results in learning complex signals. Neural fields learn to reproduce signals by mapping input coordinates to output values. They are most commonly implemented as Multilayer Perceptrons (MLP) and have found widesproad application over range of tasks: 1D time sequences (Sitzmann et al., 2020), 2D image regression (Tancik et al., 2020), 3D radiance fields (Mildenhall et al., 2020), and 4D light fields (Suhail et al., 2022).
Many previous works overfit single example (c.g. an image, sound, object, or scene) (Sitzmann et al., 2020; Mildenhall al., 2020), employing neural networks to "memorize" the target signal. However, there are many potential applications which require jointly representing multiple signals, commonly distribution over

---

Published in Transactions on Machine Learning Research (11/2022)
Reconstruction PSNR
Auto-encoder (CelebA-HQ)
NeRF (SRN Chairs)
Light Field (SRN Chairs)
28
Allention
Htpee-network
27.5
Attantis
Hypernnework Concatenation
24
Attection
Hyper-nwhwerk Concatosation
Concatenation
26
25.0
22
24
20
22.5
20.0
512
2048
8192
512
2048
8192
512
2048
8192
Latent Code Dimension
Figure 1: Main results The relationship between representation effectiveness, as mcasured by PSNR, and representation complexity, as measured by latent code size; the plot reveals how attention conditioning outperform concatenation and hyper-networks when scaling to larger representations.
manifold (a smooth subspace of all possible signals). This generalization of neural fields has been achieved though various means: latent vector conditioned MLPs (Park et al., 2019), hyper-nctworks (Sitzmann et al., 2020), and set-latent transformer representations (Sajjadi et al., 2022). In all these means, the location of a particular signal the output manifold is represented with some form of latent code.
The properties of the latent codes are important in determining what a conditional network is capable of modelling. Of particular interest. to us is the dimensionality of the latent code, which directly determines the dimensionality of the manifold on which the output signals lif. If we wish to perfectly model some distribution of data, then we must use a latent dimension at least equal to that of the manifold on which that distribution lies. We demonstrate this effect on a toy dataset where we explicitly control the dimensionality of the data manifold, and find that increasing dimension of this manifold corresponds to worse reconstruction quality across different models.
Based on this observation, and inspired by methods such as VQ-VAE (van den Oord et al., 2017), which have utilized high-dimensional conditioning to great effect in CNN-based networks, we wish to determine whether neural field networks are capable of scaling up to condition on such high-dimensional latent codes, and what factors may affect this. Unlike CNNs, which "share" computation across the different (diserete) output samples of signals that they model, neural fields typically repeat the majority of their computation for each sample. As such, the question of how to most efficiently incorporate high-dimensional conditioning signals into a neural field network does not have obvious answer problem which we intend to address in this paper. To facilitate this, we undertake a comprehensive study of three common methods of conditioning on latent inputs in neural fields: concatenated latent vectors, hyper-networks, and attention- set latent representations. We evaluate each of these methods across three application domains: image auto-encoding, novel view synthesis using neural volumetric rendering, and light. field norksrks
Performing the experiments reported in this paper required a very large amount of compute time: on the order of 23k GPU-hours. Due to the significant expense involved, we chose experimental parameters carefully and focused architectural choices which appeared most likely to affect the outcome of the experiments, and therefore the conclusions of our analysis. We also did not run any additional typer-parameter tuning or searches on top of our primary sweeps and ablation. It is our hope that by incurring this expense and sharing our conclusions, others can avoid the need to run similarly costly experiments when making design decisions for neural field implementations.
In summary, our contributions include:
A series of extensive quantitative evaluations, where we compare how concatenation, hyper-network, and attention-based neural field architectures perform in modelling high-entropy data distributions for a number of application domains: 2D image auto-encoding, 3D radiance fields, and 4D light fields, benchmarked on standard datasets.
An ablation study comparing different approaches conditioning MLP neural field networks by concate- nation when the latent dimension very high. We find that for very large latent codes, splitting the code between the hidden layers of the ML.P provides the best results;
2

---

Published in Transactions on Machine Learning Research (11/2022)
ImNET
Transformers
SRT
IDR NeRF
CPPN
DeepSDF
NeuralVol
CodeNerf
OURS
OccNet
SRN
SIREN
pi-GAN
LoLNeRF
Figure Related works a temporal overview of how techniques have evolved, with arrows roughly demoting the dependency graph of technical contributions. At high-level, reveal how recently proposed coordinate network architectures (Sajjadi et al., 2022) are significantly more effective than MLPs (Stanley, 2007) for the task of conditional image generation (Rebain et al., 2022). Abbreviations for the various methods are highlighted in Section 2.
An analysis showing that attention-based conditioning is broadly more elfective for high-dimensional latent codes, given a particular compute budget.
2
Related Works
Our work mainly focuses on coordinato-based networks, also called noural fields. Introduced by Stanley (2007) under the name of Conditional Pattern Producing Networks (CPPN), recently became popular as differentiable representations for 3D data (Tewari al., 2021; Xie et al., 2022). We now position our work with respect to the temporal progression of research in this topic (Figure 2), and in particular discussing how conditioning is implemented in architectures.
Coordinate networks w/ 3D supervision. Neural implicit networks for 3D shape representation receive as input a 3D point coordinate along with latent encoding the shape and output either oceu- pancy (OccNet (Mescheder et al., 2019) and IMNet (Chen & Zhang, 2019)) (truncated) signed distance fields (DeepSDF Park et al. (2019)). These architectures are either trained with an auto-encoder (LMNet, OccNet) auto-decoder (DeepSDF) fashion, and rely on direct 3D supervision for training. Conditioning is implemented either by input concatenation (IMNet, DeepSDF), or by modulation of activations (DeepSDF))
Coordinate networks w/ 2D supervision. Moving away from 3D supervision, Scene Representation Networks (SRN) (Sitzmann et al., 2019) pioncered the use of 2D images supervision of an underlying 3D neural scene, by relying on differentiable ray-marching. Similarly to previously discussed works, each scene in SRN is represented as latent code while conditioning realized via hyper-networks (Ha et al., 2017), but the generated images lack fine-grained dotails. Mcanwhile, NeuralVolumes (Lombardi et al., 2019) introducod the use of differentiable volume rendering as a way to learn models with high visual-quality, but still relied on grids for scene representation. Marrying methods that store radiance within a coordinate network (Ren et al., 2013) with differentiable volume rendering VolureeV)l neural scene representations (SRN), Mildenhall et al. (2020) introduced Neural Radiance Fields (NeRF) scene via differentiable rendering, and achieved novel-view synthesis results of unforeseen visual quality
effective way capture a 3D
Overcoming the spectral bias. To represent high-frequency signals neural scene representations one must overcome the so-called "spectral bias" of MLP architectures (Basri et al., 2020; Rahaman al., 2019) an inductive bias that causes fully connected neural networks to prefer representing low- fow-frequeccy signals. NeRF overcame this issue by borrowing the ideas of (sinusoidal) positional encoding from transformers (Vaswani et al., 2017), SIREN (Sitzmann et al., 2020) proposed the use of sinusoidal activations (and careful initialization). These are relatively similar solutions, with the core difference that sinusoidal encoding is applied at the input
CConurrent work by Yariv et (2020) and Niemeyer et (2020) similar spirit but have more restrictive setup.
3

---

Published in Transactions on Machine Learning Research (11/2022)
2D Image Encoding
3D Radiance Field
4D Light Field
(x, y
(r,g,b)
(xr, y, 2
(r,g, b,
(xr, y, u, 4)
(r, g, b)
(x,
(x,, y, 2)
Figure 3: Application domains We consider three main application domains: 2D image encoding. 3D radiance fields (radiance integrated along rays, but without directional dependence), 4D light fields.
layer in NeRF, and throughout the entire architecture STREN. Further, note that both of these solutions originally targeted the representation of a single scene, that is, working the overfitting (i.e. non-conditional) training regime; the next section will cover multi-scene training, which is achieved via conditioning.
Conditional networks w/ 2D supervision. The recent pi-GAN (Chan et al., 2021) and CodeN- eRF (Jang & Agapito, 2021) demonstrate conditioning FILM SIREN and MLP models respectively obtain (class-specifie) conditional neural radiance fields for novel view synthesis. While CodeNeRE struggles with low image quality, pi-GAN is capable of generating high quality samples by leveraging generative adversarial networks. Somewhat concurrently to CodeNeRF, Rebain et al. (2022) has shown that, in the conditional novel view synthesis setting, LoLNeRF can achieve competitive performance in comparison to GANs given a sulficiently large network and latent code. However, we reveal how there are diminishing returns in employing large MLP networks and large latent codes when MLP architectures are employed. We investigate this problem from the point of view of representation power. Our extensive experiments reveal how recently proposed alternatives to MLPs for coordinate-bast based neural networks (Sajjadi et al., 2022)(SRT)
overcome this issue across number of application domains.
3 Method
We experimentally analyze the representation power of different neural field conditioning mechanisms. With that objective, we first introduce the basics of neural fields in Section 3.1, their training strategies in Section 3.2, and their architecture in Section 3.3.
3.1 Neural Fields
Given a signal (1D for audio, 2D for images, 3D for geometry etc., See Figure 3) mapping bounded of coordinate inputs X c IRd to outputs O E Rm neural field with parameters represents signal, taking coordinate E X input and output fo O:
So Rd R", x fo(x)
(1)
Examples include fields receiving as input:
x E IR 1D time coordinates, outputting scalar audio intensity (Sitzmann et al., 2020);
x E 2 2D pixel coordinates, outputting RGB color for pixels (Stanley, 2007);
x E 3 3D coordinate, outputting (truncated) signed distance from a shape (Park al., 2019);
x R 4D encoded camera rays, outputting pixel color (Sitzmann et al., 2021):
R 3D coordinate and 2D view direction, outputting density and radiance (Mildenhall et al., 2020).
4

---

Published in Transactions on Machine Learning Research (11/2022)
Networks can be supervised minimizing loss function over the observations:
arg min E and
- S
(2)
When using MLP architectures to implement for directly inputting the coordinates to the neural network, they have a bias towards representing low-fregg wwwfrequency signals (Rahaman et al., 2019), loading to low-quality signal reconstruction. To encourage high-frequency signal learning, the input coordinates may be independently mapped to higher dimension using sine and cosine functions with increasingly large frequencies (Vaswani et al., 2017; Mildenhall et al., 2020):
(अ) (sin(2"mxr) sin(2' T2), ... sin (x)) cos(2" πх), cos ( "os c0s(2'-1 Tx))
Several other formulations of position encoding have been explored (Rahimi & Recht, 2007; Barron et al., 2021; Tancik et al., 2020), but we use this one in our experiments as it has proven widely effective and remains commonly used.
3.2 Neural Fields: training methodology
Coordinate networks can be used to overfit (memorize) a bounded signal for such purposes as image de- noising (Sitzmann et al., 2020) or novel view synthesis of scenes given finite set of input images and associated poses (Mildenhall et al., 2020), but this memorization process does not generalize to new inputs. To overcome these limitations, conditional networks have been designed, where the decoder takes as input an instance-specife latent code z E R" along with the coordinates to regress desired quantities, Below, we describe two approaches to training networks conditioned on latent codes: Y auto-encoders? and @ auto-decoders.
Auto-encoder. An auto-encoder (or encoder-decoder) employs domain-specific encoder & parameterized by 0g that takes a dataset element I E I as imput and outputs the latent code as ; + E(I = z E R" The element I can be an image, point cloud, audio signal etc. decoder D parameterized by 0g accepts the latent code and reconstructs the input as D : z + D(z) = î. These networks are trained to jointly optimize encoder and decoder parameters using the reconstruction objective:
arg min arg 0,
E [I|Do (Es (T) - [
(4)
Auto-decoder. Also known as Generative Latent Optimization (GLO) (Bojanowski et al., 2017). auto- decoders are a form of generative model which generates an output conditioned on a latent code z. Here, the latent code is not predicted by an encoder, rather jointly optimized with the decoder parameters. A code-book }= is used where each row consists of a latent code correspouding to each training instance. This alleviates the need of designing task-specific encoder. This neural network is trained to optimize the parameters of the decoder 0g and the latent codes 2 using the reconstruction loss:
arg min 2,0a (I,)
E
[l (z) - -II p(z)]
where p is an optional regularization function applied to the latent when particular instribution is desired. For example, choosing p(-) = || 112 results in the codes Z being distributed as a Gaussian (Park et al., 2019). During inference, commonlyy referred to as lest-time optimization in the literature, the latent code is discovered by optimizing the above loss function while keeping the network parameters constant:
arg min E (l) - 1 p(z)]
(6)
AA extension of auto-encuders, Variational Auto-Encoders (VAE) (Kingmaa & Welling, 2013), probabilistic formulation that define distribution of latent codes that generate clements matching the input dataset distribution. Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) another common method of synthesis that map latent code to output. However, for the purpose of this paper do analyze VAFs and GANs, as are interested in generative aspects of the problem, and benchmark our results as a regression task.
5

---

Published in Transactions on Machine Learning Research (11/2022)
Concatenation
Hyper Network
Attention
Iyyper
z
MLP
W
x2
Oupp
vinniion Attention
K
Indin
MLP
V
MLP
Outpur
7xx)
7(x)
MLP
7(x)
Figure Architectures Multi-layer perceptrons (MLPs) can be conditioned using concatcnation of coordinate and latent. code (left). Hyper-networks use a secondary MLP that compules weights for the primary network which operates the coordinate (middle). Attention networks condition set-latent. representation that is incorporated into the network through attention layers, with query vectors being derived from the coordinates (right).
3.3 Neural Fields: decoder architectures
Many neural field architectures have been proposed in the literature. In this section, we review some common varictics including the ones tested our experiments. Of particular relevance is the different ways in which these networks incorporate condilioning signals. For additional details on how we implement these architectures, see Section 4 and the Appendix.
Concatenation Figure 4 (left). Arguably the simplest way to construct a neural field is with an MLP applied to the input coordinates (either as x or 7(x)), and the simplest method to condition it by conentenating a latent vector to these coordinates (Park et al., 2021; Martin-Brualla et al., 2021). This input is commonly ro-coneatenated
one or more hidden layers using skip connections improve stability (Chen & Zhang, 2019). This approach has proven effective for many applications, but has undesirable characteristics such as requiring O(k(m k)) parameters for the linear layers, where is the latent dimension, and k is the hidden layer width. Critically for our purposes, when m becomes larger than k, the hidden layers effectively perform a linear dimensionality reduction of the latent codes, limiting the network's 's ability to utilize high-dimensional latents effectively. This can be partially mitigated by partitioning the latents to be distributed among the hidden layers, but even this is limited by the layer count; see Section 4.3.
Hyper-networks Figure 4 (middle). One approach lo overcoming the inefficiency of concalenation in MLPs is hyper-networks (Ha et al., 2017: Sitzmann et al., 2019). In this approach, a secondary network (z) applicd to the latent code z regresses the weights & of the MLP fe, possibly allowing for a more efficient. incorporation of latent information. While the coordinate network itself is very efficient in this setup, it off-loads complexity
the neterr, which can become very large in order to predict entire layer weights output. How much of a problem this is depends on how frequently the hyper-network needs to be invoked. If only few instances (and thus latent codes) are seen per batch, the hyper-network cost will be small. In cases like Rebain et al. (2022) however, where thousands of latent codes per batch are required, this cost can become prohibitive.
A related approach to hyper-networks is feature-wise linear modulation (FiLM) (Perez et al., 2018; Mehta et al., 2021; Chan et al., 2021), which similarly incorporates a secondary network to operate on latent codes, but restricts the output of this network feature-wise modulation of activations in the MLP, rather than predicting all of its weights. This reduces the computational load of the hyper-network, but also reduces its expressiveness in terms of how it can affect the main MLP.
Attention Figure 4 (right). More recently, a class of neural field docoder architectures has emerged based on the sequence decoder mechanism used in transformers (Vaswani et al., 2017). Introduced by Jiang et al. (2021) and extended to higher-dimensional fields by Sajjadi et. al. (2022), this architecture defines

---

Published in Transactions on Machine Learning Research (11/2022)
neural field using cross-attention layers. These layers compute attention between query derived from the encoded sample position 7(x), and a set of lokens, which fill role equivalent to latent codes in the previously described architectures. This method of incorporating conditioning information from a set-latent representation (which subdivides the latent code into fixed-width tokens) provides unique benefits compared to other approaches, namely the ability to efficiently condition on very high--dimensional inpuls and the ability to vary this conditioning in a position-dependent way. This efficiency comes from the fact that a set of latent tokens can encode information equivalent to the weights of linear layer predicted by a hyper-network, but are produced by network that is shared over the tokens rather than predicting the entire token set in single operation. To quantify this, the complexity of predicting set-latent is O(n²), compared to the O(n") required to predict weights for a linear layer with hyper-network'. Compared to other more efficient hyper-nelworkk-ikk architectures like FiLM, this also allows the flow of conditioning information into the coordinate network to vary over space, as the attention is derived from the encoded position. While it might seem at first glance that such sct-latent prediction not necessary for networks like auto-decoders, we have observed that the inclusion of a token-level mapping network can prevent training from collapsing when conditioning with learned latent table.
A number of recent works (Müller et al., 2022; Chan et al., 2022; Liu et al., 2020; Chen et al., 2022) construct neural field style networks which condition on sample positions, but also incorporate a spatial grid, where each cell is associated with latent. vector(s). Conceptually, this is similar to attention-based conditioning, as it also derives a sample-level feature from a larger set of vectors using weights derived from the sample position. Unlike the attention- based architectures we focus on, grid-based methods can perform spatial "look-up" nearly for free, significantly increasing their computational efficiency. Nonetheless, the application of these models to category-level models has been limited far, and they suffer from the curse of dimensionality in scaling to problems beyond 3 dimeusions. As such, consider grids to be a complementary approach, and focus our analysis on the other architectures previously mentioned.
4 Experiments
Given the architectures described in Section 3.3, we are interested in analysing how they scale to very high-dimensional latent codes. In Section 4.1, we verify our assumptions on image auto-encoding problem to investigate how changes in latent dimensionality and complexity of the data distribution affects performance. Then, in Section 4.2, we experiment on different application domains, each consisting of several datasets, to evaluate the latent scaling performance in practical settings. Finally, in Section 4.3, we test alternative concatenation schemes to what we choose for MLPs to demonstrate that we are performing a fair comparison against
4.1 Image auto- auto-encoding
We implement a simple image auto-cncoder which compressos images with a CNN to latent reprosentation of some dimension, using a neural field decoder to reconstruct the images from this latent code by interpreting it as a single latent code (concatenation, leperr)etwork) or a set-latent (attention). In all of these experiments we change only two variables for each dataset: latent dimension and decoder architecture. This allows us to isolate the variance between decoder architectures in their ability to utilize the information from the encoder. We test on two datasets, respectively discussed in Section 4.1.1 and Section 4.1.2, with network implementation details detailed in Section A.3.
4.1.1 Tiled MNIST Figure 5 and Table 1
We design a dataset with controllable complexity to demonstrate how much the performance of an architecture can be affected by the size of the latent code and dimensionality of the data manifold. Loosely inspired by Lin et al. (2018), it consists of images formed by a 16x 16 grid of MNIST digits where each digit is down-scaled to 16x16 pixels for a total image resolution of 256x256. The digits are chosen randomly from the 60, 000 images of the MNIST datasct, creating up to 60, 1x unique possible combinations. This on-the-fly
Forr simplicity, we assume that all network widths, token sizes, and token counts are equal to
7

---

Published in Transactions on Machine Learning Research (11/2022)
Ground Truth
Concat (N
32)
Concat (N
512)
Attn (N 32) 4
Attn (N
512)
Y
4
H
4
4
4
4
4 4 4 4
4
4
4
4
Dight
4
4
4 4
4
4
4
4 4 4 / / / /
4 4
4
4 4 4 4 4
4 4 4
4
4
4
4 4
4
4
4 4 4
4 4
4
4 4 4 4 S
4
4
4
4
4
4
4
4
4
4
4 1 1 1
4 /
4 4 4 3 3 4 5 0
4
4
4
4
4
4
4
4
4
4
/
4
/
4
4
/
4
4
16 Digits
4
/
4
/
4
4
4
/
/
4
4
/
3
/
8
3
/
3
8 8
1 1 0 5 5
/
9
/
9
1
/
3
/ 6 /
S
3 7 4 6 R
/
3
/
9
8 9
4
9
0 5 5 2
6
4
O
4
9 6
6
4
0 5
6
256 Digits
4
6
/
4 5
4
7
1
4
6 1
4
3
7
f 7
7
Y
3
y
3 /
8
3
P
2
8
2
1
8
1
2
Figure 5: Tiled MNIST
Qualitative reconstruction results.
# unique digits
16
256
Latent dimension (N)
32
128
512
32
128
512
32
128
512
Concatenation
31.5
32.8
32.6
19.5
24.4
25.0
16.9
17.0
16.8
Hyper-network
30.8
33.3
33.7
19.3
24.4
25.4
18.8
21.7
21.8
Attention
35.0
36.3
33.9
26.1
31.3
31.4
23.5
24.3
24.6
Table Tiled MNIST We compare the reconstruction quality terms of PSNR three variants of our toy dataset. We find that the reconstruction quality drops significantly with data complexity, expected, and that the best performing configurations for the two complex distributions the attention-based model with the largest latent code. As expected, we also find that large latent dinensions are not helpful for very simple data.
image formation strategy ensures that memorization of the entire dataset is impossible. To test the effect of datasot complexity on network performance, we vary the number of unique digits in cach image: cither 1, 16, or 256, by setting consecutive 16x16, 4x4, or 1 1x1 blocks of digits to use a single repeated digit. We do this so that the images of each variant have the same size and visual properties, but vary significantly in the amount of information they contain.
A Analysis. In Table 1, we report average reconstruction PSNR, leading to the following key observations:
despite the similarity of the images, the reconstruction performance varies significantly between the variants as the number of unique digits increases, confirming that the more complex datasets are indeed more difficult to model;
increasing latent code size correlates with reconstruction quality for the more complex datasets, but not for the singlo-digit dataset, confirming that latent code size affects reconstruction by accounting for the dimensionality of the data manifold;
for concatenation-basee decoders on the 256-digit variant, the quality is very poor and does not improve with latent code size; this suggests that the concatenation-conditionn network does not have the capacity to model this complex, high-dimensional data manifold.

---

Published in Transactions on Machine Learning Research (11/2022)
Ground Truth
Concatenation
Hyper-network
Attention
Figure 6: Image auto-encoding Qualitative reconstruction results on CelebA-HQ for N = 8192.
Latent dimension (N)
512
2048
8192
Decoder architecture
Concat
Hyper
Attn
Concat
Hyper
Attn
Concat
Hyper
Attn
Ce
21.4
21.1
21.3
24.7
23.5
26.0
24.5
23.6
26.3
Decoder params (x109)
0.6
2.1
1.7
1.0
2.2
1.7
2.6
2.5
1.7
Table Image auto-encoding We repeat. aulo-encoding experiments on the CelebA-HQ dataset, and report the reconstruction quality (PSNR). We again find that the attention- network in combination with high-dimensional latent codes provides the best performance. This advantage can not be explained by network capacity alone, as we find that the atlention model performs the best for the largest codes, despite having the smallest number of parameters.
4.1.2 CelebA-HQ
In addition to the Tiled MNIST dataset, also experiment with the CelebA-HQ dataset introduced by Karras et al. (2018). In contrast to the previous dataset, these 30k real images consisting of high-resolution versions of CelebA (Liu et al., 2015) human face images. We resize these images to 256 256 resolution keep the CNN encoder the same as for Tiled MNIST, and to limit the computational cost of training. In Table 2, we observe that attention-based decoder achieves the highest reconstruction PSNR, whereas concatenation and hyper-network decoders encounter saturation in performance earlier.
4.2 Novel view synthesis
For the remaining experiments, focus the more challenging real-world task of novel view synthesis, one of the main application domains of neural fields. Given more images of an object scene, this is the task of generating images from novel viewpoints. We experiment with two different neural field-based approaches to novel view synthesis: neural radiance fields (Mildenhall et al., 2020), and light field networks (Sitzmann et al., 2021). Both are analyzed using the following datasets, where we randomly select 10% of views to be held-out from training and used for testing:
HUMBI (Yu et al., 2020): is large multiview dataset of 772 human subjects across variety of demographics, captured with 107 synchronized HD cameras. We down-scale center crops of 97 views to 256 for 308 full body subjects. We multiplied the RGB values against computed segmentation mask, such that the networks only reconstruct the subjects and not the background.
SRN Cars and Chairs (Sitzmann et al., 2019): We use the rendered datasel of cars and chairs from ShapeNet (Chang et al., 2015). The chair dataset consists of 3654 models with each instance being rendered from 200 unique viewpoints which we downscale 128x 128 resolution, while the cars datasct. consists of 2151 models, with each instance being rendered from 250 unique viewpoints which we upscale to 256x256 resolution.
9

---

Published in Transactions on Machine Learning Research (11/2022)
Ground Truth
Concatenation
Hyper-network
Attention
Figure 7: Neural Radiance Fields
Qualitative reconstruction results on SRN Chairs for N = 8192.
Latent dimension (N)
512
2048
8192
Decoder architecture
Concat
Hyper
Attn
Concat
Hyper
Attn
Concat
Hyper
Attn
HUMBI
29.3
30.2
31.0
27.9
30.6
31.0
28.3
29.9
31.0
SRN-Chairs
20.1
24.5
24.1
20.2
23.2
24.4
20.9
23.5
25.0
SRN-Cars
25.8
28.6
29.3
25.0
28.0
29.1
25.9
28.0
29.4
Decoder params (x109)
0.6
2.1
2.6
1.1
2.2
2.6
2.6
2.8
2.6
Table 3: Neural Radiance Fields - We compare the performance of different conditioning approaches, on the NeRF novel view synthesis task using the reconstruction PSNR metric on HUMBI, SRN-Chairs and SRN-Cars datasets. Attention conditioning with the largest latent size achieves the best result all cases.
Latent codes. To avoid the complexity of deriving the parameters 3D scene from images through an encoder network, we adopt the approach of Rebain et al. (2022) for our novel view synthesis experiments, and implement an auto-decoder with learnable latent table. The rows of the table, which each correspond to object in the dataset, are interpreted either single latent vectors, or set-latents with fixed token width of 128 and varying number of tokens (such that the overall dimension is unchanged), depending on the decoder archilecture.
Training. All novel view synthesis methods are supervised using the pixel-wise reconstruction loss in (2) applied the training images and rendered pixel values for training views. For all datascts and architectures, training batches consist of 64 instances, with 2 views per instance, and 64 pixels sampled per image.
4.2.1 Neural radiance fields
A neural radiance field network fo defines function from a point in space and viewing dircction d to radiance and optical densily: fe R + R1, x dx (). The radiance and densily values can be integrated along a ray using the approach described by Mildenhall et al. (2020) to produce rendered per-pixel color values. In our experiments, for simplicity, and because view-dependent effects are absent from two of our datasets and insignificant in the third, we omit the view direction inputs from the NeRF network. For volume rendering, we use the hierarchical sampling approach described by Mildenhall al. (2020), but without allocating separate "coarse" and "fine" networks instead sampling both coarse and fine values from the same network; specifically, we 64 fine/importance samples, and 128 coarse/ coarse/uniform uniform samples for each ray, which we found to be the minimal counts required to avoid noticeable artifacts with our data.
Analysis. We report the reconstruction metrics on our test images in Table 3, and Figure 7 compares images rendered using different NeRF architectures. As these numbers show, attention-base decoders consistently achieve the best results for large latent codes, well as the best performance overall.
10

---

Published in Transactions on Machine Learning Research (11/2022)
Ground Truth
Concatenation
Hyper-network
Attention
Figure 8: Light Fields
Qualitative reconstruction results on SRN Chairs for N = 8192.
Latent dimension (N)
512
2048
8192
Decoder architecture
Concat
Hyper
Attn
Concat
Hyper
Attn
Concat
Hyper
Attn
HUMBI
25.2
29.2
29.8
24.7
29.0
29.8
24.9
29.0
29.9
SRN-Chair
17.4
20.3
20.7
17.6
19.9
21.7
18.2
19.9
22.2
SRN-Car
23.8
26.8
27.5
23.8
26.3
28.1
24.2
26.3
28.4
Decoder params (x109)
0.6
2.2
2.6
1.1
2.3
2.6
2.7
2.9
2.6
Table 4: Light Fields - We compare the performance of different conditioning approaches, on the light field novel view synthesis task using the reconstruction PSNR metric on HUMBI, SRN-Chairs and SRN-Cars datasets. Only attention conditioning shows consistent improvement in PSNR on all datasets as latent size is increased.
4.2.2 Light field networks
Neural light fields define function from ray to radiance: IR$ + RR3 X fo(x) è. In this formulation, the camera parameters map pixel coordinates directly to rays, which the light field network maps directly to pixel color values. For our experiments we employ Plücker coordinates (Sitzmann et al., 2021), which assign 5D ray coordinates to points on a 4D manifold, such that co-linear rays with the same dircction map to the same point. This simplifies the mapping that the network needs to learn, though at the cost of requiring view rays to originate from a point outside the bounds of the scene.
Analysis. We report the reconstruction metrics on test images in Table 4, and qualitative results in Figure 8. In all cases, the transformer decoder achieves the highest reconstruction quality, and either increases with latent code size (SRN chairs and cars), or remains approximately constant (HUMBI) indicating that either the data manifold dimensionality has been reached, or that the manifold is to complex for the network model with the capacity it has. Hyper-networks remain approximately flai decrease slightly with increasing latent size, indicating poor ability to integrate large conditioning vectors. Concatenation provides the worst performance average, and show inconsistent behaviour with respect to latent sizc.
4.3 Ablation of concatenation methods
As our primary goal is to compare attention and concatcnation as conditioning methods, is important to verify that the approach we take to concatenating large latent codes into an MLP network is optimal. To this end, we perform an ablation study for different concatenation schemes using the largest latent code size of 8192 for one of main experiment setups: CelebA-HQ auto-encoding.
Variants
PSNR
8-way split
24.46
4-way split
23.27
2-way split
22.40
skip connection
22.12
no split or skip connection
20.53
11

---

Published in Transactions on Machine Learning Research (11/2022)
The first scheme we test is simply concatenating the entire latent code to the first layer input. Next, we try adding skip connection from the input of this first layer to a layer half-way through the network. For the rest, we try splitting the latent code evenly into 2, 4, or parts, and concatenating these sub-vectors to different layers spread evenly through the network. Note that the number of splits is limited by the depth of the network (8 layers all our experiments). The result of this study is shown in the inset table.
Splitting the latent code into 8 parts (as we do in main experiments), and concatenating them to individual layers provides the best results. This would suggest that the linear dimensionality reduction effect described in Section 3.3 is indeed the limiting factor for concatenation efficiency. As such, we split all of our latent codes for concatenation- based decoders to either 8 sub-codes with widths larger than the hidden layer width of 256, or to N sub-codes when the latent dimension N is less than 2048.
256
5 Conclusion
In this paper we have provided an analysis of performance for some common conditioning architectures for neural fields. We found a strong trend in favor of attention-baan architectures being the most effective option, particularly for cases where the conditioning signal is very high-dimensional. This result is valuable for making decisions about architecture design for methods which require high-dimensional conditioning to model complex data distributions. By sharing the results of these very computationally expensive experiments, we hope to reduce the cost burden for future work in this area.
Limitations (of our analysis). For architectures like eyper-networke which split computation into an instance-level and sample-level stages, there an inherent trade-off between the efficiency of the two stages. For applications where new instances are seen infrequently, can be advantagoous to save resourees by having more expensive instance-level computation and cheaper sample-level one. If the application requires many invocations of the instance-level network however, e.g. because training batches require many hundreds or thousands of instances as in Rebain et al. (2022); Sajjadi et al. (2022), such a configuration may be prohibitively inefficient. As detailed in Section A.1, we balance the typer-paramater choices in our networks such that each is comparable in efficiency when considering both the instance-level and sample-level computations. This provides an analysis that is widely applicable to different applications, but may overlook some application-specffe routes for optimization that take advantage of caching the results of the instanco-level network.
Broader impact. Our analysis is fundamental research having broad implications for a wide range of ML applications, and thus it is difficult, if not impossible, to anticipate all specific consequences of our research. We do however note a few potential areas. First, we note that conditioned neural fields could be used for generative methods, and thus come with all the consequent baggage of synthetic media generation. We feel these issues have been widely discussed in the community at large, with our analysis not adding anything new other than the palential for incrementally higher quality synthetic data.
Secondly, we note the large amount of energy required to run these experiments. We undertook this investigation as a service to the community so that others don't have to, and make our source code available for verifiability. Our results should allow others to make informed design decisions without needing to repeat. our experiments.
Acknow wennowledgeeents.. This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant, NSERC Collaborative Research and Development Grant, Google Research (Canada), Digital Research Alliance of Canada, and Advanced Rescarch Computing at the University of British Columbia. We thank Wei Jiang Weiwei Sun, Eric Hedlin, Daniel Watson, and David Fleet for their comments.
12

---

Published in Transactions on Machine Learning Research (11/2022)
References
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for imput of non-uniform density. In ICML, 2020.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv, 2017.
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generalive adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. Tn CVPR, 2022.
Angel Chang, Thomas A. Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, L. Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository. arXiv, 2015.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
Ian Goodfellow, Jean Poadet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In ICLR, 2017.
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In ICCV, 2021.
Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. COTR: Correspondence Transformer for Matching Across Images. In ICCV, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv, 2013.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. PacGAN: The power of two samples in generalive adversarial networks. In NeurIPS, 2018.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NewrIPS, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM TOG, 2019.
Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
Ishit Mchta, Michaäl Gharbi, Connelly Barnes, Eli Shochtman, Ravi Ramamoorthi, and Manmohan Chan- draker. Modulated periodic activations for generalizable local functional representations. In ICCV, 2021.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Thomas Miller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 2022.
Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric
rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, June 2019.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Bruall and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM TOG, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In TCML, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NewrIPS, 2007.
Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi. Lolnerf: Learn from one look. In CVPR, 2022.
Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin Tong, and Baining Guo. Global illumination with radiance regression functions. ACM TOG, 2013.
Mchdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucie, Daniel Duckworth, Alexey Dosovitskiy, Jakob Usækoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry- Novel View Synthesis Through Set-Latent Scene Representations. In CVPR, 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In NeurIPS, 2019.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrIPS, 2020.
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In NewrIPS, 2021.
Kenneth O. Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programménng and Evolvable Machines, 2007.
Mohammed Suhail, Carlos Esteves, Loonid Sigal, and Ameesh Makadia. Light field neural rendering. In CVPR, 2022.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NewrIPS, 2020.
A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, Z. Xu, T. Simon, M. Nießner, E. Tretschk, L. Liu, B. Mildenhall, P. Srinivasan, R. Pandey, S. Orts-Escolano, S. Fanello, M. Guo, G. Wetzstein, J.-Y. Zhu, C. Theobalt, M. Agrawala, D. B Goldman, and M. Zollhöfer. Advances in neural rendering. In ACM SIGGRAPH Courses, 2021.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Aaron van den Oord, Oriol Vinyals, and koray kavukeuoghu. Neural discrete representation learning. In NeurIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Milia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Comput. Graph. Forum, 2022.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In NewrIPS, 2020.
Zhixuan Yu, Jae Shin Yoon, In Kyu Lee, Prashanth Venkatesh, Jaesik Park, Jihun Yu, and Hyun Soo Park. Humbi: A large multiview dataset of human body expressions. Tn CVPR, 2020.
15

---

Published in Transactions on Machine Learning Research (11/2022)
A Appendix
A.1 Balancing instance-level w/ sample-level computation
As previously noted, depending workload, a conditional network may require more compulation at the sample level (e.g. rendering all the pixels in a large image), or at the instance-level (e.g. batch sizes of hundreds or thousands of images). This can have a drastic effect on what architecture is more efficient - if most of the computation can be off-loaded from the sample level to the instance level, as in a large hyper-net then the training or inference process may be much more efficient. for cases where only few instances need to be considered once. We chose to balance our choice of hypee-parameters for the case where such efficiency gain is limited as is the case for methods like SRT and LOLNeRF. In practice for our problem domains, this means choosing batch sizes with large numbers of images with pixels sparsely sampled from each. The most noticeable effect of this choice is that we choose a network width for our hyper-networks of 64 neurons, as this keeps the memory and compute cost close to that of attention and concatenation for our training setup.
This results most of our decoders having similar parameter counts, as the total number of flops close to proportional to parameters when instance-level computations are not heavily re-used.
A.2 Neural Field decoder architecture details
The decoder takes input coordinates x as input along with encoded latents z. All networks use sinusoidal positional encoding to map raw coordinates to nigh-dimensional vectors before conditioning on them. For the concatenation and hyper-network decoders, the sct-latent output of the encoder is flattened a vector, while the set structure is retained for the cross-attention layers of the attention decoder. The network supervisod using the pixel-wise reconstruction loss of Eq. 2. The concatenation and hyper-network models both consist. of 8-layer MLPs in all cases, while the attention models use 5 attention stages with three dense layers after cach. With the exception of the hyper-network layers, all MLPs use network width of 256 neurons. All multi-head attention layers use 16 heads and 256 336demenniinal keys unless otherwise specified.
A.3 Image auto-encoding architecture details
We an SRT-style (Sajjadi et al., 2022) network to encode input image into patch-wise latent vectors; CNN feature extractor maps the image to M M D feature map, from which P patches are extracted and re-shaped to P x F latent tokens. These tokens are further processed with a single multi-head self-altention layer, resulting in P x G latent tokens z such that PG = N (the latent dimension). In all cases, the encoder architecture is identical for cach decoder architocture used.
The decoder takes pixel coordinates as input and directly outputs pixel color, In order to vary the latent dimension output from the encoder (equivalent here to the bottleneck width of the auto-encoder), the width and height of patches extracted from the last layer of the CNN before self-attention layers is varied such that the number of latent tokens varies while their individual dimensionality remains constant. We also configure the attention decoder for this application to use single-head attention (and correspondingly reduced the key dimension to 64), rather than multi-head as with the others, as we found it to provide slightly superior performance. This is likely due to the one-to-one nature of the mapping from output pixel coordinates to relevant information in the input features.
For training auto-encoders, we use batch size of 128 images with 512 pixels per image.
A.4 NeRF and light field auto-decoder architecture details
We directly store the latent code/set-latent values in a learnable latent table (initialized to zero), which is trained alongside the decoder network. For both hyper-network and attention models we include a learnable 64-dimensional embodding for each latent token which is initialized to a random value, sampled from a normal distribution we found this to be necessary, as the initially vero-valued codes would otherwise cause the training of the layers conditioned only on latents to collapse. This embedding, unlike the latent codes, is
16

---

Published in Transactions on Machine Learning Research (11/2022)
the same for each object in the dataset, so the non-zero initialization is not harmful to convergence. For concatenation, the latents are directly combined with previous layer activations, so we did not find additional embeddings to be necessary.
The NeRF decoder takes in 3D world-space sample positions as imputs, and outputs radiance and density values which are used predict color for each ray according to the volume rendering equation described by Mildenhall al. (2020). The light field decoder takes ray coordinates, mapped into Plücker coordinates, imput, and directly outputs pixel color for the ray.
For training auto-decoders, we use batch size of 128 images with 64 pixels per image.
A.5 Neural architecture details
In this section give additional details neural network architecture choices. All MLPs relu- activated, and use the original layer normalization strategy of the method each architecture is based on:
Concatenation: none (Rebain et al., 2022)
Hyper-networks: at cach layer (Sitzmann et al., 2019)
Attention: after skip connections (Sajjadi et al., 2022)
The network stage/ layer count hyper-parametent were chosen to provide as close as possible to equal parameter counts across all experiments for the largest codes without changing the architecture for cach experiment and while minimizing the number of syper-parameters that different from those used in previous works.
Finally, we give details specific to each conditioning method.
Concatenation. The concatenation decoder consists of 8-layer MLP f(x), which maps positional-encodeed coordinates to the task specific output. Conditioning is achieved by splitting the latent code to equally sized sub-codes and concatenating these to the input of each layer.
Hyper-network. The neterrk decoder consists of the hyper-network H(z), which maps latent codes to the weights of 8-layer MLP [(),, which maps positional-encoded coordinates to the task specific output. More specifically, the hyper-network first splits the latent code into 8 components, in the concatenationn-aased decoder, then each of these sub-codes is then mapped by 2-layer MLP to weight and bias tensor to form one of the layers in f(r).
Attention. The attention decoder is a function g(x) which maps positional-enccoded coordinates to the task specific output. g(x) consists of 5 network stages, each of which has cross-attention layer followed by 3-layer MLP, both of which are bridged by skip-connection as in Scene Representation Transformer (Sajjadi et al., 2022). The cross-attention layer uses the same standard multi-head formulation as SRT and maps latent tokens to key and value vectors, and the input network activations from previous stages to query vectors. 16 heads are used for NeRF and light fields, and one head is used for image auto-encoding (as this gave slightly better results).
17


Reviews:
[{"review_id": "C99GM9BN5C", "reviewer": "Reviewer_HshV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general I find the details given in the manuscript not sufficient to 1) understand properly what they did 2) how the experiments were conducted. This makes is not possible to evaluate the study, see below.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors conduct a large scale study on the architecture bias to condition neural fields. They compare concatenation, HyperNetworks and an Attention-based concatenation mechanism on different datasets and observe an advantage over most setups for the  Attention-based concatenation mechanism.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please provide a clear description of the architectures, their parameter count, their training time, hyperparameters used. Although you try to describe the different architecture in words, I am not sure what you do, until I see a proper mathematical description how exactly the 3 versions, especially the HyperNetwork and the Attention version, are constructed and what happens if you change e.g. the latent dimension. \n\nIn general, I am not convinced about the approach put forward in this paper. I would have liked a much more fine grained study + ablations on some moderate dataset, e.g. the MNIST one studied here, but where the architecture search is much more thorough. What roles do initialisation, regularisation (strength), learning rates, etc play. \nAlso architecture wise,  the HyperNetwork can also be used with sharing meaning that the HyperNetwork gets as input a \"layer token\" and a conditioning z token producing the weights of every single layer (see e.g. https://arxiv.org/abs/1906.00695)\n\nIn the ablation that you did, even more splits for the concatenation method then leads to even improved performance, better than the attention mechanism? \nAlso, please provide ablations for the attention mechanism. When does it fail, what makes it work? \nIt seems that often performance of methods decrease with more dimensionality, can you try different initialisation schemes to prevent that from happing? Or is this related to regularisation and one wants a lower dimensional latent space? Why does this not apply for the attention base mechanism?\n\nMore technical issues:\n\n1. \"If we wish to perfectly model some distribution of data, then we must use a latent dimension at least equal to that of the manifold on which that distribution lies.\" I would be very careful with these kind of statements. I agree that the latent dimension plays a role in practice to model a mapping from Z to some Image distribution but theoretically this is a much more delicate issue since one can densely sample from a high prob distribution through a mapping coming from a one-dim (e.g. Uniform) distribution, see e.g. https://arxiv.org/abs/2006.16664. \nSince this is one of the major claims of the paper, I would study these issues in detail.  \n\n2. \"For example, choosing ρ(·) = ∥ · ∥22 results in the codes Z being distributed as a Gaussian\". I think this is not true: You are not modeling the posterior or a distribution as far as I understand but a set of parameters for which you penalize large values. Why would these \"samples\" converge to one from a gaussian? \n", "broader_impact_concerns": "Does not apply I think. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q1n4ixDSgv", "reviewer": "Reviewer_8ztu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths ===\n\n- (S1): The authors pose a very concrete question, and do a thorough job at trying to answer it. They experiment across various datasets with differing complexities, ranging from toy to large scale. In the toy experiment with MNIST, the authors reduce the intrinsic dimensionality while retaining the image size by duplicating digits, which allows them to decouple these two factors; I found this idea quite clever.\n- (S2): The writing is excellent: it reads very clearly, they are no typos/errors, and overall feels quite polished. The basics of what neural fields are and how they are used are well-explained. I also liked the inclusion of Figure 2 to visualize the dependencies between the different works.\n\n=== Weaknesses ===\n\nI did not observe significant weaknesses, given the type of the work. While clearly there is no novelty presented in the paper, this is expected from a \"large scale evaluation\"-style work, and it doesn't make it any less useful. I have some clarification questions (rather minor) which I defer to the \"Questions\" section below.\n\n=== Questions ===\n- (Q1): In the attention conditioning, the network queries the set-latent with a query based on the location x. Is it fair to assume that each of the vectors in the set-latent will possibly end up corresponding to a part of the object that is a coherent slice of the space? In other words, does it make sense to view this as a relaxed/learned version of having a different latent vector per every voxel of some predefined grid?\n- (Q2): At the bottom of page 6, the authors talk about the complexity of predicting the set-latent and how that compares to the complexity of predicting the network weights using a hypernetwork. Is this discussion specific to the auto-encoder setup (as in the auto-decoder the latent would not have to be predicted)?\n\n=== Nitpicks ===\n- In \"Our analysis is fundamental research (...), and thus difficult, if not impossible, to anticipate all specific consequences of our research\", shouldn't there be a verb somewhere near \"difficult\" e.g. \"it is difficult\"?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the topic of conditioning neural fields by a latent vector, and aims to find the best way to infuse that information into the network. The authors consider several popular choices, and run extensive experiments, comparing them across various datasets and latent vector sizes. They conclude that attention over a set of latent subvectors works best, especially at scale. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I think the work is already in good shape, and I don't have significant changes to request.", "broader_impact_concerns": "I see no unaddressed ethical concerns, and the discussion the authors include in their paper is sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5J2XqTjKTk", "reviewer": "Reviewer_NJ11", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe observational study is very meaningful as the current neural field does not have a consensus on which design choice to use.\n\nExperiments are considered over various neural field applications, e.g., image, 3d radiance field, and light field\n\n------------\n\n**Weakness**\n\n\nThe writing of the methodology part is slightly confusing.\n- Except for the \"Concatenation\", the mathematical definition is not clear (as concatenation is somewhat trivial). \n- Can the author provide the explicit mathematical definition of \"Hypernetwork\" and \"Attention\"?\n\n\nWhile the finding is very interesting, it is slightly unclear why \"Attention\" is the best choice.\n- Can the author provide some intuition about the observation?\n\n\nWhile authors have roughly categorized the design choices (into three), there exist various methods that are known to be effective. Considering more design choices will definitely give more intuition to the readers. \n- Functa [1]: only adds a bias term on a globally shared network \n- CIPS [2]: similar to Pi-GAN\n- Modulated SIREN [3]\n\n\nMore extensive experiments across neural field architectures\n- For instance, does this observation holds for various neural field architectures such as SIREN [4], and FFN [5]?\n\n------------\n\n**Questions**\\\nHave the author tried such observational studies on different modalities such as audio (t) or video (x,y,t)\n\n------------\n\n**Reference**\\\n[1] Dupont et al., From data to functa: Your data point is a function and you can treat it like one, ICML 2022\\\n[2] Anokhin et al., Image Generators with Conditionally-Independent Pixel Synthesis, CVPR 2021\\\n[3] Mehta et al., Modulated Periodic Activations for Generalizable Local Functional Representations, ICCV 2021\\\n[4] Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020\\\n[5] Tancik et al., Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, NeurIPS 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the design choice of neural field conditioning. The author mainly considered 'concatenation', 'hyper-network', and 'attention' approaches in 2D, 3D, and 4D signals, where the backbone neural field is based on MLP- architecture with positional encoding. The authors empirically demonstrate that the 'attention' achieves the best performance across modalities and the scaling, i.e., enlarging the conditioning dimension, on the 'attention' strategy also shows consistent improvement.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Mathematical Definition**: Some explicit mathematical definitions of the design choice (in the main text) will be encouraged.\n\n**Intuition**: Providing some intuitions about the observation, i.e., \"Attention\" is better than others, will be great.\n\n**more verification** (optional): Since this paper is an observational study, more extensive verification will be encouraged to convince the claim. While this is optional, more extensive experiments in the following area will truly make the paper stronger.\n- (i) hyper-network design choices [1,2,3]\n- (ii) neural field architectures [4,5] \n- (iii) more modalities, e.g., audio, video ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the design choice of neural field conditioning, and considers 'concatenation', 'hyper-network', and 'attention' approaches in 2D, 3D, and 4D signals, where the backbone neural field is based on MLP architectures. The authors empirically demonstrate that the 'attention' achieves the best performance.

It received three reviews. After rebuttal, two reviewers recommended Accept, and one reviewer recommended Leaning Accept. On one hand, all the reviewers agree that the authors have posed a very concrete question, the empirical study is meaningful, and experiments are comprehensive. The paper is also well written. On the other hand, two reviewers mentioned that more clear definition of "HyperNet" and "Attention" is needed. Reviewer HshV suggested extra experiments to further strengthen the paper. In the paper revision, the authors have provided clarification on additional architecture details. 

Overall, all the reviewers are happy about the paper, and the Action Editor would like to recommend accept of the paper. 
Decision: Accept (poster)