Coordinate-Aware Modulation for Neural Fields
Authors: Joo Chan Lee
Daniel Rho
Seungtae Nam
Jong Hwan Ko
Eunbyung Park
Venue: ICLR 2024 spotlight (2024)
Keywords: Neural Fields
Neural Representation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Neural fields, mapping low-dimensional input coordinates to corresponding signals, have shown promising results in representing various signals. Numerous methodologies have been proposed, and techniques employing MLPs and grid representations have achieved substantial success. MLPs allow compact and high expressibility, yet often suffer from spectral bias and slow convergence speed. On the other hand, methods using grids are free from spectral bias and achieve fast training speed, however, at the expense of high spatial complexity. In this work, we propose a novel way for exploiting both MLPs and grid representations in neural fields. Unlike the prevalent methods that combine them sequentially (extract features from the grids first and feed them to the MLP), we inject spectral bias-free grid representations into the intermediate features in the MLP. More specifically, we suggest a Coordinate-Aware Modulation (CAM), which modulates the intermediate features using scale and shift parameters extracted from the grid representations. This can maintain the strengths of MLPs while mitigating any remaining potential biases, facilitating the rapid learning of high-frequency components. In addition, we empirically found that the feature normalizations, which have not been successful in neural filed literature, proved to be effective when applied in conjunction with the proposed CAM. Experimental results demonstrate that CAM enhances the performance of neural representation and improves learning stability across a range of signals. Especially in the novel view synthesis task, we achieved state-of-the-art performance with the least number of parameters and fast training speed for dynamic scenes and the best performance under 1MB memory for static scenes. CAM also outperforms the best-performing video compression methods using neural fields by a large margin. Our project page is available at https://maincold2.github.io/cam/.

Published as a conference paper at ICLR 2024
COORDINATE-AWARE MODULATION FOR NEURAL FIELDS
Joo Chan Lee', Daniel Rho?, Seungtae Nam', Jong Hwan Ko " Eunbyung Park Sungkyunkwan University, ²KT (mainceld2, stnamjef, jhko, (atckepaaekeeeee edu, daniel rēkt. com
ABSTRACT
Neural fields, mapping low- low-dimensional input coordinates to corresponding sig- nals, have shown promising results in representing various signals. Numerous methodologies have been proposed, and techniques employing MLPs and grid representations have achieved substantial success. MLPs allow compact and high expressibility, yet often suffer from spectral bias and slow convergence speed. On the other hand, methods using grids are free from spectral bias and achieve fast training speed, however, at the expense of high spatial complexity. In this work, we propose a novel way for exploiting both MLPs and grid representations in neu- ral fields. Unlike the prevalent methods that combine them sequentially (extract features from the grids first and feed them to the MLP), we inject spectral bias-free grid representations into the intermediate features in the MLP. More specifically, we suggest a Coordinate- Aware Modulation (CAM), which modulates the inter- mediate features using scale and shift parameters extracted from the grid represen- tations. This can maintain the strengths of ML.ss while mitigating any remaining potential biases, facilitating the rapid learning of high-frequency components. In addition, we empirically found that the feature normalizations, which have not been successful neural filed literature, proved be effective when applied in conjunction with the proposed CAM. Experimental results demonstrate that CAM enhances the performance of neural representation and improves learning stabil- ity across a range of signals. Especially in the novel view synthesis task, we achieved state-of-the-ard performance with the least number of parameters and fast training speed for dynamic scenes and the best performance under 1MB memory for static scenes. CAM also outperforms the best-performing video compression methods using neural fields by large margin. Our project page is available at https://maaincol22.iitoee wwth.li.i.iiiiiiioooooooao
INTRODUCTION
Neural fields (also known as coordinate- seaddnaae-bbsedd or implicit neural representations) have attracted great
attention (Xie et al. 2022) in representing various types of signals, such as image (Chen et al. 2222 lb:
Mehta et al.] 2021 video (Rho et al. 2022, Chen et al., 2022h). 3D shape (Tancik et al. 2020
Chabra et al. 2020 and novel view synthesis (Mildenhall et al. 2020; Barron et al.) 2021| 2022).
These methods typically use multi-layer pereeptron (MLP). mapping low-dimensional inputs (co- ordinates) to output quantities, as shown in Fig. (a). It has achieved very compact representation by representing signals with the dense connections of weights and biases the MLP architec- ture. However, notable drawback of MLPs is their inherent spectral bias (Rahaman et al. 2019), which leads them to learn towards lower- freeuffry or smoother patterns, often missing the finer and high-frequency details. Despite the recent progress, such frequency-based activation func- tions (Sitzmann et al.) (020) and positional encoding (Tancik et al. (202)), deeper MI. structures and extensive training duration are needed to achieve desirable performances for high-frequency signals (Mildenhall et al. 2020).
With fast training and inference time, the conventional grid-based representations (Fig. [-(b) have been recently repopularized in neural fields literature. They can represent
signals effectively (w/o MLPs or w/ small MLPs, hence no architectural bias), achieving promising recon- struction quality (Fridovich- Keil et al. 2022 Chan et al. 2022 Takikawa et al. 2022). However,
Corresponding authors.

---

Published as a conference paper at ICLR 2024
scale grid r X
shift grid B
X
x
(a) MLP
(b) Grid MLP
(c) CAM
Figure 1: Feature representations based the (a) MLP, (b) Grid MLP, (C) CAM. The dot in CAM means a Hadamard product.
the grid structures (lypically representing volume features with high resolution and large channels) cause a dramatic increase in memory footprints. Although many recent works have explored reduc- ing the memory usage through grid factorization (Chen et al., [022a] Fridovich- Keil et al.) (003),
hash encoding (Müller et al. 2022), or vector quantization (Takikawa al. 2022). constructing compact yet powerful grid representation remains a challenge.
A typical approach of leveraging both grids and MLPs is to combine them sequentially (Müller et al.) 2022 Yu al. 2021aa] Takikawa et al. 2022), extracting the feature from the grid representations first and feeding them to MLPs. MLPs in these approaches play a secondary role in representing signals, and the small-size MLPs are generally used to finalize or refine the features from the grids. Therefore, the grids represent most of the signals' contents, and higher resolutions of the grids are required to achieve better performance, resulting in significant memory requirements.
In this work, we propose a novel way of exploiting grid representations in neural fields. Based on MLP architectures,
suggest coordinate-aware modulation (CAM), which modulates interme- diate features of the neural networks using the grids (Fig. (c)) More specifically, CAM extracts scale and shift parameters from the grid representations given the input coordinates, then multiplies the extracted scale parameters to the intermediate features in MLPs and adds the shift parameters. Since CAM utilizes an interpolation scheme commonly used in recent grid representations, it can extract scale and shift parameters at any arbitrary location. The main idea behind the proposed CAM to inject spectral bias-free representations into the intermediate features in MLPs. will assist in mitigating any remaining potential biases in MLPs and help them quickly learn high-frequency components.
In addition, we found that feature normalization techniques (Toffe & Szegedy 2015 Ulyanov et al., 2016) proved to be effective when applied in conjunction with the proposed CAM. Normalizing intermediate features in neural fields has yet to show meaningful gains in the representation perfor- mance. However, without normalization techniques, training deep neural networks in general often requires careful learning rate schedules and other hyperparameter searches (Bjorck et al.) 2018). and we observed similar phenomena in training neural fields. Given the same network architecture and task (training Mip-NeRF), different learning rate schedules resulted in significant performance vari- ations (further discussed in App. C.2 We have demonstrated that CAM benefits from the feature normalizations, showing fast and stable convergence with superior performance.
We have extensively tested the proposed method on various tasks. The experimental results show that CAM improves the performance and robustness in training neural fields. First, we demonstrate the effectiveness of CAM in simple image fitting and generalization tasks, where CAM improved the baseline neural fields by safe margin. Second, we tested CAM on video representation, applying CAM to one of the best-performing frame-wise video representation methods, and the resulting method set a new state-of-the- compression performance among the methods using neural fields and frame-wis wine representations. We also tested CAM on novel view synthesis tasks. For static scenes, CAM has achieved state-of-the-art performance on real scenes (360 dataset) and also showed the best performance under a 1MB memory budget synthetic scenes (NeRF synthetic dataset). Finally, we also tested on dynamic scenes, and CAM outperformed the existing methods with the least number of parameters and fast training speed (D-NeRF dataset).
2 RELATED WORKS
Neural fields, or implicit neural representations, use neural networks to represent signals based on coordinates. Recent studies on neural fields have shown promising results in variety of vision
2

---

Published as a conference paper at ICLR 2024
tasks such as image representation (Sitzmann et al.
2020
Dupont al. 2021), video representa-
tion (Rho et al. 2022: Chen et al, 202 3D shape representation (Tancik et al. 2020 Chabra
et al. [2020] Park et al. 2019; Mescheder et al., 2019; Martel et al. 2021), novel view synthe-
sis (Mildenhall et al. 2020; Barron et al., 2021; Müller et al.] [022] Fridovich-Keil et al., 2022
Yu et al.] 2221aa] Chen et al. 2022a Yu et al. 202Tb), and novel view image generation (Schwarz
et al.) [0200] Chan et al. 2021 Gu et al., 2022; Deng et al. (2022). Neural networks (typically
using MLPs neural fields) tend to be learned towards low-frequency signals due to the spectral bias (Rahaman et al. 2019). Several studies have been conducted to mitigate this issue by proposing frequency encodings (Mildenhall et al. 2020 Tancik et al.) 2020; Barron al.) 2021) or periodic
activations (Sitzmann et al. 2020; Mehta et al.) 2021). Nevertheless, this challenge persists in the literature, demanding the use of complex MLPs and extensive training time to effectively represent
higu-frey signals (Mildenhall et al. 2020).
An emerging alternative to this MIP--eppendent paradigm is the use of an auxiliary data struc- ture. typically grids, incorporated with interpolation techniques. Such approach has notably re- duced training times without sacrificing the reconstruction quality (Fridovich-Ket) Keil et al., [022]
Chan et al. 2022 Takikawa et al. 2022). ) However, these grid frameworks, usually designed with high-resolution volumetric features, demand extensive memory consumption as shown in Fig. (b).
While numerous studies have made efforts to minimize memory usage via grid factorization ( Chen
iet al. 2022a Fridovich-Keil Keil et al., 2023), pruning (Fridovich-Keil et al. 2022] Rho et al.) 2023).
hashing (Miller et al. 2202), or vector quantization (Takikawa et al., 22022, the pursuit of memory- efficient grid representation remains an ongoing focus in the field of neural fields research.
Combination of an MLP and grid representation. The aforementioned grid-based methods gen- erally use a small MLP to obtain the final output from the grid feature. In other words, the grid structure and an MLP are sequentially deployed. Most recently, NFFB (Wu et al., (023) proposed combining two architectures in a different way. by designing each of multiple sets of MLPs and
grids to represent different frequency signals, similar to the concept of wavelets. Nonetheless, is worth noting that NFFB demands task-specific designs for individual models. In contrast, CAM is a suug-and-play solution that can be easily deployed without the need for any modifications to the original model configurations.
Modulation in neural fields. Feature modulation in neural networks has been a well-established concept. spanning across diverse domains including visual reasoning (Perez et al. 2018), image gen- eration (Ghiasi et al. 20177: Chen et al. 2019), denoising (Mohan et al. 2021 )b. and restoration (He) let al. 2019). They typically employ an additional network (or linear transform)
represent mod- ulation parameters, learning a well-conditional impact on the intermediate features of the base net- work. Neural fields literature follows the paradigm by representing modulation parameters with the function of noise vector (Pi-GAN (Chan et al.] 202I)). datapoints (COIN+ (Dupont et al., (2222), ) wise latent feature (ModSiren (Mehta et al. (00TT), or input coordinate (MFN, FINN (Fathony et al. 2021 Zhuang 2024).. In contrast to other methods that integrate periodic functions into their approach, both FINN and our proposed method utilize pordinate-dependeen
parameters to directly influence the intermediate features. However, while FINN acts as filter by using the same vector for all layers, our model represents different scale and shift (scalar) values in each layer. Furthermore, the utilization of grid representation for scale and shift parameters in our model avoids introducing any network architectural bias. This is distinct from all the aforementioned methods, which can induce architectural bias by incorporating separate linear layer following positional encoding.
3 METHOD
Coordinate-aware modulation. Existing literature has delved into the representation of signals presenting continuous yet diverse characteristics across arbitrary coordinates, mainly relying on ei- ther neural networks (typically MLPs) or grid structures. We rethink this paradigm and propose coordinate-aware modulation (CAM), which combines both architectures in parallel. CAM ex- ploits the implicit representation of neural networks, which ensures compactness regardless of high- dimensional coordinates, while representing high- frequeneyynny signals effectively using grids. More specifically, CAM modulates the intermediate features of the neural networks based on input coor- dinates, where the scalar factors for scale and shift modulation are parameterized by grids. CAM can retain compactness since the grids represent single-channel modulation parameters, different from
3

---

Published as a conference paper at ICLR 2024
HW
Normalize N Seatures
Normalize N' rays
Normalize NxC spatial features
Courd. (x.y) (Nx2)
Coord. (d. 8) (Nx2)
Coord. (t) (Nx1)
(a) CAM in image task
(b) CAM in NeRF
(c) CAM in video task
Figure 2: Visualization CAM on different domains.
the general use of grids that represent large-channel features with high resolution. Formally, taking a ID feature of an MLP as an example, the formulation is as follows,
Fn,c The (X; (X;
where F, F E RNXC are an intermediate feature tensor and the modulated output feature (N: hatch size, C: channel size), and n, c denote the batch and channel index of the feature, respectively, 7(:1), B) N + RN are the scale and shift function of input coordinates X E R NDb (D: input coordinate dimension), outputting scalar values from the single-channel grids I, given each coordinate, (: D), B,, (-: B) denote each scale and shift factor for batch n.
Coordinate priority for CAM. The grids are adopted to parameterize single-channel features (mod- ulation parameters), but they can face challenges with the curse of dimensionality, especially with high-dimensional input coordinates (e.g., 6-dimensional coordinates for dynamic NeRFs). To avoid the complex dimension of grids, we strategically prioritize which coordinates to use for representing modulation parameters into grids, among diverse coordinates of each task. Visual signals can have several dimensions, including space, viewing direction, and time. At the core, spatial components construct distinct scenes, where the viewing directions determine which aspect of the scene becomes visible, and the temporal coordinates represent dynamic movements in the scene. Among the view direction and time coordinates, we empirically found that considering temporal coordinates is more beneficial for CAM. This be interpreted that visible scene determined by spatiality and view direction, is the basis of effectively defining a time-varying scene. We establish this hierarchy of coordinates, prioritizing the highest-level components among the coordinates (denoted as x+) to be regarded for modulation (e.g., temporal coordinates (() for dynamic NeRFs and view direction coordinates (é.8) for NeRFs). Given that image and frame- name-wise video representations involve only spatial and time coordinates, respectively, we use the complete input coordinate by denoting it as X, the following sections.
Feature normalization. We standardize the intermediate feature F with its mean and variance before applying the modulation. Although general neural representation methods cannot take ad- vantage of feature normalization due to its regularizing property for fitting, we empirically found that normalization integrated with CAM facilitates and stabilizes model convergence. We hypothe- size that the enforcing diverse distribution of standardized features acts de-regularization, which stands for fitting signals. We compute the mean and variance along with as many dimensions as possible, excluding the batch dimension.
Although CAM serves as a universal method that can be applied to any neural fields for a wide variety of tasks, each task possesses its unique characteristies and intermediate feature shapes. In the following sections, we will provide a more in-depth explanation of the CAM approach for each specific task.
3.1 IMAGE
We can formulate neural field function of 2-dimensional coordinate that outputs the corre- sponding color in order to represent images (Sitzmann et al., 2020 Tancik et al. 2020). When a stack of 2-dimensional coordinates X IR N'2 pass through neural network, CAM normalizes and modulates a latent feature F¹ E NCC of each layer I, where is the channel (or feature) size (we will omit superseript for brevity). As images only have spatial coordinates, obtain
4

---

Published as a conference paper at ICLR 2024
the modulation parameters corresponding to these coordinates. More precisely, Fig. illustrates how CAM works in the task, and CAM can be formally written as follows:
Fn,c = In 7x;;1) Fu,. (l((F) (o +6
+ +))(;
1 (n C の Fa,e: (F C
(Fnec-Maa Pn - (F))
(3)
where F, E RNC are latent and modulated latent feature tensors, respectively. The mean and variance functions (.)) IR- + TE normalize features over every dimension except for the batch dimension. Similarly, the scale and shift functions y(: T,, 8(; B) TR, R output scalar values for each coordinate, and 7n (=<=: [:; (-; (:; B) denote each scale and shift factor for batch n. We can extract values from the grid representations for scale and shift parameters (I', B e Rd Xdy, dg and dy are the grid resolutions) by bilinearly interpolating values using neighboring input coordinates X.
3.2 NOVEL VIEW SYNTIEESIS
Neural radiance fields (NeRFs). A NeRF model uses an MLP architecture to model a function of volume coordinate (x, y, =) and view direction (ò, 0) that outputs RGB color and density d. To calculate the color of each pixel (camera ray), a NeRF samples S points along the ray and aggregates color and density values of the sampled points using the volume rendering equation (Mildenhall eet al. 2020). Since outputs of sampled points in a ray will be merged to get the color of a ray, we view a pack of points per ray as a single unit. It constructs and latent features E Nx Based on the proposed priority, CAM is applied for NeRFs according to the view directional coordinates of N ray units (()00) E N22 (Fig. (b)), formally defined as follows:
input coordinate tensor X E NS5x
Pr,n, = ((((000))) Fn, - pn(F))
+ (x(((())) B), vo2(F)+ +c
(4)
(F(()) = SC Σ Fr.ser (F) = SC
[(Pmne-Hu(EE) - (F)),
(5)
s,c
s,c
where (()) NXSXC R. denote mean and variance functions, and H j,, (F) and o (F)
represent the mean and variance for ray when F is given. As mentioned in Sec. 3 we normalize over all dimensions except for the batch size. 7(: I), ( B) R2 + R. are scale and shift functions, parameterized by two grid representations B Rdexds; do and de are resolutions of azimuth & and elevation 0 dimension, respectively. Similar to (F) and (F). the scalars 7n (x;;TT) and (X; B) denote the scale and shift value, respectively, for ray n.
Dynamic NeRFs build upon the static NeRFs concept by introducing the ability to model time- varying or dynamic scenes, representing 4D scenes that change over time (Pumarola et al. 2021). This is achieved by adding a time coordinate / to the input of the NeRFs. Therefore, the overall pro- cess for CAM follows as in Eq [q except that the modulation parameters are obtained corresponding to time coordinates (() E T N from two 1-dimensional grids r, B E Rdd (d, the resolution of the temporal dimension).
3.3
VIDEO
Videos can be represented as a function of temporal and spatial coordinates. However, this pixel- wise neural representation demands significant computational resources and time, limiting its prac- tical use (Chen et al. 2021a). To tackle the challenges associated with high computational costs and slow training/inference times, NeRV (Chen al. 202 and its variations (Li et al., 2022 Lee al.) 2023) adopted a frame-wist wise representation approach and use neural fields as function of only the temporal coordinate t. This not only accelerated training and inference time but also improved compression and representation performance (Chen et al. 202 These Trame-wise video repre- sentation models leverage convolutional layers to generate a video frame per temporal coordinate t. More precisely, an input coordinate tensor X E NNx1 associated with N temporal coordinates is
5

---

Published as a conference paper at ICLR 2024
Table 1: Performance evaluation for image regression and generalization measured in PSNR.
Table 2:
Effectiveness in the NeRF task. denotes the reported value in the original paper.
Regression
Generalization
Method
#Params
Natural
Text
Natural
Text
Method
#Params
Time
PSNR
I-NGP
237K
32.98
41.94
26.11
32.37
NerfAce
0.6M
38 m
31.55
K-planes
37M
38 m
32.36
FFN
263K
30.30
34.44
27.48
30.04
CAM
266K
(+1.91)
32.21
(+15,73)
50.17
(+0.71)
28.19
(+3.05)
33.09
CAM
3.7M
13M
51 m
m
32.60
32.18
supplied to the neural network to generate intermediate feature tensors F R RNXCXHXW where N, C, H and W denote the number of frames batch size, the number of channels, the feature's height and width, respectively. Then, we can define CAM as follows:
Fno,hh.- = (X;1(X;11) Fnch. (n.o(F) + (X;
(6)
(F) HWW Σ Faiehia (F)
HW E(Fn, (F))
where u(-), o²()): BBXxCxHW + NxC denote mean and variance functions. The reason for not normalizing over every dimension except the batch dimension is to keep the computational costs affordable (see App. B). Motivated by [[yanov et al. (2016). we exclude the channel dimension, and represent channel-wise modulation parameters by scale and shift functions 7(: ((; The grids for scales and shifts are denoted by I and B, where and B are of size Rd respectively. Here, d, represents the grid resolution in the time dimension, and C represents the channel size which is the same as the channel size in the feature tensor F. Fig. (c) illustrates how CAM works in frame-wise video representation neural fields.
Nx R Xx
4 EXPERIMENTS
We initially assessed the effectiveness of CAM in terms of mitigating spectral bias. Then, we evalu- ated our proposed method on various signal representation tasks, including image, video, 3D scene, and 3D video representations. Finally, delved into the reasons behind its superior performance, conducting comprehensive analyses. All baseline models were implemented under their original configurations, and CAM applied in plug-and-play manner. CAM includes feature normaliza- tion throughout the experiments, except for efficient NeRFs (e.g., NerfAce (Li et al., 2023)), where found that the normalization is ineffective for pre-sampled inputs. We provide implementation details for each task in App.
Param 12.7K/ MSE 4.97
Param 14.7K/ MSE 3.34
Param 13.2K MSE 0.07
Param 15.2K/ MSE
(a) MLP
(b) MLP+PE
(c) MLP+CAM
(d) MLP+PE+CAM
Figure Performance on ID signal regression. The yellow dotted line represents GT.
4.1 MOTIVATING EXAMPLE
We begin by demonstrating the spectral bias-free representation of CAM evaluated on ID sinu- soidal function regression (Fig. B). The figure indicates that the MLP not capable of representing the high-frequeney signal, even though positional encoding (PE) is applied. In contrast, when CAM is applied to the MLP, the resulting model successfully represents the signal, even with less param- eter overheads compared to PE. The model applying both PE and CAM shows the most accurate

---

Published as a conference paper at ICLR 2024
representation. These results demonstrate that CAM can be effective solution for resolving the spectral bias of the MLP while maintaining compacmess.
Train
GT
FFN
CAM
Grid 8
GT
Mip-NeRE (óbit) CAM (obit)
200
Fey dh
Amgtaaimbs
obty
of
Train
CAM
Grid
FFNeRV
CAM
(b)
Figure 4: Qualitative results on (a) image generalization task with visualization of grids extracted from the last hidden layer and (b) novel view synthesis and video representations.
4.2 RESULTS ON VARIOUS TASKS
Image. Tab. [ shows the results of two subtasks, image regression and generalization. CAM improves performance of FFN both tasks across two image datasets, with negligible additional parameters. While I-NGP demonstrated impressive results in Natural image regression, highlight- ing its superiority in overfitting, it fell short in terms of image generalization. In contrast, CAM consistently shows high performance in both tasks, demonstrating its overall effectiveness. Fig. 4 (a) shows the quality results for image generalization. As the figure shows, both scale and shift grids (T, B) reflect the shape of the entire image, indicating the grids effectively represent the sig- nal. Especially for the text image, CAM allows distinguishing the text and background, resulting in significantly increased performance.
Table 3: Qualitative results evaluated on NeRFs. The sizes are measured in megabytes (MB).
Table 4: Performance evalua- tion the 360 dataset, which comprises unbounded real scenes. Among 9 scenes, evaluate publicly available scenes. CAM
NeRF Synthetic
NSVF Synthetic
L.LFF
Bit
Method
Size
PSNR
Size
PSNR
Size
PSNR
NeRF
5.00
31.01
5.00
30.81
5.00
26.50
applied on Mip-NeRF 360.
TensoRF
71.9
33.14
Rs 70
36.52
179.7
26.73
32
Mip-NeRF
2.34
33.09
2.34
35.83
2.34
26.86
Method
#Params
PSNR
+ CAM
2,34
33.42
2.34
36.56
2.34
27.17
Mip-NeRF I-NGP
0.6M
25.12
Rho al.
1.69
32.24
1.88
35.11
7.49
26.64
84M
27.06
TensoRF
16.9
32.78
17.8
36.11
44.7
26.66
Zip-NeRF
84M
29.82
Mip-NeRF 0.58
32.86
0.58
35.52
0.58
26.64
Mip-NeRF
360
9M
29.11
CAM
0.58
33.27
0.58
36.30
0.58
26.88
CAM
9M
29.98
Novel view synthesis on static scene. We first present the superiority of CAM over representations based on an MLP or grid with small MLP using the NeRF synthetic dataset. As the baseline models, we adopted NerfAcc (Li et and grid-based representations (Fig (b), respectively. We modulate the intermediate features of NerfAcc, utilizing modulation parameters represented by tri-plane factorized grids with singular proposed channel. For fair comparison with K- K-planes, here we refrained from implementing priority and used spatial coordinates to represent modulation parameters. As shown Tab. 2 CAM outperforms other baselines, resulting in the best visual quality with compactness and comparable training duration, validating its efficiency.
2023) and K-planes (Fridovich-Keil
al.] 2023) for MLP-
We also evaluated with more powerful baseline models, Mip-NeRF and Mip-NeRF 360. Tab. show the qualitative results for the NeRF synthetic, NSVF, LLFF, and real 360 datasets. Through- out all the datasets, CAM showcases significant improvement in PSNR, with negligible increase

---

Published as a conference paper at ICLR 2024
in the number of parameters. Especially for the 360 dataset, CAM achieves state-of-the-ar perfor- mance. We also tested on lower bit precision; we quantized every weight parameter including T and B. As Tab. 3] shows, CAM exhibits robustness to lower bit precision and remains effective. Furthermore, the CAM-applied 8-bit model consistently outperforms the 32-bit original Mip-NeRF. Consequently, CAM achieves state-of-the-art performance under a 1MB memory budget on NeRF synthetic dataset, as shown in Fig. 5 As the qualitative results using Lego (Fig. (b)) shows, the baseline performs poor reconstruction containing an incorrectly illuminated area while the CAM- applied model reconstructs accurately. This indicates that modulation according to view directions results in robustness for representing view view-dependent components.
33.5
Mip-NeRF-CAM (8bit)
Mip-NeRF (Bbit)
PSNR
Rho
TensoRF (8olt)
PSNR
FFNeRV+CAM
FFNARV-CAM (pruned)
34
FFNeRV
31.5
HNeRV
33
H.264
NeRV (pruned)
31
32
0.2
2
20
0
0.02
0.04
0.08
0.08
0.1
0.12
0.14 0.16
(MB, logscale)
Bits per pixels (BPP)
Figure 5: The rate-distortion curve evalu- ated on NeRF synthetic dataset.
Figure 6: The rate-distortion curve on UVG dataset (best viewed in color).
Dynamic scene. We used the D-NeRF dataset
(Pumarola et al., 2021) to evaluate CAM for novel view synthesis under dynamic scenes, as shown in Tab. 5
Table 5: Performance evalua- tion of dynamic NeRFs.
CAM is applied on NerfAce (Li et al., 2023) for T-NeRF (a variant of D-NeRF). CAM sets a new bench- mark, outperforming the previous state-of-the-ar by more than PSNR, even while using the least parameters. Furthermore, our model is time-efficient, needing only an hour for training, thanks to its foundation on Nerface that boasts rapid processing due to effi- cient sampling.
Method
#Params
PSNR
D-NeRF
1.IM
29.67
Vox
12M
32.67
K X-planes
37M
31.61
NerfAcc
0.6M
32.22
+ CAMM
0.6M
33.78
Video. In Fig. (b). the qualitative results for video representation highlight the enhanced visual quality achieved by CAM. We offer detailed results of video representation performance in App. C.I and here, we focus on showcasing video compression performance, a central and practical task for videos. Fig. T visualize the rate-distortion for video compression. In the range from low to high BPP, CAM improves compression performance compared to the baseline FFNeRV by significant margin. It achieves comparable performance with HM, the reference software of HEVC (Sullivan) et al. 2012). Distinct from HEVC, commercial codec designed under the consideration of time efficiency, HM shows significantly high performance under heavy computations. HM has a decoding rate of around 10 fps using CPU (Hu et al. (023). while our model is built on FFNeRV (Lee al. 2023). a neural representation capable of fast decoding, allowing for real-time processing with a GPU (around 45 fps at 0.1 BPP). To our knowledge, our compression performance is state-of-the- art among methods that have the capability for real-time decoding.
4.3 ANALYSIS AND ABLATION STUDIES ON CAM
Motivation. We analyzed the intermediate feature distribution in the image generalization task, where the features can be visualized straightforwardly as depicted in Fig. (a). CAM shows a high variance of pixel-wise features while improving the visual quality. This observation underscores the idea that the representation power can be boosted when the features of different coordinates become more distinct from each other. CAM is a strategic approach to achieve this, while it maintains compactness by representing only modulating scalar factors into grids.
Mitigating spectral bias. We visualized the error map in the frequency domain (Fig. (b)) to validate that CAM is actually capable of representing high-frequency components. CAM reduces the errors in high frequency noticeably with only negligible grid parameters (263K for the MLP vs. 3K for the grid in Tab. T). indicating its effective mitigation of the MI.P' spectral bias.
8

---

Published as a conference paper at ICLR 2024
(a)
0.57
0.32
Before CAM
8,
After CAM
Var 0.48
PSNR
Var 0.16
PSNR
Image
FFN
CAM
D.6
0.4
0.2
Supatightt
0.0
Erron(xte-3)
-CAM Figure 7: (a) Visualization of the pixel-wise distribution before and after applying CAM on the final feature. The same color indicates the same distribution (mean and variance). We provide variance between pixels of each feature (described in Ap). and output PSNR. (b) Error map in the frequency domain: A more centralized pixel of the maps indicates
Image
FFN
+CM
Image
FFN
in the lower frequency.
Coordinate priority. As shown in Tab. 6 CAM with highest-level coordinates based on proposed priority achieves the optimal performance. CAM with spatial coordinates is effective for modalities with only spatial coordinates (images), as have shown in Tab. [ However, when the input modality be- comes more complex in NeRFs and dynamic NeRFs, spatiality-aware modulation can be meaningless in spite of the requirement of large additional memory (even with the factorized grids). Furthermore, al- though using both time and view direction coordinates increases performance compared to the baseline in D NeRF, a single prioritized component demonstrates the most efficient result.
Table 6: Ablation study the proposed pri- ority. S, D, and T denote space, direction, and time coordinates.
CAM coord.
Buseline
#Params
PSNR
S
0.6M
32.22
NerfAce (D-NeRF)
13.1M
32.57
0.6M
32.44
0.6M
32.49
0.6M
33.78
0.6M
33.09
Mip-NeRF
13.1M
32.70
0.6M
33.42
Effeet of feature normalization. As shown in Tab. 7 normalization with CAM consistently enhances the performance for diverse tasks, while naively applying normalization typically degrades performance. In ad- dition, CAM allows one of the known advantages of normalization, decreasing the magnitude of gradients and improving convergence speed (loffe & Szegedy 2015), further discussed App.
Table 7: Ablation study on the feature nor- malization, evaluated on Natural images, Ready video, and Lego scene. CAM-N in- dicates CAM without normalization.
Task
Base
BN
LN
IN
CAM-N
CAM
Image Video
30.3
23.6
30.8
30.9
32.2
31.6
22.1
31.5
31.9
32.3
NeRF
35.7
35.2
35.4
35.9
36,2
5 CONCLUSION
We have proposed a Coordinate- Aware Modulation (CAM), novel combination of neural networks and grid representations for neural fields. CAM modulates the intermediate features of neural net- works with scale and shift parameters, which are represented in the grids. This can exploit the strengths of the ML.P while mitigating any architectural biases, resulting the effective learning of high-frequency components. In addition, we empirically found that the feature normalizations, previously unsuccessful neural field literature, are notably effective when integrated with CAM. Extensive experiments have demonstrated that CAM improves the performance of neural repre- sentations and enhances learning stability across wide range data modalities and tasks. We renew state-of-the-ee aft performance across various tasks while maintaining compactness. We believe opens up new opportunities for designing and developing neural fields in many other valuable applications.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the Ministry of Science and ICT (MSIT) of Korea under the National Research Foundation (NRF) grants (RS-2023-00245344) and Institute of Information and Com- munication Technology Planning Evaluation (IIPP) grants 0-01821, IITP-2021- -00-0000),,), and by the Technology Innovation Program RSS-2023-00235718) funded by the Ministry of Trade, Industry & Energy (1415187474).
L1TP-2023-2020-
REFERENCES
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Parrin-Bruuala, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855-5864, 2021.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf' 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19697-19705, 2023.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. Advances in Neural Information Processing Systems, 2018
Rohan Chabra, Jan E Lenssen, Eddy 1gg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision, pp. 608-625, 2020.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5799-5809, 2021.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123-16133. 2022.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, 2022a.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. In Advances in Neural Information Processing Systems, pp. 21557-21568. 2021a.
Hao Chen, Matthew Gwilliam, Ser-Nam Lim, and Abhinav Shrivastava. Hnerv: A hybrid neural representation for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10270-10279, 2023.
Ting Chen, Mario Lucie, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adver- sarial networks. In International Conference on Learning Representations, 2019.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2021b.
Zeyuan Chen, Yinbo Chen. Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2047-2057, 2022b.
10

---

Published as a conference paper at ICLR 2024
Junwoo Cho, Seungtae Nam, Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Streamable neural fields. In European Conference on Computer Vision, pp. 692-612,20 2022.
Yu Deng, Jiaolong Yang. Jianfeng Xiang, and Xin Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10673-10683, 2022.
Emilien Dupont, Adam Goliñski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Com- pression with implicit neural representations. arXiv preprint arXiv:2103.03122 2021.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Arnaud Doucet. COIN++ Neural compression across modalities. Transactions on Machine Learning Research, 2022.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießnnr, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, 2022.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2021.
Sara Fridovich-Keit Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5501-5510, 2022.
Sara -ridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479-12488, 2023.
Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Ex- ploring the structure of a real-time, arbitrary neural artistic stylization network. arXiv preprint 1705066700 2017.
Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d aware generator for high-resolution image synthesis. In International Conference on Learning Repre- sentations, 2022.
Jingwen He, Chao Dong, and Yu Qiao. Modulating image restoration with continual levels via adaptive feature modification layers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Zhihao Hu, Dong Xu, Guo Lu, Wei Jiang, Wei Wang, and Shan Liu. Fve: An end-to-end framework towards deep video compression in feature space. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4569-458) 2023.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-4544 2015.
Joo Chan Lee, Daniel Rho, Jong Hwan Ko, and Funbyung Park. Ffnerv: Flow-guided frame- wise neural representations for videos. In Proceedings of the ACM International Conference on Multimedia, pp. 7859-7870, 2023.
Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa. Nerface: Efficient sampling accel- crates nerfs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 18537-18546, 2023.
Zizhang Li, Mengmeng Wang, Huaijin Pi, Kechun Xu, Jianbiao Mei, and Yong Liu. E-nerv: Expe- dite neural video representation with disentangled spatial-temporal context. In European Confer- ence on Computer Vision, pp. 267-284, 2022.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, pp. 15651-15663, 2020.

---

Published as a conference paper at ICLR 2024
Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wet- zstein. Acom: adaptive coordinate networks for neural scene representation. ACM Transactions Graphics, 40(4), 2021.
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 14214-14223, 2021.
Alexandre Mercat, Marko Viitanen, and Jaroo Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics, 38(4), 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European Conference Computer Vision, pp. 405-421, 2020.
Sreyas Mohan, Joshua Vincent, Ramon Manzorro, Peter Crozier, Carlos Fernandez-Cranda. and Eero Simoncelli. Adaptive denoising via gaintuning. Advances in neural information processing systems, 34:23727-23740. 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Transactions Graphics, 41(4), 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer, D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- on Machine Learning, pp. 5301-53100 2019.
Daniel Rho, Junwoo Cho, Jong Hwan Ko, and Eunbyung Park. Neural residual flow fields for efficient video representations. In Proceedings of the Asian Conference on Computer Vision, pp. 3447-3463, 2022.
Daniel Rho, Byeonghyeon Lee. Scungtae Nam, Joo Chan Lee, Jong Hwan Ko, and Eunbyung Park. Masked wavelet representation for compact neural radiance fields. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 20680-20690, 2023.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances Neural Information Processing Systems, pp. 20154- 20166, 2020.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, pp. 7462-7473, 2020.
12

---

Published as a conference paper at ICLR 2024
Gary J Sullivan,
Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions on circuits and systems for video technology, 22(12):1649-16688 2012.
Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Ja- cobson, and Sanja Fidler. Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Pro- cessing Systems, pp. 7537-7547, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 2016.
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/avc video coding standard. JEEE Transactions on circuits and systems for video technology, 13(7): 560-576, 2003.
Zhijie Wu, Yuhe Jin, and Kwang Moo Yi. Neural fourier filter bank. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14153-14163, 2023.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, volume 41, 641-676, 2022.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3752-5761,22 2021a.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 45887 2021b.
Yixin Zhuang. simple and effective filtering scheme for improving neural fields. Computational Visual Media, 2024.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A IMPLEMENTATION DETAILS
In this section, we provide brief explanation of the functional form of grids and specify our imple- mentation details for diverse tasks.
A.1 FUNCTION OF AND B
I' and are grid structures represent the scale and shift factors, where each grid is trained function of coordinates with infinite resolution, outputting coordinate-correst corresponding components. The output in infinite resolution is aggregated by nearby features in the grid based on the distance between the coordinates of the input and neighboring features.
A.2 1D SIGNAL
We conducted the experiment for regressing ID periodic function, following the previous works (Ra-
haman et al. 2019 Cho et al. 2022). We constructed the target function f(x) = Σi= sin (2mkk + 6s), where The learning rate was set to 10-3 and we trained for 1500 iterations using the Adam optimizer. We used a 4-layer MLP with 64 channels as the baseline and also set the grid resolution of 64. When applying PE, we enlarged the single-channel coordinate to 32 channels, and we concatenated the original and enlarged inputs.
15, 10, .... 50), de (O, 2m) and uniformly sampled in the range of [0, 1].
A.3 IMAGE
For the 2D image representation task, we used Natural and Text image datasets (Tancik al. 2020), which include 512 512 images, respectively. The resolution of the grids (dg and dy) was set to 32 32. Using two subtasks, assessed the ability to regress the training data points and to generalize well on unseen data points. The first subtask to accurately represent target image at resolution of 512 x 512, using the same image for training, and it aims to measure the ability for fitting signals. Another subtask trains neural fields using a smaller image with a resolution of 256 x 256, but evaluates using the original image with resolution of 512 512.
We used FEN Tancik et al., 2020) as the baseline model, which was originally developed Jax a few years back. Due to its older environment, we opted for a Pytorch implementation simplify the experimental process. We constructed a baseline model following the original paper (MLP with layers, 256 hidden channels, ReLU activation, and sigmoid output). Each model was trained for 2000 iterations using the Adam optimizer. The learning rate was initially set to 10-3 and 10 for neural networks and grids, respectively, multiplied by 0.1 1000 and 1500 iterations. The manually tuned parameters for each dataset in FFN were also used in this experiment, where the gaussian scale factor was set to 10 and for Natural and Text, respectively.
For I-NGP (Müller et al. 2022), we used hash grids with 2-channel features across 16 different resolutions (16 to 256) and a following 2-layer 64-channel MLP. The maximum hash map size was set to 25
The variance Fig. a)) denotes the mean of the variance all pixels at same channel. For- mally, the variance e of H W pixel-wise C-channel features X CxCHH*M be expressed as, v a Ec (H, (H) (Xao)). where var(HH.W)()) HxW + IR computes the variance of H W values and Xeh "xx is features at the channel ch.
A.4 NOVEL VIEW SYNTHESIS
Static scene. We used synthetic (NeRF (Mildenhall et al. 2020 NSVF (Liu et al.] 2020
forward- -acing (LLFF MMidenhall et al., 2019)), and real-world unbounded (360 (Barron et al. ) datasets for evaluating novel view synthesis performance. As baseline model, we used Mip- NeRF (Barron et al.) (02T)) for single-scale scenes, except for 360 dataset, where we used Mip-NeRF 360 (Barron al. 2022) We implemented CAM based Mip-NeRF and Mip-NeRF 360 official
14

---

Published as a conference paper at ICLR 2024
Table 8: Compression performance evaluated on UVG videos at various levels. BPP denotes "bits
per pixel".
Video (#frames)
Beauty (600)
Bospho (600)
Honey (600)
Ready (600)
Jockey (600)
Shake (300)
Yacht (600)
Avg.
PSNR
33.65
34.59
38.89
33.89
27.1
33.43
28.76
32.86
BPP
0.0144
0.0149
0.0148
0.0142
0.0145
0.0115
0.0148
0.0144
PSNR
34.21
38.39
39.58
37.29
31.62
35.18
32.53
35.57
BPP
0.0454
0.0459
0.0442
0.0439
0.0448
0.0355
0.0455
0.0442
PSNR
34.51
39.87
39.71
38.32
33.64
36.65
34.35
36.73
BPP
0.0752
0.0751
0.0728
0.0721
0.0735
0.0743
0.0748
0.0739
PSNR
34.78
40.91
39.86
38.92
35.23
37.24
35.84
37.56
BPP
0.1122
0.1109
0.1087
0.1068
0.1089
0.0980
0.1108
0.1088
PSNR 35.06
41.71
40.01
39.3
36.5
37.71
37.13
38.24
BPP
0.1563
0.1530
0.15114
0,1480
0.1508
0.1249
0.1535
0.1500
codes in the Jax framework. While following all the original configurations, incorporated CAM into every MLP linear layer until the view direction coordinates were directly inputted. For the scale and shift grids (I, B), the values of dg and de were set to 4 and for forward-facine scenes, and 10 and 3 for other scenes, respectively. For quantization, we applied layer-wise min-max quantization- in Rho al. (2023). We compared our method with NeRF (Mildenhall et al. 2020). TensoRF (Chen et al. 2022a). Rho et al. (203) for NeRF synthetic, NSVF, and LLFF datasets in Tab. 3 and with I-NGP (uüller al. 2022) and Zip-NeRF (Barron et al. 2023) for 360
aware training (QAT),
dataset in Tab 4
Dynamic scene. We used the D-NeRF dataset (Pumarola et al., 2021 evaluate CAM for novel view synthesis under dynamic scenes. CAM was implemented on NerfAce (Li et al. 2023) with the grid resolution de of 10. NerfAcc for dynamic scene was originally based on T-NeRF (Pumarola et al. 2021), which deploys deformation network and canonical network. We incorporated CAM into every linear layer in the canonical network, until the view direction coordinates were directly inputted. We compared our approach with the baseline NerfAce and recent state-of-the algo- rithms for dynamic NeRF (D-NeRF (Pumarola et al. 2021). TiNeu (Fang et al. 2022). and K-planes. (Fridovich-Keil et al. 2023)).
A.5 VIDEO
Video representation. To measure the video representation performance of neural fields, used the UVG dataset (Mercat et al. (022)), which is one the most popular datasets in neural based videos with resolution of 1920 x 1080. Among video representing neural fields (Chen et al.) 2021a [ et al. 2022) Lee et al. 2023). we used FFNeRV (Lee et al. 2023) as our baseline model because of compactness and representation performance, We implemented CAM based on FFNeRV official codes in the Pytorch framework. To ensure consistency, we maintained all the original configurations including QAT, with the exception of applying CAM between the convolutional and activation layers of each FFNeRV convolution block. In regards the scale and shift grids (T, B), we set dr to 60 for both the 32-bit and 8-bit models, and 30 for the 6-bit model.
video representation. The UVG dataset contains
Compression comparison. For video compression results, we followed the compression pipeline used in FFNeRV, which includes QAT, optional weight pruning, and entropy coding. Although FFNeRV quantized to 8-bit width for model compression, we further lowered the bit width to 6- bit, except for the last head layer. This was done because CAM exhibits robust performance even with 6-bit, where the baseline FFNeRV shows poor performance, as shown in Tab. I0 Fig. 6 of the main paper depicts the rate-disten rate-distorion performance of our approach compared with widely- video codecs (H. (Wiegadd et al.) 2003). HM (HEVC (Sullivan et al. 2012) test model)), neural video representations (FFNeRV (Lee al. 2023). NeRV (Chen al. 2021a). HNeRV (Chen al.
15

---

Published as a conference paper at ICLR 2024
2023 ) Detailed compression performances of our model without pruning for each UVG video
various levels are reported in Tab.
Table Performance evaluation under different settings for Trame-wise video representation.
Norm Unit
B B shape
PSNR
Params (M)
Time/Epoch (sec)
(H, W)
R4xo
32,25
11.4
69.1
(H, W)
Rd
31.93
11.3
68.8
(C, H. W)
Rdxc
32.37
11.4
102.8
H,
Rd.
32.39
11.3
104.0
B ADAPTATION FOR 4D TENSOR
We generally proposed compute the mean and variance along with many dimensions as possible excluding the batch dimension, and represent scalar features in grids I, B. However, we introduce some adaptations for 4D intermediate tensors in frame-wise video representation: excluding also the channel dimension and representing channel-wise modulation factors in the grids. This is because of heavy computation from the large normalization unit, which causes dramatic increase in training time (about 50%), shown in Tab. 9 When we exclude channel axis, representing channel-wist modulation factors shows better than representing scalar factors. It worth noting that our general proposal achieves the best performance, highlighting the flexibility of CAM where we can trade performance and complexity.
C ADDITIONAL EXPERIMENTAL RESULTS
Table 10: PSNR video representation. The leftmost column denotes the bit precision of neural networks. BPP denotes "bits per pixel".
Method
Beauty
Bospho
Honey
Jockey
Ready
Shake
Yacht
Avg.
BPP
FENeRV
34.28
38.67
39.70
37.48
31.55
35.45
32.65
35.70
0.2870
32
34.29
38.86
39.69
37.82
32.25
35.47
33.03
35.95
CAM
0,2894
(+0.01)
(+0.19)
(-0.01)
(40.34)
(40.70)
(+0.02)
(+0.38)
(+0.25)
FFNeRV
34.21
38.41
39.60
37.29
31.48
35.26
32.48
35.55
0.0718
34.27
38.82
39.67
37.63
32.12
35.39
32.90
35,86
CAM
0.0723
(+0.06)
(+0.41)
(+0.07)
(+0.34)
(+0.64)
(+0.13)
(+0.42)
(+0.31)
FFNeRV
34.09
37.26
39.13
36.63
30.47
34.54
31.65
34.85
0.0538
34.21
38.25
39.21
37.16
31.57
35.02
32.50
35.45
CAM
0.0540
(+0.12)
(+0.99)
(+0.08)
(40.53)
(+1.10)
(+0.48)
(+0.85)
(+0.60)
C.I VIDEO REPRESENTATION
Tab. TO shows the video representation performance, measured in PSNR. The CAM-applied models consistently beat the baselines, regardless of videos. The performance gap is much wider for fast- moving videos (e.g., Ready and Jockey) than it is for static videos (e.g., Beauty and Honey). This result demonstrates that extended representational capacity in the temporal dimension due to grids surely improves performance representing time-varyinn varying information. addition, the performance gap between video representations with and without CAM widened as the bit precision decreased (from 0.25 to 0.60). These results imply that our method can useful for neural fields designed for storage-constrained situations.
C.2 EFFECT OF NORMALIZATION
In addition to the result in Tab. 7 we analyze the actual benefits of feature normalization in CAM. One of the known advantages of normalization is that it decreases the magnitude of gradients and
16

---

Published as a conference paper at ICLR 2024
(Mip-NeRFF
0.018
33
Mp-NoRF
0.016
Train PSNR
Gradient Norm
0.014
0.01
CAM (Mip-NeRF))
27
Mip-NeRF
0.008
2004 (10xxig)
0.008
500k LR)
0.004
0.002
200
400
600
800
1000
200
400
600
800
1000
Iterations (k)
Iterations
(a)
(b)
Figure 8: Analysis convergence using Lego scene. (a) Train PSNRs with different learning schedules, while quantization-aware trained to 8-bit. (b) Gradient norm of weights during training.
prevents them from diverging, which allows the use of higher learning rate and improved conver- gence speed (loffe & Szegedy 2015). As shown by the decreased level of gradients in Fig. [C(c), CAM benefits from the stabilizing advantage of normalization, achieving comparable and even su- perior performance to the baseline, with only 1/5 and half of the training duration, respectively (Fig. [ (a)). Without CAM, different learning rate schedules resulted in significant performance variations (a learning rate schedule over 1,000K iterations 500K iterations).
Inference speed and GPU memory requirement of CAM compared to MI.P-based and grid-based methods, using Mic scene.
Table 11:
Method
Test chunk
PSNR
Inf. FPS
Inf. Mem.
K-Planes
34.10
0.25
3.8 GB
Nerface
33.77
0.51
4.4 GB
1024
+CAM
36.03
0.26
4.7 GB
Nerface
33.77
1.19
10.5 GB
4096
+CAM
36.03
0.67
8.8 GB
Nerfacc
33.77
1.45
19.5 GB
8192
+CAM
36.03
1.01
16.4 GB
C.3 INFERENCE SPEED AND MEMORY
We report the inference speed and GPU memory requirements of the models in Tab. 2 evaluated on the 'Mic' scene. As shown in Tab. II K-Planes requires small memory while showing slow inference. CAM reduces the original NerfAcc's s speed when testing chunk size is small. How- ever, increasing the testing chunk size reduces the speed gap between using CAM and not using it. Intriguingly, CAM even lowers memory usage under these conditions. We interpret that CAM facilitates more effectively trained occupancy grid and helps bypass volume sampling, offsetting the additional computational demands introduced by CAM itself.
C.4 PER-SCENE RESULTS.
evaluated the performance on various datasets for novel view synthesis. provide per-scene results for NeRF synthetic (Tab. [2). NSVF synthetic(Tab, [3), and LLFF (Tab. 14] 360 (Tab. 15) and D-NeRF (Tab. 16) datasets.
17

---

Published as a conference paper at ICLR 2024
Table 12: Per-scene performance on the NeRF synthetic dataset measured in PSNR.
Bit
Method
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
Mip-NeRF
35.14
25.48
33.29
37.48
35.70
30.71
36.51
30.41
33.09
32
+ CAM
35.24
25.74
34.07
37.89
36.24
31.48
36.04
30.64
33.42
Mip-NeRF
34.68
25.48
33.20
37.28
35.29
30.52
36.18
30.28
32.86
8
CAM
34.98
25.80
33.77
37.77
35.95
31.48
35.96
30.47
33.27
Table 13: Per-scene performance on the NSVF dataset measured in PSNR.
Bit
Method
Bike
Lifestyle
Palace
Robot
Spaceship
Steamtrain
Toad
Wincholder
Avg.
Mip-NeRF
38.51
34.77
37.00
36.65
38.09
36.94
33.58
31.12
32
CAM
39.06
35.21
37.41
37.70
41.24
37.49
33.59
30.77
36.56
Mip-NeRF
38.20
34.46
36.85
36.46
38.00
36.77
32.84
30.55
35.52
+ CAM
38.88
34.92
37.15
37.52
40.94
37.43
33.13
30.45
36.30
Table 14: Per-scene performance on the LLFF dataset measured in PSNR.
Bit
Method
Fern
Flower
Fortress
Horns
Leaves
Orchids
Room
Trex
Avg.
Mip-NeRF
24.97
27.83
31.73
28.01
21.00
20.07
33.22
28.02
26.86
32
CAM
25.06
28.39
31.73
28.76
21.40
20.40
33.40
28.22
27.17
Mip-NeRF
24.95
27.56
31.27
27.66
20.88
20.07
32.97
27.73
26.64
CAM
25.06
27.72
31.45
28.18
21.27
20.37
33.13
27.88
26.88
Table 15: Per-scene performance on the 360 dataset measured in PSNR.
Method
Bicycle
Bonsai
Counter
Garden
Kitchen
Room
Stump
Avg.
Mip-NeRF 360
24.37 24.30
33.46
29.55 30.62
26.98 26.99
32.23 33.60
31.63
26.40
29.23
CAM
35.44
32.91
26.03
29.98
Table 16: Per-scene performance on the D-NeRF dataset measured in PSNR.
Method
Balls
Hell
Hook
Jacks
Lego
Mutant
Standup
Trex
Avg.
NerfAcc
39.49
25.58
31.86
32.73
24.32
35.55 36.29
35.90
32.33
32.22
+ CAM
41.52
27.86
33.20
33.89
25.09
37.57
34.81
33.78
18


Reviews:
[{"review_id": "mwBsN9SkoN", "reviewer": "Reviewer_oV38", "summary": "Authors propose a new architecture for neural fields, i.e mapping low-dimensional input co-ordinates to the signal values, called CAM (co-ordinate aware modulation). The main idea is to modulate intermediate features using scale and shift parameters which are inferred from the low-dimensional input co-ordinates. Authors show that, while regular a regular MLP shows heavy spectral bias, and just grid representation is computationally very expensive, CAM can mitigate the spectral bias learning high frequency components, while also being compact. In addition, CAM facilitates adding normalization layers which improves training stability. Authors empirically show that CAM achieves competitive results image representation, novel view synthesis, and video representation tasks, while being fast and very stable to train.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ Authors are tackling a very relevant problem, with wide interest to practitioners. \n+ Paper is well written and easy to follow. \n+ Claims in the paper are sounds. I particularly like that the argument about spectral bias and not learning high frequency components is verified empirically in Section 4.3 \n+ Experiments are sound and covers a wide range of tasks. Results are strong with performance comparable or exceeding the state of the art.", "weaknesses": "+ While the paper is generally strong, I believe that it lacks certain references which can put the work in a better context. There is a long history of using feature modulation in deep learning. A good example is [FiLM](https://arxiv.org/abs/1709.07871). This is also used for image generation/reconstruction tasks like in [generation](https://arxiv.org/abs/1810.01365), [denoising](https://arxiv.org/pdf/2107.12815.pdf), [image restoration](https://arxiv.org/abs/1904.08118) and [style transfer](https://arxiv.org/abs/1705.06830). Adding these references, and including a discussion around it can put feature modulation in a better context. \n\n+ Can authors include inference speed/inference memory requirements to put regular MLP methods, grid based method, and CAM in prospective? \n\n+ The choice of not using the all lower dimensional inputs to infer the scale and shift parameters, but a subset based on the problem is interesting. Have you conducted ablation studies that this is in some way beneficial? \n\nA bit tangential but:\n+ Do you think CAM can benefit decoder MLP for a triplane based representation as well? It would be cool to see some experiments and demonstrate the generality here. \n+ In addition, I think text to 3D is one another domain where the speed and training stability of CAM can benefit quite a lot. If authors can demonstrate a couple of results comparing the training stability and speed using CAM augmented MLP in DreamFusion, that would be a great additon to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FdD7DJnqNg", "reviewer": "Reviewer_kCkz", "summary": "The paper introduced a coordinate-aware modulation module that combines MLP features and grid representations for neural fields. Unlike the popular methods that chain the features, this new method not only preserves the strengths of MLP but also mitigates the bias problem by leveraging grid-based representation. The authors conducted experiments on tasks in multiple domains and the results demonstrate its capability of modeling high-frequency components and advantages over prevalent neural field features.", "questions": "- Could you answer the first question I posted in the \"Weaknesses\" section?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of the paper was clearly stated \n- The proposed approach is simple yet effective\n- The paper is well structured and the idea is easy to follow \n- Experiments are comprehensive. It covers various domains such as images, videos, etc.", "weaknesses": "- The numbers of the baseline models seem to be from the authors' own implementation, which makes it less appealing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oAsCHnrvEG", "reviewer": "Reviewer_7EJV", "summary": "This work proposes combining grid input and MLP structure at intermediate features level for Neural fields application. This naturally extends MLP-only or grid-only methods, which despite its competitive performance have had downsides, such as not being able to represent high-frequency content or being computationally intensive.", "questions": "Don't have specific questions", "limitations": "", "rating": 8, "confidence": 1, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The biggest strength of CAM is its simplicity and its plug-and-play nature. I believe this will have much far-reaching impact in the Neural fields literature, compared to other highly sophisticated & implementation-heavy frameworks designed to maximize PSNR value at all costs. Similar to how widely Batch/layer-normalization has been used by the entire field.\n- Extensive experimentation on diverse tasks and against different baselines add to the credibility of the work.", "weaknesses": "- While there are no specific weaknesses to point out, I don't think Figure 1 or Figure 2 convey the idea that well. Figure 1 probably will be better served by displaying more detailed mechanism (exammple of x, example of the values for \\Gamma, etc.).\n- Also giving a brief description of what functional form \\Gamma and B take would be informative for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KAkMJJxODP", "reviewer": "Reviewer_DCsj", "summary": "This main contribution of this paper is to use a grid-based approach to provide a scale and bias for the features generated at each layer of an implicit neural network, an approach used to encode any kind of signal. This is in contrast to the typical approach where the grid-based approach provides an input to the implicit neural network. In other words, the proposed approach is: a) use the input coordinate to recover the scale and bias for all layers of the MLP, b) recursively apply each layer of the network, normalize features, apply scale and bias for the appropriate layer. The authors also discuss which subset of input coordinates should be used to define the grid. Other contributions of the paper include the use of the aforementioned feature normalization in implicit neural representation and a comparison to various benchmarks on image encoding and generalization, novel view synthesis in static and dynamic NERF and video compression.", "questions": "* On page 8, the paper states that HM decodes at 10fps using a GPU. I was very surprised, because, as far as I know, HM does not use a GPU. I also could not find a mention of this fact in (Hu et al., 2023). While it is common to compare decoding speed on different hardware (CPU for traditional codec, GPU for ML methods), I think it is misleading to state that HM uses GPU. Could you please comment on/clarify this?\n* Could you please comment on the choice of coordinates used in the grid to define scale/bias?\n* Could you please provide further explanation about Figure 8?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n* As far as I know, use a grid-based approach for scale/bias in implicit neural representation is new.\n\nQuality:\n* Evaluation is performed on several different tasks, with good results, thus I believe the strength of the method is demonstrated.\n* The baselines used are generally competitive and recent.\n\nClarity:\n* The paper is well written.\n\nSignificance:\n* good results on multiple tasks, state-of-the-art in some.", "weaknesses": "Quality:\n* I find the use of FFN (Tancik, 2020) as the only baseline in the image task disappointing. While not the most significant experiment of the paper, I think the use of more recent baselines and in particular of other grid-based approaches, for example at least instant-NGP, would make the comparison on images more significant. I also want to add that I find the baselines in the video experiment to be adequate, and arguably the video experiment is more important.\n\nClarity:\n* I might have missed it, but for me, the paper does not sufficiently discuss/explain the reasoning behind the choice of coordinates that are used in the grid to select a scale and bias. For example, the scale/bias depends on the pixel coordinates only for a picture. As far as I understand, scale/bias are thus the same for each channel. But for the video, the scale/bias depend on pixel coordinates, time and channel. I do not understand why the channel becomes important for the video. I noticed the ablation study (Table 6), but it only covers the NERF experiments. This ablation study also does not discuss making the scale/bias depend on both direction and time.\n* I do not understand what is the variance represented in Figure 8. Is it the variance between the elements of the input of the last layer of the MLP? \n* I find the notation in equation 1 and subsequent equations a bit confusing. Both $\\gamma_n$ and $\\beta_n$ takes as input the full batch $X$. This suggests that any element of the vectors $\\gamma$ and $\\beta$ may depend on the full batch $X$. I suspect this is not the case due to how grid based approaches typically work but I am not sure. Would it make sense to change the notation to $\\gamma(X_n;\\Gamma)$ or  $\\gamma(X^{(n)};\\Gamma)$, to be closer to the notation already used in equation 4?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)