How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodings
Authors: Samuel Audia
Soheil Feizi
Matthias Zwicker
Dinesh Manocha
Venue: ICLR 2025 Poster (2025)
Keywords: neural tangent kernel
compute graphics
scientific computing
fourier feature encodings
multigrid parametric encodings
encodings
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We explore the exceptional performance of multigrid parametric encodings, commonly found in compute graphics, through the lens of the neural tangent kernel (NTK).

Abstract: Neural networks that map between low dimensional spaces are ubiquitous in
computer graphics and scientific computing; however, in their naive
implementation, they are unable to learn high frequency information. We present
a comprehensive analysis comparing the two most common techniques for mitigating
this spectral bias: Fourier feature encodings (FFE) and multigrid parametric
encodings (MPE). FFEs are seen as the standard for low dimensional mappings, but
MPEs often outperform them and learn representations with higher resolution and
finer detail. FFE's roots in the Fourier transform, make it susceptible to
aliasing if pushed too far, while MPEs, which use a learned grid structure, have
no such limitation. To understand the difference in performance, we use the
neural tangent kernel (NTK) to evaluate these encodings through the lens of an
analogous kernel regression. By finding a lower bound on the smallest eigenvalue
of the NTK, we prove that MPEs improve a network's performance through the
structure of their grid and not their learnable embedding. This mechanism is
fundamentally different from FFEs, which rely solely on their embedding space to
improve performance. Results are empirically validated on a 2D image regression
task using images taken from 100 synonym sets of ImageNet and 3D implicit
surface regression on objects from the Stanford graphics dataset. Using peak
signal-to-noise ratio (PSNR) and multiscale structural similarity (MS-SSIM) to
evaluate how well fine details are learned, we show that the MPE increases the
minimum eigenvalue by 8 orders of magnitude over the baseline and 2 orders of
magnitude over the FFE. The increase in spectrum corresponds to a 15 dB (PSNR) /
0.65 (MS-SSIM) increase over baseline and a 12 dB (PSNR) / 0.33 (MS-SSIM) increase over the
FFE.

Published as a conference paper at ICLR 2025
How LEARNABLE GRIDS RECOVER FINE DETAIL IN Low DIMENSIONS: A NEURAL TANGENT KERNEL ANALYSIS OF MULTIGRID PARAMETRIC ENCODINGS
Samuel Audia, Soheil Feizi, Matthias Zwicker Dinesh Manocha University of Maryland, College Park {a jaudia, sfizzi, zwicker, dmanocha] edu
ABSTRACT
Neural networks that map between low dimensional spaces are ubiquitous in com- puter graphics and scientific computing; however, in their naive implementation, they are unable to learn high frequency information. We present a comprehensive analysis comparing the two most common techniques for mitigating this spectral bias: Fourier feature encodings (FFE) and multigrid parametric encodings (MPE). FFEs seen as the standard for low dimensional mappings, but MPEs often out- perform them and learn representations with higher resolution and finer detail. FFE's roots in the Fourier transform, make it susceptible to aliasing if pushed too far, while MPEs, which use learned grid structure, have no such limitation. To understand the difference in performance, we use the neural tangent kernel (NTK) to evaluate these encodings through the lens of an analogous kernel regression. By finding lower bound on the smallest eigenvalue of the NTK, we prove that MPEs improve network's performance through the structure of their grid and not their learnable embedding. This mechanism is fundamentally different from FFEs, which rely solely their embedding space to improve performance. Re- sults are empirically validated on a 2D image regression task using images taken from 100 synonym sets of ImageNet and 3D implicit surface regression objects from the Stanford graphics dataset. Using peak signal-to-noise ratio (PSNR) and multiscale structural similarity (MS-SSIM) evaluate how well fine details are learned, we show that the MPE increases the minimum eigenvalue by 8 orders of magnitude over the baseline and orders of magnitude over the FFE. The in- crease in spectrum corresponds to 15 dB (PSNR) 0.65 (MS-SSIM) increase over baseline and a 12 dB (PSNR) 0.33 (MS-SSIM) increase over the FFF.
INTRODUCTION
Recent advancements in computer graphics. scientific machine learning, and the broader class of implicit neural representations (Essakine et al. 2024) rely the simple coordinate- multi- layer perception (MLP) network, which maps between low dimensional spaces (typically R IR,,,
or 1. Despite their diminutive size, coordinate-based MLPs have empowered inverse rendering
(Barron et al. [021 Mildenhall et al.) [2220] Müller et al. 2022, Pumaaola et al., 2020: Yu et al., 2020 Zhang et al. 2020), implicit surface regression (Wang al. 2021a), solving the rendering equation (Hadadan et al. 202T), and enabled Physics-Informed Neural Networks (Raissi et al. 2017) in the growing field of scientific machine learning. These networks need to have a small memory and computational footprint to maintain the tight timing required in computer graphics (Müller et al.) 202T) and to allow for computing higher order derivatives in scientific machine learning. Though cooddintte-base MLPs suffer from what known as the spectral bias problem (Basri et al. 2020) 2019, Wang et al.) 2020 in which they learn higher frequency details orders of magnitude slower than low frequency details. To understand the cause of this bias, machine learning practitioners turn to the neural tangent
simple MLP networks meet these requirements, in their naive implementation.
kernel (NTK).
Introduced by Jacot et al. 2018), the neural tangent kernel describes the training dynamics a neural network in its infinite width limit. Though using the NTK directly has been shown to produce

---

Published as a conference paper at ICLR 2025
worse results than its finite width counterpart (Li et al., 2020), empirical evaluation of the kernel for given data set and architecture allows practitioners to apply classical techniques, such as eigenvector decomposition, to better understand the underlying network. In fact, the spectrum of the NTK can be used to show a network's ability to learn high frequency information in finite width networks (Tancik al., 2020: Wang et al. 2020) Yang & Salman 2009). The eigenvectors for these finer details correspond to the lowest eigenvalues (Basri et al., 2019). so if network adaptations are able to raise the spectrum as a whole, the network is better able represent the underlying function. For coordinate based MLPs, this adaption is frequently an encoding that maps the input into higher dimensional latent space before being passed the network.
The two most popular encodings are parametric encodings (Hadadan et al.) 2021 Müller et al., 2022) and Fourier feature encodings (FFE) (Mildenhall et al., 2020; Tancik et al.) 2020; Vaswani et al. 2017: Wang et al., 2020). Parametric encodings use auxiliary data structures with learnable parameters to build higher dimensional embedding space. These data structures frequently take the form of grid, with samples being interpolated from fixed grid points and concatenated. Fourier feature encodings, in contrast, contain no learnable features and instead use a series of sines and cosines, similar to that of Fourier transform, to embed the input into high dimensional unit hypersphere. Its ease of implementation and amenability to analysis has made the Fourier feature encoding the dominant choice in scientific machine learning (Cuomo et al. 2022 Lu et al.) 2019] Raissi et al.) 2017); however, in computer graphics, parametric encodings have shown orders of magnitude better performance than Fourier feature encodings (Hadadan et al. 2021; Müller et al. 2022) at the cost of a larger memory footprint. Both encodings have been extensively evaluated, ignoring their NTK spectrum, on graphics problems such as 2D image regression, 3D shape regres- sion, inverse rendering, and radiosity calculations. See Tancik et al. (2020) for the FFE examples and Müller et al. (2022); Hadadan et al. (202T) for MPE examples.
The FFEs popularity has led to multiple investigations of its effect on the NTK spectrum (Tancik al. 2020, |Wang et al., 2020). No such analysis has been known previously for parametric encod- ings. We remedy this fact, by providing an in depth analysis of the NTK for parametric encodings. Specifically, we make the following contributions:
We derive the neural tangent kernel for the MPE. Through this derivation, prove that the multigrid encoding raises the eigenvalue spectrum of the neural tangent kernel as compared to the baseline coordinate based MLP by forming lower bound its eigenvalues. Our results provide the first theoretical justification for why MPEs are able to learn finer detail and discontinuities better than networks with no encoding.
We isolate the superior performance of MPEs to their learnable grid and not their embed- ding space by evaluating the NTK with and without the contributions from the grid. This is fundamentally different from FFEs, which rely solely on the embedding.
We empirically evaluate the NTK spectrum for the multigrid parametric encoding, Fourier feature encoding, and baseline identity encoding for 2D image regression 100 synonym sets in ImageNet (Deng et al. 2009) and 3D implicit surface regression Mescheder et al. (2018) three meshes from the Stanford graphics dataset (Turk & Levoy (1994) to val- idate our proof. Peak signal-to to-lltto-oiee ratio (PSNR) and multi-scale structural similarity index measure (MS-SSIM) Wang et al. (2003) measure how well the learned images learns fine details in the input. The MPE with the smallest grid cells was shown to increase the minimum eigenvalue 2 orders of magnitude over the highest frequency FFE and orders of magnitude over the baseline MLP. This corresponds to an increase of 12 dB (PSNR) / 0.33 (MS-SSIM) and 15 dB (PSNR) / 0.65 (MS-SSIM), respectively.
2 RELATED WORK
Encodings embed a low dimensional input into a higher dimensional space before passing it to the network. Much like the kernel trick for support vector machines (Schölkopf et al. 1999), the hope is that this new space is then easier for the network to act on. A key example of this fact is found in the transformer architecture (Vaswani et al.] 2017). Transformers augment their input with a positional encoding which gives the attention block information about input's 's location in larger sequence. The positional encoding updates inputs with a series of sines and cosines, providing an early example of the FFE. NeRF (Mildenhall et al. 2020 used the positional encoding inspiration
2

---

Published as a conference paper at ICLR 2025
to help coordinate based MLP solve the inverse rendering problem, in which scene is learned from images. Though NeRF has been iterated on, improved, and extended (Barron et al., [02T]] 2020). the FFE encoding has been mainstay. The two most Pumarola et al. 2020 Zhang et al.
common FHFs are the axis-aligned logarithmic, as explored in this paper, and the Gaussian FFE
(Rahimi & Recht 2008 Tancik et al. 2020), which introduces randomness into the encoding.
Multigrid encodings. in contrast, evolved from learned grid (Fridovich- Keil et al. 2021) and voxel (Chibane al. 2020) representations. Grid parameters, such as spherical harmonic coefficients, are learned through backpropagation and gradient descent. These representations, though intuitive, often resulted in large memory usage when capturing fine detail. So, the grid resolution was reduced, and intermediate results were passed to an MLP network to bridge the gap between the grid and finer details. This idea lead to the creation of the multigrid parametric encoding (Hadadan et al., 2021) and the hash grid encoding (Müller et al. 2022). The encodings are very similar, with the hash grid encoding being optimized for fast access on a graphies processing unit. Empirically, these encodings have outperformed FFEs in many graphics applications, such 2D image regression and implicit surface regression. Though parametric encodings have predominantly been applied to graphics applications, the scientific machine learning community has started to take notice and the hash grid encoding has been used to solve Burgers and the Navier-Stokes equations (Huang & Alkhalifah 2023).
The strong empirical performance of both types of encoding has sparked an interest in understanding how they work. Some authors considered empirical metrics such as gradient confusion and activa- tion regions (Lazzari & Liu 2023), while others have investigated the encodings through the lens of the neural tangent kernel (Tancik al. 2020). Though gradient confusion and activation regions provide interesting insight, activation regions are restricted to ReLU activation functions, and gra- dient confusion is heavily dependent on the stochastic sampling in training. Therefore, we focus our analysis on the NTK. In the infinite width limit, any common machine learning architecture can be represented by kernel regression problem using the NTK (Yang Salman 2019). Then, given a small enough learning rate, the training dynamics are equivalent to solving a simple ordinary differential equation (Jacot et al.] 2018). This representation shows that features along the largest eigenvalues will converge faster Basri et al. 2020; 2019 Jacot et al., 2018 Tancik et al. 2020) Wang et al.] 2020). Though this kernel does not always exist in closed form, we can empirically evaluate the finite width parallel using automatic differentiation (Novak et al. (022). Analogous to a first order Taylor expansion of the weights, the finite width kernel provides valuable NTK insights for the network sizes used in practice. Previous analysis of the NTK for coordinate based MLPs has been restricted to FFEs. We seek to include MPES in this extensive literature, and show that the eigenvalue spectrum can be used to explain MPE's improved performance over FFEs and the baseline MLP network.
BACKGROUND AND NOTATION
In this section, we briefly explain the necessary background to understand our key results. After an overview of our notation, describe the structure and mathematical formulation of the two most popular encodings. We then give a brief overview of the NTK and how it relates to the spectral bias problem by connecting the eigenvalues to the convergence on their corresponding eigenvector.
Notation. We consider data set, X = {x1, 12,,,.,.. } where x; E IRd 0 < d < 3, E NJ. Samples are typically drawn from Cartesian space or the pixel space of an image. The corresponding target Y = [y1. yyv] is similarly low dimensional. The coordinate based MLP is then given by the function fo(x) Rdd. 1.2, or parameterized by the learnable weights 0, where d, is the dimension of the embedding space. Matrices are denoted by bold capital letters, while vectors are bold lower case letters. Encodings are denoted by 7(x) : Rdd Rde, and the overall network is given by fe o.
A Axis-Aligned Logarithmic Fourier Feature Encoding, Axis-aligned logarithmic Fourier feature encodings are implemented by passing the original network input through series of sines and cosines, mimicking a Fourier transform. More frequencies are added by increasing a hyperparameter L. Making this learnable parameter has not been shown to work well in practice (Tancik et al.) 2020). L needs to be balanced with the frequency content of the function, or else the result will begin alias and decrease in quality TTncik al. 2020). Written out, the encoding appears as
3

---

Published as a conference paper at ICLR 2025
Y
X
Figure 1: The above figure shows example of the multigrid parametric encoding (MPE). The sample location (blue dot) is mapped to the surrounding grid cells (blue and green squares). The grid contains & learnable scalars at each intersection point. Bilinear interpolation is performed on these learnable parameters independently. All learnable parameters are then concatenated with the origin and coordinate before being passed to the network.
cos(2 TF(x)= = [sin(2"x) x), cos((2x)) sin(2' x), cos((2 (os(2'x)..... x).... sin(2"-11x), "
(I)
x E Rd and (x(x) E IR 2dL The logarithmic step helps the encoding shift to different frequencies due to the cyclic nature of sines and cosines. Slight variations this encoding exist, such as phase parameters and coefficients being drawn from a standard normal distribution (Tancik al. 2020
A Axis-Aligned Multigrid Encoding. Temporarily taking on the notation of Euclidean coordinates, and y, consider the 2D bilinear interpolation function defined by,
[g(a(() (n)
yi (n))
g(a(n) (n)) g(02"" (n) (m))
(n) y/2
(n) g(r 9) ArAy 22
(m) x1
y
x
x
(((2 (m)
y
y
Similar functions for linear and trilinear interpolation exist for ID and 3D applications. On an evenly spaced grid, Ar and Ay represent the width and height of grid cell. The coordinates 11 ./ [m)
(m) 11 3/2 .2 _v!"' and 222 .v(")) represent the four corners of the nh cell,
(n) [m) (n)
starting in the top left corner and moving clockwise. g(I, y) is then any arbitrary function that we interpolating.
We define a series of grids of decreasing resolution, either linearly or logarithmically. from which we interpolate the input each coordinate. Each point in the grid contains one or more learnable scalar parameters, typically initialized by (O, 0.01) (Hadadan et al. (021). The interpolation at each resolution is then concatenated (Müller al. 2022) along with the original input before being passed to the network. The encoding is shown graphically in Figure and is given mathematically by
(M, (0.00 (0.00(x) (x)
(0.k) @ g(
(x) @
(L.k) Bg
(x) Bx.
k is the number of the learnable, scalar values at given grid point, and typical value for & is in the single digits. Similarly, is the index of the grid. practice, only handful of layers are needed for good results. Similar encodings exist, such as the sparse multigrid encoding (Hadadan et al.) (021)) and the hash grid encoding (Müller et al. 2022). Both variations work similarly and decrease the memory requirements of the encoding without changing their theoretical properties.
Neural Tangent Kernel. Recent work has made great strides in the explainability of neural networks using what known the neural tangent kernel. Jacot et al. (2018) have shown that as the width of the network tends towards infinity and the learning rate of stochastic gradient descent goes to zero, the training can be described as kernel regression with the NTK. This result can be thought

---

Published as a conference paper at ICLR 2025
of intuitively the Taylor expansion about the optimal weights. The NTK then represents the first order derivative in the expansion. This kernel is given by
@S((x.) 05((x))
K KNTk(X;, Xj) EarN
00
80
The kernel is formed over all pairwise comparisons within the training data This produces a positive semi-definite Gram matrix. With suitable initialization and infinite width, this kernel becomes deterministic and remains constant during training (Jacot al., 2018). Assuming KNTK is invertible, the network's predictions on a new set of data at training step & is given by (Wolberg) 2006
fo(Xtest, t) KteacKTkk e C-KNKKYYYY.
(5)
Kteat is given by the neural tangent kernel the pairwise inner products of the training and test data set, and Y is the concatenated ground truth training data set described in the notation section. Though originally formulated the infinite width setting, the finite width evaluation the NTK has been reliable performance predictor of small and medium sized neural networks (Tancik al., 2020).
Spectral Bias. The NTK allows a more classical understanding of the network. KNT is positive semi-definite. Therefore, it can be decomposed into an orthogonal matrix and a diagonal matrix containing the eigenvalues À such that K NTK Q" AQ. Positive semi-definiteness means that all X's are greater than or equal to O. We compare the predictions of the training data by evaluating fx(X, t) Y. Multiplying the NTK and its inverse produces the identity matrix, giving
1x Y (I -KNKKItY -Y.
(6)
Using the fact that KNTK! Q" e -A. our training loss simplifies to
()(f((X))) -)== -QQ]
From Equation 7 we see that the loss has a dependence on the eigenvalues and eigenvectors of the NTK and that larger eigenvalues will decrease the training error faster along that dimension. Basri jet al. (2019) found that eigenvalues corresponding to higher frequency relationships in the dataset lower for coordinate-based MLPs. As result, networks will fit lower frequency, smooth, rela- tionships much faster than they will the higher frequencies. By comparing the eigenvalue spectrum produced by the composition of encodings and the NTK, we can evaluate encodings' bias towards higher frequency features, which corresponds to an increased number of large eigenvalues.
4 NEURAL TANGENT KERNELS FOR MULTIGRID PARAMETRIC ENCODINGS
We explore the structure of the neural tangent kernel for different encodings and prove that the MPE raises the eigenvalue spectrum the kernel over the baseline network. In special cases, the NTK can be written out in closed form for its infinite width limit. We, however, focus the finite width kernel to evaluate training. By extending previous work on the NTK of MLP networks (Jacot et al.) 2018 Yang & Salman 2019), we derive the kernel for MPEs and FFEs. We start by analyzing single layer MLP network, extend it to include encodings, and discuss how it is easily extended for MLPs with multiple layers.
Consider a single layer network without encoding and width n. Taking on the parameterization scheme in Jacot et al. (2018), scale the activation by the dimension and introduce scalar, 3, to scale the bias. 3 is typically set to O.I. Given weights, w() paxd and W R and biases, b E gnx1 we consider the function
fo(x) W(2) of (1)x+ x xb)
(8)
V
5

---

Published as a conference paper at ICLR 2025
Viewing 0 as the concatenation of all learnable parameters, such that 0 = w w(1) bj, take the derivative with respect to each parameter matrix. The i,j element of KNTK then given by the inner product between and xj in the data set. Therefore.
K(, NTK
= EA-N(
(x)(x,)
050(x,), = Ba-N (o((())) + +bbb,
(W((), +b)b))
00
00
0Φ
(W
(w X, + Bb), (
(w Xj + Bb))
+
X4
xj 00
00
8
(W
de
1x + Bb), W(2) dØ ( (w + [b)) (9)
0e
00
We now include the multigrid parametric encoding. As gradients are computed element wise across the training dataset, only need to consider single grid cell, layer, and learnable scalar time.
Using the notation from section 3.3, define Az = x2 x, Ar = I (*) A = 8/2 (n) y.
and Δy1 y 2ith We drop the superscript to consider single cell, with learnable weights W11, W12, W21, and W22- Writing out the matrices and multiplying through, we get
[Ar
Am]]
Win
w12
Aya Ay
w1a + W2 + u22 Azi. An1
W21
W/2
(10)
should be apparent that the gradient with respect to the weights in the grid cell is
Ar2 Vwā(z,)) = y) = ArAy Ar
1
Aayz
Ay]].
(1I)
The new function is the composition of fo with the encoding, "ra As before, we concatenate all learnable parameters and compute the hth 7th index of KNTK. The gradients of the grid parameters are independent due to the concatenation, so we first compute the contribution of a single grid parameter. The full kernel is then the sum of each grid cell contribution. Let ī(x;) be the output of the bilinear interpolation on a single grid cell, then the kernel element is
(x,))
05o()0(x))))
K (i.)) NTRMPE Ee-NN
00
00
1
Ee-
(0 W (x;) + Bb), di (W)) g(x,) + (b))
1 (w (xx,
Oo
Oó
((x))(x) + (b), ((x)
(W [(x;) + (b))
00
00
82
O6b
3b), 2 dg
(2)
(WI)g
W
(w))) + (b))
00
00
1 (w w())
00
dØ
(W g(x) 9b), w W/)gg (xx) (w()g((,)) + Bb))] (12)
n
00
00
The first three terms correspond to the original MLP with the input in the embedding space of the MPE instead of the original coordinate space. Let's denote this with k
M.P.P The last term is new term induced by the parameters in the grid, which we'll label K MP As layers along the L dimension and trainable parameters along the & dimension are independent, their kernel contribution simply Equation E2 repeated for each layer and parameter plus the kernel evaluated on the original coordinates. Expanding out across all elements x; E X, we get the following kernel for the MPE:
L
k
k Kla
KNTKAPE = KNTK +
L
MLP +
MPE
(13)

---

Published as a conference paper at ICLR 2025
The NTK induced by the composition of the MPE and MLP is the sum of the original NTK, Equation 9 without any encoding and the NTK produced by each layer and learnable parameter, Equation 12 Though we only discuss a single layer MLP to illustrate the structure of the NTK,
the extension to multiple layers is well established and can be built recursively from the above definitions (Jacot et al., 2018: Yang & Salman 2019). Adding layers to the MIP will simply add terms to Equation 2 and Equation [2] Equation [3] however, still holds for deeper MLPs. We use this fact prove that the eigenvalue spectrum of the composed kernel is greater than that of the base MLP.
Theorem 1, Given a dataset X, with n samples, the corresponding neural tangent kernel for a MLP network, and the neural kangent kernel for the same dataset and MLP composed with MPE. The 8th eigenvalue, sorted in descending order, of each kernel follows X < <XNLP + +Xp XMPE < X 4LP4 MPE, ; Vi E [L....,m.. X are the eigenvalues of the NTK for the network with no encoding. X MME is the smallest eigenvalue of the matrix produced by the encoding layers shown in Equation
13
X MLP4MPP are then the eigenvalues of the NTK for the encoded network.
Proof. Let KMLP by the neural tangent kernel for a MLP network evaluated on a training dataset X Let KMPE be the neural tangent kernel for the composed MLP and MPE evaluated on the same dataset. From Equation 13 we see that KMPE = MMLP + K" where K* is the ker- nel produced by the sum over all learnable parameters in the grid. By construction, each K square, symmetric Gram matrices as they are constructed by the inner product. Each K is, there- fore, positive semidefinite, making KAPE the sum of positive semidefinite matrices. Let (K) denote the jth eigenvalue of the matrix K. It follows from Weyl's inequality (Weyl 1912 that Vi [I,..., n2,, d MLLL S (LLP) + X (K+) S (KMPE).
We see that the MPE changes the kernel both through its embedding space and the learnable pa- rameters contained within the grid. By adding more layers in the grid or more Icarnable parameters at the grid nodes, we can increase the spectrum of the kernel at the cost of additional computation and memory usage. It is possible that the minimum eigenvalue of the additional matrices is zero; however, as will be demonstrated in the next section, this is far from true in practice.
For comparison, the NTK for the FFE can be computed as
0f((TF(x)) (f((TF(x)))
(i.f) K NTKEFE Ra-N(
00
00
Ea
W
yF (x;) + (bb,, oi (() (x,) + (b))
1 (w(2)
Ob ( (W())))(()) + (b))
(x,) + 3b), w( (x,)
7r
00
00
B2
Oo
(2)
W (x(z) + Bb) w
( yr()))(())) + [b))] (14)
n
00
Equation 14 shows that the FFE improves the NTK solely through its embedding space, and Theo- rem [ does not apply. The MPE, in contrast, has two mechanism to influence the kernel, but which dominant? To isolate the improvements in the MPE, we compute the spectrum both with and without the KMPE term (Figure 2). Without the contributions of the learnable grid, the MPE has little to no effect on the eigenvalues as compared baseline. We conclude that the MPF derives its performance from the learnable parameters and not the higher dimensional embedding space, while the FFE's performance rests solely on the embedding.
EXPERIMENTS AND RESULTS
We present an empirical analysis of the neural tangent kernel for multiple configuration of the FFE and MPE in the context of 2D image regression and a 3D implicit surface regression problem. Additional results can be found in Appendix A. The image regression problem provides a controlled setting that is casily interpretable. allowing to understand how the theoretical properties of the kernel translate to empirical results, considering convergence rates, prediction accuracy, and sen- sitivity to hyperparameters. 3D implicit surface regression demonstrates that the theory holds in
7

---

Published as a conference paper at ICLR 2025
NTK Spectrum on Subset of ImageNett
Nustirrd
(kn Grid)
10
Ligerrabee Rogittdd
201
20
20
10-1
1000 2000
3000
4000
5000
0000
7000
8000
Eipenssatue Indux
Figure 2: This plot isolates the improvements of the MPE to the learnable grid. The NTK is com- puted for a 2D image regression random images from 100 synonym sets in ImageNet (See the Experiments and Results section for more details). The solid line and dashed line are the spectrum at the end and the middle of training, respectively. MPE (No Grid) is the NTK of the MPE without the contributions of K MEE and is purely for theoretical analysis. Without the grid, the spectrum is barely above the baseline. With the grid, the spectrum is magnitudes higher.
higher dimensions as well, and provides a theoretical basis for previous results by Müller et al. 2022).
Experimental Setup. We evaluate the NTK in three settings. The first setting explores the re- lationship between the eigenvalue spectrum and the encoding hyperparameters on single image. The second setting extends our results to a larger class of images by evaluating the NTK on ran- domly sampled images from 100 synonym sets from ImageNet (Deng et al. (2009). The final setting demonstrates that the theory holds when learning a 3D implicit surface from mesh. The network size was held constant each domain. For image regression. 2 hidden layers with 512 neurons each were used, while the 3D surface regression used 8 hidden layers with 256 neurons each. The ReLU activation function was use in intermediate layers. For more details on 2D regression, please see Appendix A and see Appendix E for 3D implicit surface information. The baseline corresponds to the network with no encoding, while the FFE and MPE hyperparameters were set as follows. The scaling experiment hyperparameters (Table 3) were hand selected, while the ImageNet hyper- parameters (Table 2) and 3D implicit surface hyperparameters (Table were selected using Optuna (Akiba al. 2019).
Evaluating the NTK and Regression. We used automatic differentiation and the fast finite width NTK calculation (Novak al. (022) to evaluate the kemnel during training. The cigenvalues of the kernel were computed at the beginning, middle, and end of training. The eigenvalues are then sorted from high to low and plotted on logarithmic scale. To evaluate the regressed image, we report both peak signal-to-noij ratio (PSNR) in dB and multi-scale structural similarity index measure (MS- SSIM) Wang al. 2003). PSNR reports the max signal power as compared to the noise (average error); however, as PSNR is an MSE based metric, it may not capture fine detail, so we also include MS-SSIM which is more sensitive to the fine feature of the image. Higher scores in both metrics corresponds to higher quality outputs from the network as compared to ground truth.
Scaling Analysis. F shows the regressed image at the end of training. Qualitatively, we can see the spectral bias and how the encodings mitigate it. The baseline is only able to learn blurred image. The low frequency FFE greatly improves the results but still lacks sharpness. This fine detail is learned more easily by the higher frequency FFEs. Both MPEs casily Iearn high frequency details without greatly increasing the network's 's input size. This qualitative analysis is corroborated by the PSNR and MS-SSIM values in Table 3
Figure plots the corresponding eigenvalue spectrum for each of these regression problems shown in Figure The eigenvalue spectrum the middle and end of training are plotted as the dashed and solid lines, respectively. The baseline encoding has the worst performance, with the eigenvalues quickly dipping below le 6. The left most plot shows the comparison against all three FFEs. As
8

---

Published as a conference paper at ICLR 2025
ImageNet Fourier
Scaling Coarse
Metric
Multigrid 45.28
Baseline
Low
Mid
High
Fine
Baseline
PSNR
33.47
29.94
25
35
36.5
34.5
41.5
20
M-SSIM 1
0.79
0.39
0.32
0.20
0.23
0.40
0.37
0.73
0.08
Table 1: This table presents image quality metrics between the regressed image and ground truth across experiments. PSNR and MS-SSIM evaluate this similarity, with the latter being more sensi- tive to fine detail. As expected, we see that the MPE has the highest scores, while the FFE and MPE perform well over the baseline. See Figure 3 for a visual representation of these trends.
Base
(FFE) Low
(FFE) Mid
Truth
(FFE) High
(MPE) Coarse
(MPE) Fine
Figure 3: We compare performance of different encodings on image regression. We show the ground truth image (leftmost) along with a network with no encoding (top left), 3 configurations of the Fourier feature encoding (FFE), and two configurations of the multigrid parametric encoding (MPE) (see Table 3). No encoding produces a blurred image, but as we add encodings, the finer details start to be resolved. An increase in detail is seen in the FFEs, while the MPEs perform well with both the coarse and the fine grid.
expected, increasing frequency corresponds to an increase in the eigenvalues; however, we see that the FFE saturates, and there is little improvement between the mid and high frequencies. The middle figure plots the same comparison for the MPEs. Again, as theory predicts, the higher performing fine MPE has higher eigenvalue spectrum than the coarse grid. The right figure then compares the two MPES to the high frequency FFE. The fine grid has the highest spectrum overall, while the FFE and coarse MPE flip around the middle of the plot. This could explain why the PSNR and MS-SSIM values for the two encodings are so similar. The coarse MPE will quickly learn finer detail than the baseline, but the FFE will be able to learn the very fine details at the smallest eigenvalues. Appendix A shows the same results across encodings on additional images.
ImageNet Analysis. Plots of the average eigenvalue spectrum for the tuned encodings ImageNet are found in Figure 5a. Again, solid lines and dashed lines are the spectra at the end and middle of training. respectively. As images could have different numbers of pixels, the spectra were scaled to 8000 values. The mean was computed for each category. Tuning reflects the use of encodings in practice, and we find that the MPE shows clear benefits in the spectrum on a wide variety of images. The clear delineations of the spectra also demonstrates the robustness of this evaluation on wide class of images and strong alignment between the performance of the network and the spectrum of the NTK.
OccupancyNee Analysis. Lastly, we plot the average eigenvalue spectrum for evaluations across three 3D meshes from the Stanford graphics dataset Turk & Levoy] (1994) in Figure [5)] Again, we see a clear benefit of the MPE, with the spectra raising above the other encodings. The minimum eigenvalues of the FFE and MPE are similar, but the overall spectrum for the MPE is higher. This corresponds to better training across all eigenvectors. Both encodings show a clear improvement over the baseline, as expected. Visualization of the 3D surface are in Appendix E. These results show that the theory easily extends from 2D to 3D, and that the NTK spectrum is a powerful tool in understanding the performance of encodings across domains.

---

Published as a conference paper at ICLR 2025
Opersstues Taggett
Eiperaation Bingent
302
32"
33"
33"1
33"4
3371
337
30000 Indes 3500 580 13003
10300 350.0
Figure 4: The NTK eigenvalue spectrum is compared for the cases found in Figure and Table 3 All encodings perform better than the baseline. The left plot shows the comparison of the baseline and the FFEs. The middle plot shows the same for the baseline and the MPEs. The right blot then compares the high frequency FFE to both MPE. The FFEs seem to saturate, while the fine grid MPE gives the best performance. These trends are backed by the PSNR values reported in the table. Interestingly, the coarse MPE crosses over the FFE, giving it strong performance early in training but allowing for higher PSNR values in the FFES at the end of training.
NTC Spectrum on Subcet ImageNet
Average NTK Spactrum Occupancyhot
Openakee Magrituce
104
10
22
30
30
22'4
32"
1000
2032
3000
4000
8000
1500
0032
1000
2003
(a) ImageNet
(b) OccupancyNet
Figure 5: This figure compares the mean of the eigenvalue spectra of different encodings across ran- domly sampled images from 100 synonym sets in ImageNet and three 3D meshes on OccupancyNet. The dashed line shows the mean spectra at the midpoint of training. We find that tuned encodings have fairly regular performance: the MPE outperforms the FFE, which outperforms the baseline. This result shows that the spectrum is stable across different images and domains.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
We present an analysis of the multigrid parametric encoding compared to the Fourier feature encod- ing and baseline coordinate- MLP to understand the MPE's strong empirical performance in many graphics applications. We follow previous analyses on coordinate based MLPs for graphics and scientific machine learning and leverage neural tangent kernel theory to connect the eigenvalue spectrum with the learning of higher frequency information. We prove a lower bound on the eigen- values of the MPE and give strong empirical evidence for this improvement being isolated to the learnable grid as opposed the embedding space. The MPE, therefore, mitigates spectral bias in a fundamentally different way than the FFF. Our analysis. however, does not account for the influence of different activation functions, and the space of possible problems too large to compute the NTK for each application. Evaluations on a 2D image regression and 3D implicit surface problem demon- strated that the theory holds in multiple domains, while providing easily interpreted visual results. Through metrics such PSNR and MS- MS-SSIM, we showed strong alignment between the theory and empirical results. Our work greatly improves the community's 's understanding of encodings and could be leveraged in future works better optimize them to specific applications by adjusting the interpolation kernel or tuning other hyperparameters.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project was supported in part by a grant from an NSF CAREER AWARD 1942230, ONR YIP award N00014-221-22771, ARO's Early Career Program Award 310902-00001, Army Grant No. 1 the NSF award CCF2212458, NSF Award No. 2229885 (NSF Institute for Trustworthy AI Law and Society, TRAILS), MURI grant 14262683, an award from meta 3 314593-00001, and an award from Capital One.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2623-2633 7 2019. 1907.10902v1
doi: 10.1 10.1145/399250033330701 /ttps:///aaxx
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. Proceedings of the IEEE International Conference on Computer Vision, pp. 5835-5844, 2021. ISSN 15505499, doi: 10. M..10//CCC4892222000000000 abs 2oos2210.110 13415v3
URL https::/aaxiv..ot
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The Convergence Rate of Neu- ral Networks for Learned Functions of Different Frequencies. Advances in Neural Information Processing Systems, 32, 2019. ISSN 10495258. /RRp:tttps///ariv.oorr wwtp://wwwwgggga////////aaaa/
00425v3
Ronen Basri, Mcirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritch- man. Frequency Bias in Neural Networks for Input of Non-Uniform Density, 37th Interna- tional Conference on Machine Learning, ICML 2020, ParFF168147-1::626667, 2020. URL
https://arxiie org/ abs 2003. 04560v1
Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit Functions Feature Space for 3D Shape Reconstruction and Completion. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 6968-69799 3 2020. ISSN 10636919, doi: 10.1109/CVPR22600222200 202.00000000000000000 URL hwtppptptt/.x.....aaaaaaaa wwwp:xxwwv.og.//..oo abs 2003 01456v2
Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific Machine Learning through Physics-Informed Neural Net- works: Where we are and What's next. Journal of Scientific Computing, 92, 2022. ISSN doi: 10. 10.0007//10115-22--019999 05624v4
15737691.
URL httts httt:t:t//axii////e http:///awww..//////////000/
Jia Deng, Wei Dong, Richard Socher, Li Jia Li, Kai Li, and Li Fei-Fei. ImageNet: Large- Scale Hierarchical Image Database. 2009 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2009, pp. 248-255, 2009. doi: 10. (D09/CCVRR2209.5206222
Amer Essakine, Yanqi Cheng, Chun-Wun Cheng, Lipei Zhang, Zhongying Deng, Lei Zhu, Carola- Bibiane Schönlieb, and Angelica Aviles-Rivero. Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey. 11 2024. URL https:://arxiv. org/abs/ 2411 03688v1
Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022-June:5491 5500, 12 2021. ISSN 10636919. doi: 10.1 /0w9/CVVR52888.22222000
URL https:
Saeed Hadadan, Shuhong Chen, and Matthias Zwicker. Neural radiosity. ACM Transactions Graphics, 40(6), 12 2021. ISSN 15577368. doi: 10.1145737781333488599. URL https:
7/dl mell.ccm.com www..//.rrr/////////oooooo 11w.e//....11111111111

---

Published as a conference paper at ICLR 2025
Boris Hanin and David Rolnick. Deep ReLU Networks Have Surprisingly Few Activation Patterns. Advances in Neural Information Processing Systems, 32, 6 2019. ISSN 10495258. URL https:
//arxiv. org abs 1906. 00904v2
Xinquan Huang and Tariq Alkhalifah. Efficient physics-informed neural networks using hash encoding. Journal of Computational Physics, 501, 2 2023. doi: 10. 10.1016/1..p.2224, 112760. URL http:://arxiv.org/abbs.. 2302. 13397http://dxx doi /ttp:pppp///ddddooo.///111. or./10.
1016/;.. jep. 2024. 112760
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. Advances in Neural Information Processing Systems, 2018-
December:8571-8855 6 2018. ISSN 10495258. URL httpss//aaxvv.oo httpa////axxii.oo//aa/////////
07572v4
J Kiefer and J Wolfowitz. Stochastic Estimation of the Maximum of Regression Function. The Annals of Mathematical Statistics, 23(3):462-466, 3 1952. ISSN 00034851. URL http:// www. jstor. http://tbbeeee222239/
John Lazzari and Xiuwen Liu. Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics. 1 2023. /ttttp:://a.i.......ar/a// abs///ass//300 05816v4
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Learning Parametrree Two-Layer ReLU Neural Networks beyond NTK. Proceedings of Machine Learning Research, 125:2613-2682. 7 2020. ISSN 26403498. URL htth:httt//aaaatt
org/ abs 2007 04596v1
Lu Lu, Xuhui Meng. Zhiping Mao, and George E. Karniadakis. DeepXDE: deep learn- ing library for solving differential equations. SIAM Review, 63(1):208-228, 2019. doi: 10.1 N0.1137/19M1277067 oxg/10.1137/19M11728
/aarr//////////// 907.900..0000 300.004502tttp///kk doi.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2019-Junc:4455- 4465, 12 2018. ISSN 10636919. doi: 10.100//VPPR.2019.00555 URL https://arxiv. org/abs abs 1812.8111101111 03828v2
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Lec- ture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12346 LNCS:405-421, 2020. ISSN 16113349. doi:
(())))2 URL https:
org abs// 002.0003.8899940 08934v2
Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. Real-time neural radiance caching for path tracing. ACM Transactions on Graphics (TOG), 40)4), 7 2021. ISSN 15577368. doi: 10. 041144//4450226.459988 /ttp///////////aaaaaaaaaaao/oo//
3459812
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. ACM Transactions on Graphics (TOG), 102) 7 2022. ISSN 15577368. doi: 10. 00.1145/3528223.5555 3530011 URL https:///t..aaaaa acm...a.mmmmmmmmmmmmmm awt.am.m.o.ooo....oo N0.1145/352822 3530127
Roman Novak, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Fast Finite Width Neural Tangent Kerne. Proceedings of Machine Learning Research, 162: 10211010117044 2022. ISSN 26403498.
URL https: /attss//aaxx.v.orr/////////// 000.006.8872000 08720v1
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High- Righ-Perffrmance Deep Learning Library. Advances Neural Information Processing Systems, 32, 12 2019. ISSN 10495258. URL wttpt:ttt/.. a/t:::///aiiii or. http://wwwwwgga///////ooe/e 0003vl
703v1
12

---

Published as a conference paper at ICLR 2025
Albert Pumarola, Enric Corona, Gerard Pons- and Francese Moreno-Noguee D-NeRF: Neural Radiance Fields for Dynamic Scenes. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 10313-10322, 11 2020. ISSN 10636919, doi: abs/ 2011. 13961v
10. 1.11009CCVR444437.02 2021 0.1018. URL httph: /aatps://arxivvovtt
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems 20 Proceedings of the 2007 Conference, 2008. ISBN 160560352X.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physies Informed Deep Learning (Part I): Data-driven Solutions Nonlinear Partial Differential Equations. 2017. URL https:
waaxii..org/abs///111111 10561v1
Bernhard Schölkopf, Christopher C Burges, and Alexander J Smola (eds.). Advances in kernel methods: support vector learning. MIT Press, Cambridge, MA, USA, 1999. ISBN 0262194163.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Advances in Neu- ral Information Processing Systems, 2020-Decems 2020. ISSN 10495258. URL hetps:
// //axxivccrg/aaa /006 10739v1
Greg Turk and Marc Levoy. Zippered polygon meshes from Range images. Proceedings of the 21st Anmual Conference Computer Graphics and Interactive Techniques, SIGGRAPH 1994, pp. 311-318, 7 1994. doi: 10. 10.1145/99216.119244/UUPPLL
JEILE/331I-TUR..R URL https: acm. wtt.//t/////oooooooo/oo///oo...t 11tp://....111/9222111 192241
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, volume 2017-Decem, pp. 59999-609,, Neural information processing systems foundation, 6 2017. ISBN 7706003622v7 URL httLtttss///xxiii org./ba///aassese/
03762v7
Peng Shuai Wang, Yang Liu, Yu Qi Yang, and Xin Tong. Spline Positional Encoding for Learning 3D Implicit Signed Distance Fields. IJCAI International Joint Conference Artificial Intelligence. pp. 1091-1097, 2021a. ISSN 10450823. doi: 10.24963/ijeca.2221/155 URL https://
arxiv. arxiv.org/abe.
01553v2
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Computationall Physicx, 449, 2020. ISSN 10902716. doi: 10.
2012..11902
http://awwwww..ooaaa/o///aoo 20t..//000000.....00. 14527v1
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of Fourier feature net- works: From regression to solving multi-scale PDEs with physics-informed neural networks. Computer Methods Applied Mechanics and Engineering, 384:1 13338, 2021b. ISSN 0045-
7825. doi:
Zhou Wang, Eero P. Simoncelli, and Alan C. Bovik. Multi-scale structural similarity for image quality assessment. Conference Record of the Asilomar Conference Signals, Systems and Computers, 2:1 1-11-948222 2003. ISSN 10586393, doi: 10. 10.1 N..109/AASSSC.223312222222
Hermann Weyl. Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differen- tialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung). Mathematische Annalen, 71(4):41-447 12 1912. ISSN 00255831. doi: 10. 10.1007/BB0114584/MMETT https:://ine spr nnger wwwm/artiiee e wtt.ee/......1110. 1007 1468844
URL
John Wolberg. Kernel Regression. In Data Analysis Using the Method of Least Squares, pp. 203- 238. Springer- Verlag, 2 2006. doi: 00.100733-540-317200
Greg Yang and Hadi Salman. A Fine- Grained Spectral Perspective on Neural Networks. 2019. URLItttpe www.riiwvvvooooeeooo httpg/abs/199 10599v4
13

---

Published as a conference paper at ICLR 2025
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. PixelNeRF: Neural Radiance Fields from One or Few Images. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 4576 4585, 12 2020. ISSN 10636919. doi: 10.1 N0.11099CVP444437.200
.00455. URL htttss///axxiii http://////wvv.ooaa///////s wtw.b///11..002 02190v.
Kai Zhang, Gernot Riegler, Noah Snavely, Cornell Tech, and Vladlen Koltun. NeRl++: Analyzing and Improving Neural Radiance Fields. 10 2020. URL /tLLattpe///xxxxioo////////////p 0 07492v2
14

---

Published as a conference paper at ICLR 2025
A TRAINING DETAILS AND FURTHER EXPERIMENTS
A. TRAINING DETAILS
The PyTorch (Paszke et al. 2019) library was used implement all models and analyses. Runs were completed on internal grid infrastructure using a single NVIDIA RTX A5000 graphics card, 8 CPU cores, and 56 GB of system memory. The GPU handles training and computation of the neural langent kernel, while the CPU handles the eigenvalue decomposition, it was found to be faster. Each analysis takes approximately 30 minutes to complete.
2D image regression trains the network on single image. The pixel coordinates and red, green, and blue color values were normalized to between 0 and There was no split between training and test data because, with the complete dataset, the coordinate based MLP is not able to learn the high frequency information the image. Training was done using stochastic gradient descent Kiefer & Wolfowitz (1952) with mean squared error loss between the linear activation of the final layer and the ground truth pixel value.
To evaluate the NTK during training, slight variations made the network architecture. NTK initialization (Jacot et al. 2018) of the linear layers was used in which the weights and biases are sampled from (0, I) and a layer, 1, is given by a Vile W() + (b((),, where (1()) the size of the input vector, (0) to that layer. Though this parameterization has the same approximation power the standard MLP, the additional factor greatly reduces the gradients during backpropagation, requiring us to set the learning rate to 100 on some experiments.
A.2
DOG IMAGE
PSNR vs Epach
Loss vs Epoch
0.008
Bascline
Loss (95E)
Fourler Low
Fourter Hid
Fourier High
20
0.004
Grid
Beseliee
arid
Fourler
Fourner High
Coarse
Grid
0.000
Figure 6: We report the peak signal-to-noise ratio (PSNR) and mean squared error (MSE) for the image regression problem shown in Figure 3 We compare a baseline network with no encoding, Fourier feature encoding (FFE), and 2 multigrid parametric encoding (MPE). These results parallel what was seen in the regressed images. The MSE highest and PSNR lowest for the baseline encoding corresponding to the lowest quality image. As frequencies increase in the FFE, the MSE lower and the PSNR increase. The fine MPE outperforms even the highest frequency FFE by 5 dB PSNR, while the coarse MPE is par with the higher frequency FFEs. We also note that the MPE lowers the error at faster rate than the FFE, with loss dropping below 1e-3 125 epochs sooner.
Figure 6 compares the MSE loss and PSNR across encodings. As expected, the baseline network has the highest training error and the lowest PSNR. We find that these plots follow the qualitative results from Figure 3 The low frequency encoding halves MSE and increases the PSNR by 5 dB. The mid and high frequency encodings show similar results both the regressed image and MSE and PSNR. The higher frequency encodings halve the loss again and increase the PSNR by 10 dB as compared to the low frequency encoding. The coarse multigrid encoding results in similar MSE and PSNR compared to the mid and high frequency encodings; however, it trains much faster with the loss dropping below le-3 125 epochs sooner than the mid and high frequency encodings. The best performing encoding then the fine multigrid encoding with PSNR another 5 dB higher than the
15

---

Published as a conference paper at ICLR 2025
coarse MPE, mid PPE, and high PPE. Loss is also the lowest for the fine MPE. From these results, NTK theory would tell us that the kernel's ' eigenvalues would follow the same trend.
A.3 WINDMILL AND LAKE IMAGES
We include results on two other images for the 2D image regression problem across encodings. We show that the same results from the main paper hold across multiple data sets. We include both a natural lake scene and a combination man made and natural scene featuring a windmill and flowerbed. Both qualitative look at the regressed images and MSE and PSNR show the same trends. The base network unable learn fine details in the image. The three FFEs show improving results with increasing frequency. The fine grid encoding then has the best performance with the lowest training error, the largest PSNR, and the highest eigenvalue spectrum.
Base
(FFE) Low
(FFE) Mid
Truth
(FFE) High
(MPE) Coarse (MPE) Fine
Figure 7: Results for the 2D image problem on an image of a windmill in a natural setting. See Section 5 for details on training and the parameters of each encoding. With no encoding (top left), the regressed image is blurry with no fine, high frequency details. The low frequency (top middle) encoding learns slightly more detail, but still results in a blocky image. The mid frequency encoding (top right), begins to show strong agreement with the ground truth image. The high frequency (bottom left), coarse grid (bottom middle), and fine grid (bottom right) encodings show even stronger agreement with the ground truth image, to the point where it is difficult to tell the difference with the human eye.
Egenvsk en Nes Rere
vs Spach
Earch
Knerler
lavter oig
10 Horerrop
Figure 8: We plot the eigenvalue spectrum (left) at epoch 150 (dashed line) and epoch 300 (solid) line, the PSNR throughout training (middle), and the loss curves (right) to compare the performance of the encodings quantitatively the windmill image. Similarly to the main paper, find that the strongest performing encodings, highest PSNR and lowest training loss, has the highest magnitude eigenvalues spectrum. Lower eigenvalue spectra then correspond lower PSNR and higher loss throughout training-
16

---

Published as a conference paper at ICLR 2025
Base
(FFE) Low
(FFE) Mid
Truth
(FFE) High
(MPE) Coarse
(MPE) Fine
image of lake in natural setting. See Section 5 for details on training and the parameters of each encoding. With no encoding (top left), the regressed image is blurry with fine, high frequency details. The low frequency (top middle) encoding learns slightly more detail, but still results in a blocky image. The mid-frequency encoding (top right), begins to show strong agreement with the ground truth image. The high frequency (bottom left), coarse grid (bottom middle), and fine grid (bottom right) encodings show even stronger agreement with the ground truth image, the point where it is difficult tell the difference with the human eye.
Figure 9: Results for the 2D image problem
Eqerwakes hound linger fare!
Loyler sii
Orotise
Figure 10: We plot the eigenvalue spectrum (left) at epoch 150 (dashed line) and epoch 300 (solid) line, the PSNR throughout training (middle), and the loss curves (right) compare the performance of the encodings quantitatively on the lake image. Similarly to the main paper, we find that the strongest performing encodings, highest PSNR and lowest training loss, has the highest magnitude eigenvalues spectrum. Lower eigenvalue spectra then correspond to lower PSNR and higher loss throughout training.
B CHANGE OF NTK DURING TRAINING
When plotting the eigenvalue spectrum, only show results epoch 150 and epoch 300. In Figure I1 we show the results for epoch 0 and epoch 300. As shown, the spectrum at the start of training very different from the spectrum the end of training. We also note that eigenvalues across encodings all seem to have similar values. NTK theory says that the kernel is stable in the infinite width limit (Jacot et al. 2018), and a width of 512 should be large enough to see this effect. So why does that not hold in this case? Recent works by Wang et al. (2211) 2020) have shown that the eigenvalue spectrum does, in fact, change during training real world scenarios. The reason given that the initial initialization is too far from the optimal parameter values, so large changes in weights are still necessary for learning. This most likely exacerbated by the fact our inputs are sealed to the unit interval and not unit normal distribution. This breaks the lazy training assumption for the infinite width limit of the kernel; however, we are focused on the finite width kernel and the training dynamics given different encodings. Furthermore, the spectral bias analysis using the eigenvalues of the NTK still holds. This can be seen by considering the Taylor expansion about the optimal weights. Given slight perturbations, we expect the kernel to be elatively stable in this region, and its
17

---

Published as a conference paper at ICLR 2025
eigenvalues allow to conclude with eigenvectors will be fit faster during training. This is reflected the fact that the kernel spectrum is stable after the start of training, as shown in the main Figures.
Eigenvalues Neural Tangent Kernel
10a
10
Eigenvalue
20:4
Baseline
Fourler Mid High
10-12
Grid Coarse
5000
10000
15000
20000 25000
Eigenvalue Index
Figure 11: We plot the cigenvalues at the start of training (dotted linc) and at epoch 300 (solid line). We see that the spectrum very low and close between encodings at the and separates by the end of training. Though this appears to break the lazy training observation of the infinite width kernel (Jacot al. (2088). we are concerned with the finite width kernel and the spectral bias analysis still holds as long consider small regions about the weights at time. As show in Figure the kernel relatively constant the second half of training.
training, but raises
C VISUALIZING MULTIGRID PARAMETERS
Though we have shown that the multigrid encoding is able to raise the eigenvalue spectrum and expressiveness of coordinate based MLP, it is also interesting to look at what the parameters in the grid are learning. In Figure 12 we plot the parameters as a gray scale image for both the coarse and fine encodings. Each encoding contains two learnable scalars each grid point. The (th parameter is shown on the left while the 1x is plotted on the right. We see that the grid, in fact, begins to learn the image, while the backing MLP adjusts the color and helps interpolate. We also see small gaps in the fine encoding, most likely due to grid cells that were unused during training. This shows that it's easy to waste space, especially with uniform sampling. In these cases, the sparse and hash grid encodings could help reduce the memory footprint without changing the analyses found in the main paper.
If the multigrid encoding is learning the image, then does the MLP need to do less work? Put another way, can we reduce the size of the MI.P to save memory and computation. To investigate this question, regions encode which neurons the MLP cause the ReLU activation function take on value of 1. This produces binary encoding that can be used to define activation pattern, A, which is vector in {0, 1 #newrons Each unique activation pattern then defines an activation region by
plot the activation regions (Hanin & Rolnick] 2019) in Figure I3 Activation
R(A,0) 0) = far (ddi(x) (ReLU(((()))) = at, VI, vi...., k, Vay E A).
(15)
Plotting the activation region for each pixel then shows the different sets of activations present the network. The closer the image looks noise, the more activation patterns present. In Figure 13 we show both the activation regions across the unit square and the total number of activation regions present across the different encodings. We find that the network has similar activation for both the MPEs and higher frequency FFEs. As such, we believe that the network is still playing a large role in the architecture, and should be cautious about reducing its capacity.
18

---

Published as a conference paper at ICLR 2025
Fine
Coarse
k=0
k=1
k=0
k 1
This figure shows the learned scalars at grid points in the MPE as a gray scale image. This allows us get greater insight into what the encoding doing. We plot results for both the fine grid encoding (left) and the coarse grid encoding (right). Each encoding has single layer and two learnable parameters at each node. The 0'h parameter shown in the left column and the 1 parameter is in the right column. We find that the encoding is learning representations similar to the regressed image. We also note that the fine grid encoding has small white patches the middle most likely from where no pixel coordinate landed in the ground truth image.
Figure 12:
Base
(Fourier) Low
(Fourier) Mid
Act Courd
(Fourier) High
(Grid) Sparse
(Grid) Dense
Figure 13: In this figure plot the total number of activation regions found in the network across encodings as well as the activation region plotted per pixel of the regressed image. Note that the multigrid parameters are slightly different in this instance, with the sparse encoding having 8 layers with 2 parameters at each grid cell, while the dense encoding has 24 layers with 2 parameters at each grid cell. We find that even though the grid encoding seems to learning the image (Figure T2 the baking network still has increased activation.
D HYPERPARAMETERS
This section lists the hyperparameters used by encodings in each experiment. perparameters used in the ImageNet evaluation. These parameters were found using Optuna Akibal et al. (2019) to reflect real world training. The tuner was set to maximize the PSNR value between the network output and ground truth. Table 3 gives the hand selected parameters that were used to investigate how adjusting the parameters affects the output. We see that increasing the FFE's frequency improves the PSNR, but there are diminishing returns, as there is smaller difference
the hy-
19

---

Published as a conference paper at ICLR 2025
Hyperparameters Batch Size
Results
Encoding
Learning Rate
Epochs
Grid Parameters
PSNR
Multigrid 0.3932
100
k 3,
2,
x=
[96, 277]
45.28
Fourier 0.3865
100
92
L= 6
33.47
Baseline 0.2394
100
10
N/A
29.94
Table This table shows the results of hyperparameter tuning using the Optuna library. trials were conducted where the tuner was able to select values for learning rate, batch size, and encoding specific parameters. The tuner sought to maximize the peak signal-to-noise ratio (PSNR) between the regressed and true image. The multigrid parametric encoding (MPE) is able maintain large
lead in PSNR over the Fourier feature encoding (FFE) and baseline. These parameters were used in
larger sweep to characterize how the encodings behave across wide class of images.
Hyperparameters
Results PSNR
Encoding
Learning Rate
Epochs
Batch Size
Grid Parameters
Low (FFE)
100
300
32
L ==4
25
Mid (FFE)
100
300
32
8
35
High (FFE)
100
300
32
L = 16
36.5
Coarse (MPE)
100
300
32
== 2, L 1, 100
34.5
Fine (MPE)
100
300
32
k: k= L = 1, z 200
41.5
Baseline
100
300
32
N/A
20
Table This table shows the parameters used in scaling experiments the different encodings. We compare 3 FFFS of increasing frequency and 2 MPEs with fine or coarse grids (e.g., greater than or less than the number of pixels in the image). The PSNR column reports the regressed image against truth at the end of training. For details on training, please see Experimental Setup.
between Mid and High than there is between Low and Mid. The MPE, in contrast, works well for both the coarse and fine configurations. Lastly, Table gives the Optuna found encoding parameters for the 3D implicit surface regression. The tuner sought to minimize the training loss. Results for the implicit surface evaluation can be found in Appendix E.
Hyperparameters Batch Size
Encoding
Learning Rate
Epochs
Grid Parameters
Armadillo (Base)
0.92224
4000
13187
N/A
Armadillo (FFE)
0.78930
4000
9799
L T=7
Armadillo (MPE)
0.99469
4000
10903
k: 1, = 1, 44
Buddha (Base)
0.92224
4000
13187
N/A
Buddha (FFE)
0.78930
4000
9799
=
Buddha (MPE)
0.78445
4000
9267
k = 2, 2, =
[38,102] 102)
Dragon (Base)
0.92224
4000
13187
N/A
Dragon (FFE)
0.78930
4000
9799
Dragon (MPE)
0.80905
4000
9989
k = = 2, [33, 136)
Table 4: This table provides encoding parameters used in the evaluation of 3D implicit surfaces from meshes (see Appendix E). Parameters were found using Optuna minimizing the training loss. Grids were restricted to no more than 3 layers deep and trainable parameters per node.
E 3D IMPLICIT SURFACE
The NTK spectrum was evaluated on 3D implicit surface regression to demonstrate that the theory holds in 3D problems as well. The implicit surface was training using an layer MLP with 256 neurons pre hidden layer. The final layer was passed through sigmoid and then loss was evaluated by binary cross entropy. The input mesh was scaled to normalize its longest axis between 0 and randomly sampled each epoch in the scaled mesh's 's bounding box and input to the
1. Points network. An output of meant the point was outside the mesh and I meant it was inside. To visualize the surface, ray marching method was used render the 0.5 level set of the function,
20

---

Published as a conference paper at ICLR 2025
Referarce
Nuliirid
Frufer
Docel
Figure 14: This figure plots the depth field at the 0.5 level set of the learned implicit surface. This rendering shows the network's ability capture the structure of the input mesh. We find that the MPE outperforms the FFE and baseline in all scenarios, with baseline struggling to capture any recognizable features of the original mesh.
which is expected to represent the surface of the mesh. Again, there is no distinct separating be- tween training and test data, and optimization was done using stochastic gradient descent Kiefer & Wolfowitz (1922).
As expected, we see that the MPE outperforms the FFE and baseline terms fine detail. The baseline network fails to even learn recognizable structure in all cases. The FFE is able to represent some structure, but struggles with finer detail. The corresponding average eigenvalue spectrum was plotted in the main paper in Figure 5b The hyperparameters used in each network can be found in Table 4
21


Reviews:
[{"review_id": "pkQUk49hSg", "reviewer": "Reviewer_EUYS", "summary": "The article proposes a multi-grid parametric encoding approach\nto capture high-frequency information in images. \nThrough a neural tangent kernel analysis angle,\nit is proven that the proposed encoding can learn an NTK \nkernel with a higher eigenvalue spectrum than using no encoding. \nThe superior performance of this approach is further analyzed \nin detail by separating the contribution from the learnable grid, and \nthe embedding space. On the image regression task of ImageNet, \nthe proposed encoding ourperforms Fourier feature encodings.", "questions": "- It is unclear in the definition (1), what is the dimension of x,  is it in R^d? \nIf so what about the dimension of the output \\gamma_F(x), is it 2d x L ? \n- It is not clear what is the notation round + in eq. 3 means.\n- eq. 7 is hard to understand. If K_NTK is a matrix of size NxN, would the size \nof Q be also NxN? In this case, why it makes sense to write Q ( f_theta(x,t) - y ) if y is not on dimension N?\n- Is there an expectation missing in eq. 9 to define the K_NTK, as in eq. 4?\n- The statement of Theorem 1 is not so clear, it would be better to write directly what lambda_i^MPE means as in the proof.\nWhat is the size of the training set X^n in this proof? Isn't it the X in the notation section?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- It is an interesting idea to analyze the impact of different\nencoding schemes of a coordinate of an image using the spectrum\nin the  NTK framework to make the theory concise. \n- The proposed multi-grid parametric encoding works very well in \nthe image regression task.", "weaknesses": "- The writing of the article should still be improved\nfor the reviewer to understand the methodology. Please clarify the questions below.\n- Conceptually, it is still unclear how the spectrum of the NTK \nis related to the high-frequency information of an image, i.e. \nthe large modes of the Fourier transform of a signal X = (x_1,...,x_N) of length N.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ELzD0LCCBr", "reviewer": "Reviewer_uZqD", "summary": "The paper provides a theoretical and empirical analysis demonstrating that Multigrid Parametric Encodings (MPEs) improve neural network performance in learning fine details and handling discontinuities. By deriving the neural tangent kernel (NTK) for MPEs, the authors show that multigrid encoding elevates the NTK's eigenvalue spectrum compared to coordinate-based MLPs, explaining why MPEs capture detail more effectively. Their analysis isolates this improved performance to MPEs’ learnable grid structure rather than the embedding space alone, a distinction from Fourier Feature Encodings (FFEs). Empirical results on 2D image regression with ImageNet data reveal that MPEs, especially those with smaller grid cells, achieve higher Peak Signal-to-Noise Ratio (PSNR) scores, indicating better detail preservation than both FFE and baseline MLP approaches.", "questions": "1. Can the presented theory be applied to other neural network architectures, such as Convolutional Neural Networks (CNNs)?\n2. Do the presented figures show results only for single-layer fully connected neural networks?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper has a well-established theory.\n2. The paper is well structured.\n3. The authors aim to fill the gap in understanding why MPEs improve a network’s performance through the structure of their grid and not their learnable embedding.", "weaknesses": "1. The paper offers limited novelty and applicability. The authors demonstrate that MPEs enhance network performance through grid structure rather than learnable embeddings by analyzing a lower bound on the eigenvalues of the Neural Tangent Kernel (NTK). However, they use well-known methods to address the problem, which limits the work's novelty.\n2. Most of the paper is focused on describing the existing theory which is well described in literature.\nThe evaluation process is insufficient; in my opinion, images from a single dataset (ImageNet) are not enough to thoroughly validate the theory.\n3. In Equation (9), derivatives are denoted by a single prime symbol without specifying the parameter with respect to which they were calculated, creating confusion and potential errors in further derivations, which makes the paper difficult to follow. \n4. Furthermore, the analysis focuses primarily on a single-layer MLP network. While the authors state that the theory can be easily extended to deeper networks, they do not provide this extension, even in the appendix. The structure and width of individual layers in finite-width networks introduce variability in the NTK. For example, deeper layers in finite networks may capture complex feature hierarchies, while shallow layers contribute differently to the NTK. This dependency adds analytical challenges, as each layer’s contribution can influence the network’s performance and generalization in unique ways. This is the main reason why the evaluation process should also be conducted on deeper neural networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HydvikhUa4", "reviewer": "Reviewer_pL4m", "summary": "This paper studies how Multi-grid positional encodings affect the spectrum of the Neural Tangent Kernel of coordinates-based MLPs, and show that it forms a provably better encoding (compare to Fourier) to recover high frequency details.", "questions": "- in Fig. 1, is $d=2$ dimension of the input in the notations or 3 as suggested by the input of the MLP? Either way it's a bit confusing\n- In theorem 1, the result states that the spectrum of NTK of MLP+MPE is uniformly lower bounded by the spectrum of NTK of MLP, could it be possible to quantify given some assumptions how much above it it is? For now we have an empirical answer.\n- What's unclear to me after reading this paper is: since there is a consensus in computer graphics that MPE is better + it seems to work so well out-of-the-box in this work, why isn't everyone (typically in scientific ML) using MPE?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper doesn't make assumptions about what the reader knows or doesn't know and redefines terms and gives clear examples as well as clear expressions for the tools and objects mentioned.\n- Isolating the effect of the embedding size and the grid is really important theoretically and practically. I wouldn't necessarily have thought to ask for this analysis, but seeing it in the paper is definitely a strength.\n- The work is to the point with a clear theoretical contribution backed by solid experiments.", "weaknesses": "- nit: in notations, it would be nice to discuss why $d \\leq 3$ is important for the theoretical analysis and the limitation.\n- nit: in notations, $f_\\theta(x)$ is not a function, it's an expression. Same for $\\gamma(x)$. Btw, they shouldn't both use $x$ as an input.\n- FFE NTK contribution: line 256 says \"we derive the kernel for MPEs and FFEs\", while line 134 says \"Previous analysis of the NTK for coordinate based MLPs has been restricted to FFEs\". I think it's really important to clarify what exactly the contribution is when it comes to the NTK of FFE based MLPs. If the latter is what is actually the case, then a reference to this work would be appropriate.\n- Use of PSNR: because PSNR is an MSE-based metric, it is actually not really sensitive to fine details. I think using a metric like HFEN or a multiscale-SSIM would make more sense in this context.\n- Large-scale realistic experiments: while I do think the current state of the experiments is really solid, for a more impactful work I think larger scale experiments with NeRFs or other SotA models would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AHnwhGKGp9", "reviewer": "Reviewer_L1id", "summary": "The authors look at the Multigrid Parametric Encoding through the lens of Neural Tangent Kernel analysis.  They find that MPE have a raised eigenvalue spectrum compared to baseline encodings.", "questions": "If you use non-parametric networks for encoding an image, maybe it would be useful to also include comparisons to classical image compression algorithms and show their signal to noise ratio?\n\nWhat does Identity correspond to in the plots?\n\nIt would be nice to see the actual learned grid superimposed on the image itself.", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The approach seems straightforward and shows an improvement over the baseline.\n\nI appreciate the theoretical contribution and subsequent evaluation of another approach.", "weaknesses": "I feel like the choice of the grid is not discussed enough.  There is either the choice of having a regularly spaced grid as well as having a learnable one.  There would at least be some intermediate option as well, like having an irregular grid, as it is done in the fast multipole method.  There it has also been observed that using a regular grid leads to unstable results.  Hence, it's not convincing that the grid actually needs to be learnable to be useful, like the authors claim.\n\nThe baseline network is not really defined.  After reading through the paper, I am still not sure what you mean by that and how you trained it.\n\nWhile the analysis performed appears sound, it begs the question of what the take-away message is.  Does MPE always outperform FFE?  Figure 2 is missing the eigenvalue spectrum for FFE.  \n\nAs it stands, the examples seem a bit cherry-picked.  It would improve the paper, if the authors could provide an average eigenvalue spectrum for both approaches considered that gives an idea of how it generalizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)