On the Frequency-bias of Coordinate-MLPs
Authors: Sameera Ramasinghe
Lachlan Ewen MacDonald
Simon Lucey
Venue: NeurIPS 2022 Accept (2022)
Keywords: Coordinate networks
implicit neural representations
implicit regularization

Abstract: We show that typical implicit regularization assumptions for deep neural networks (for regression) do not hold for coordinate-MLPs, a family of MLPs that are now ubiquitous in computer vision for representing high-frequency signals. Lack of such implicit bias disrupts smooth interpolations between training samples, and hampers generalizing across signal regions with different spectra. We investigate this behavior through a Fourier lens and uncover that as the bandwidth of a coordinate-MLP is enhanced, lower frequencies tend to get suppressed unless a suitable prior is provided explicitly. Based on these insights, we propose a simple regularization technique that can mitigate the above problem, which can be  incorporated into existing networks without any architectural modifications.

On the Frequency-Bias of Coordinate-MLPs
Sameera Ramasinghe
Lachlan Macdonald
Simon Lucey
{f @firstnnme.lastnamm lalaaattaaeaadelllllllaaaaaaaaaa @dadelaide.ee.. University of Adelaide
Abstract
We show that typical implicit regularization assumptions for deep neural networks (for regression) do not hold for coordinate-MLPs, a family of MLPs that are now ubiquitous in computer vision for representing high-frequenee signals. Lack of such implicit bias disrupts smooth interpolations between training samples, and hampers generalizing across signal regions with different spectra. We investigate this behavior through Fourier lens and uncover that as the bandwidth of coordinate- MLP enhanced, lower frequencies tend to get suppressed unless a suitable prior is provided explicitly. Based on these insights, we propose simple regularization technique that can mitigate the above problem, which can be incorporated into existing networks without any architectural modifications.
Introduction
It is well-established that deep neural networks (DNN), despite mostly being used in the over- parameterized regime, exhibit remarkable generalization properties without explicit regularization Neyshabur et al. [2014], Zhang et al. [2017], Savarese et al. [2019], Goodfellow et al. [2016]. This behavior questions the classical theories that predict an inverse relationship between model complexity and generalization, and is often referred to in the literature as the "implicit regularization" (or implicit bias) of DNNs Li and Liang [2018]. Kubo et al. [2019], Soudry al. [2018], Poggio al. [2018). Characterizing this surprising phenomenon has been the goal of extensive research in recent years.
In contrast to this mainstream understanding, networks), a class of MLPs that are specifically designed to overcome the spectral bias of regular MLPs, do not follow the same behavior. Spectral bias, as the name suggests, refers the propensity of DNNs to learn functions with low frequencies, making them unsuited for encoding signals with high frequency content. Coordinate-MLL MLPS, on the other hand, are architecturally modified MLPs (via specific activation functions Sitzmann et al. [2020], Ramasinghe and Lucey [2021] or positional embedding schemes Mildenhall et al. [2020], Zheng al. [2021]) that can learn functions with components. By virtue of this unique ability, coordinate-MI.P Ms are now being used extensively computer vision tasks for representing signals including texture generation Henzler et al. [2020], Oechsle et al. [2019], Henzler et al. [2020]. Xiang et al. [2021], shape representation Chen and Zhang [2019]. Deng et al. [2020]. Tiwari et al. [2021], Genova et al. [2020], Basher et al. [2021] I] Mu et al. [2021], Park et al. [2019], and novel view synthesis Mildenhall et al. [2020], Niemeyer et al. [2020], Saito al. [2019], Sitzmann et al. [2019], Yu et al. [2021], Pumarola et al. [2021], Rebain al. [2021], Martin-Bruala al. [2021]. Wang al. [2021]. Park et al. [2021]. However, show in this paper, these architectural alterations entail an unanticipated drawback: coordinate-ML. trained by conventional means via stochastic gradient descent (SGD), incapable of simultancously generalizing well at both lower and higher ends of the spectrum, and thus, are not automatically biased towards less complex solutions (Fig. 1). Based on this observation. question the popular understanding that the implicit bias neural networks is more tied to SGD than the architecture Zhang et al. [2017, 2021].
show that coordinate-MLPP (or implicit neural
will
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Strictly speaking, the term "generalization" is not meaningful without context. For instance, consider a regression problem where the training points are sparsely sampled. Given more trainable parameters than the number of training points, a neural network can, in theory, learn infinitely many interpolants of the training points. Therefore, the challenge is to learn function within a space restricted by certain priors and intuitions regarding the problem at hand. The generalization then can be measured by the extent to which the learned function is close to these prior assumptions about the task. Within regression problem, one intuitive solution that is widely accepted by the practitioners (at least from an engineering perspective) is to have a form of "smooth" interpolation between the training points, where the low-order derivatives are bounded Bishop [2007]. In classical machine learning, in order to restrict the class of learned functions, explicit regularization techniques were used Craven and Wahba [1978], Wahba [1975], Kimeldorf and Wahba [1970]. Paradoxically, however, over- weer-arraeeteiize neural networks with extremely high capacity prefer to converge to such smooth solutions without any explicit regularization, despite having the ability fit more complex functions Zhang et al. [2017, 2021].
Although this expectation of smooth interpolation is valid for both a) general regression problems with regular MLPs and b) high-frequency signal encoding with coordinate-MLPs a key difference exists between their end-goals. In a general regression problem, we do not expect an MLP to perfectly fit the training data. Instead, we expect the MLP to learn a smooth curve that achieves a good trade-off between the bias and variance (generally tied to the anticipation of noisy data), which might not exactly overlap with the training points. In contrast, Noordinate-MLI. are particularly expected to perfectly fir the training data that may include both low and high fluctuations, while interpolating smoothly between samples. This difficult task requires coordinate-ML.. with both low and high frequencies (often with higher spectral energy for low frequencies, as the power spectrum of natural signals such as images tends to behave as 1/ f2 Ruderman [1994]). We show that coordinate-MLPP naturally do not tend to converge such solutions, despite the existence of possible solutions within the parameter space.
to preserve rich spectrum
It should be pointed out that it has indeed been previously observed that Moordinate-MLP tend to pro- duce noisy solutions when the bandwidth is increased excessively Tancik et al. [2020], Ramasinghe and Lucey [2021], Sitzmann et al. [2020]. However, our work complements these previous obser- vations: First, the existing works do not offer an explanation on why providing a coordinate-MLP with the capacity to add high frequencies to the spectrum would necessarily affect lower frequencies. In contrast, we elucidate this behavior through a Fourier lens, and show that as the spectrum of the coordinate-MLPSs is enhanced via seper-parameters or depth, the lower frequencies tend to be suppressed (for a given set of weights), hampering their ability to interpolate smoothly within low-frequency regions. Second, we interpret this behaviour from a model complexity angle, which allows us to incorporate network-depth into our analysis. Third, we show that this effect is common to many types of coordinate-Mo MMPPs i.e. MLPs with a) positional embeddings Mildenhall et al. [2020], b) periodic activations Sitzmann et al. [2020], and c) non -or-ioiooiic activations Ramasinghe and Lucey [2021]. Finally, we propose a simple regularization term that can enforce coordinate-Me MMLPs to preserve both low and high frequencies in practice, enabling better generalization across the spectrum.
Our contributions are summarized below:
We show that coordinate-MI.Ps are not implicitly biased towards low-complexity solutions, i.e, typical implicit regularization assumptions do not hold for coordinate-MIl ML.Ps. A bulk of previous mainstream works try to connect the implicit bias of neural networks to the properties of the optimization procedure (SGD), rather than the architecture Zhang et al. [2017, 2021]. On the contrary, we provide counter-evidence that the implicit bias of neural networks might indeed be strongly tied to the architecture.
We present a general result (Theorem 4.1), that can be used to obtain the Fourier transform of shallow networks with arbitrary activation functions and multi-dimensional inputs, even when the Fourier expressions are not directly integrable over the input space. Utilizing this result we derive explicit formulae to study shallow Coordinate-MLPS from a Fourier perspective.
Using the above expressions, we show that shallow coordinate-MLA tend to suppress lower frequencies when the bandwidth is increased via hyper-parameters or depth, disrupting smooth interpolations. Further, we empirically demonstrate that these theoretical insights from shallow networks extrapolate well to deeper ones.

---

We propose simple regularization technique to enforce smooth interpolations between training samples while perfectly fitting The proposed technique can be easily applied to existing Noordinate-MI.PP without any architectural modifications.
the training points, preserving a rich spectrum.
Our analysis stands out from previous thcoretical research Tancik et al. [2020], Ramasinghe and Lucey [2021], Zheng et al. [2021] on coordinate-MLP's for several reasons: a) we work in a relatively realistic setting excluding assumptions such as infinite-width networks or linear models. Although we do consider shallow networks, we also take into account the effect of increasing depth. b) All our expressions are derived from first principles and do not demand the support of additional literature such as neural tangent kernel (NTK) theory. c) we analyze different types of coordinate-Me MMLLns within a single framework where the gathered insights are common across each type.
2 Related works
Understanding and characterizing the aston- ishing implicit generalization properties of DNNs has been an important research topic in recent years. Despite the overwhelming empirical evidence, establishing a rigorous theoretical underpinning for this behavior has been a challenge to this date. The related research can be broadly categorized into two: investigating implicit regularization on the a) weight space Bishop [1995], Soudry et al. [2018], Poggio et al. [2018], Gidel et al. [2019] and the b) function space Maennel et al. [2018), Kubo et al. [2019], Heiss et al. [2019]. Notably, Maennel et al. [2018]<s ana- lyzed ReLU networks under macroscopic as- sumptions and affirmed that for given input data, there are only finitely many, "simple" functions that can be obtained using regular ReLU MLPs, independent of the network size. Kubo al. [2019] showed that ReLU- MLPs interpolate between training samples almost linearly, and Heiss et al. [2019] took a step further, proving that the interpolations can converge to (nearly) spline approximations. Seminal works by Zhang et al. [2017] and Zhang et al. [2021] showed that these generalization properties are strongly connected to the SGD. In contrast, we show that Coordinate-MI.F tend to converge to complex solutions (given enough capacity), despite being trained with SGD. Further, all above works consider shallow networks in order to obtain precise theoretical guarantees, Similarly, we also utilize shallow architectures for a part of our analysis. However, our work differs from some of the above-mentioned theoretical work, we do not focus on establishing rigorous theoretical bounds on generalization. Rather, we uncover, to the best of our knowledge, critically overlooked shortcoming of coordinate-MLPs. and present plausible reasoning for this phenomenon from a Fourier perspective. We give a brief exposition of coordinate-Mil MMLPs in the next section.
Capeeity (eperparaaeeen)
Soalt Clapenty Mepill
Ground
Figure 1: Implicit regularization assumptions do not hold for CoordinateMMLP'. The network is trained with 10% of the pixels in each instance. As the capacity of the net- work is increased via depth or hyperparameters, coordinate- MLPs tend to produce more complex solutions even though they are trained with SGD. This is in contrast to the regular- MLPs, where the networks converge "smooth" solutions independent the model capacity Kubo et al. [2019], Heiss et al. [2019]. Further, this provides evidence that the implicit regularization of neural networks might be strongly linked to the architecture rather than the optimization procedure, as opposed to mainstream understanding Zhang et al. [2017, 2021].
3 Coordinate-MLPS
Coordinate-MLPP aim to encode continuous signals f IR + ....... images, sound waves, or videos, as their weights. The inputs to the network typically are low-dimensional coordinates, e.g. (z. y) positions, and the outputs are the sampled signal values at each coordinate e.g. pixel intensities. The key difference between coordinate-MLPs and regular MLPs is that the former is designed to encode signals with higher frequencies mitigating the spectral bias of the latter- via specific architectural modifications. Below, we will succinetly discuss three types of Coordinate-MLPS.
e.g.,
Random Fourier Feature (RFF) MIPs are compositions of positional embedding layer and subsequent ReLU layers. Let Ω denote the probability space R" equipped with the Gaussian measure
3

---

of standard deviation 2To > 0. For D 2 1, write elements L of S2" as matrices [1...... 1p] composed of D vectors drawn independently from f. Then, the random Fourier feature (RFF) positional embedding 7 -2P x R 1R22 is
(L, x) = [e x), elp,, x))",
where x R" + R2 is the random function defined by the formula
e(1. [sin(1 x), cos(1 x)].
(2)
The above layer is then followed by a stack of ReLU layers.
Sinusoidal MLPs were originally proposed by Sitzmann et al. [2020]. Let W and b the weights and the bias of a hidden layer of sinusoidal coordinate- MMLL,, respectively. Then, the output of the hidden-layer is sin(2 W x + b)), where a is a syper-parameter that can control the spectral-bias of the network and is the input.
Gaussian MLPs are recently proposed type of coordinate- Miattte-MMPPMM. by Ramasinghe and Lucey [2021].
The output of Gaussian hidden-layer is defined by e-
(W-x+b)2 2n
where a is hyper-parametee
In the next section, will derive a general result that can be used to obtain the Fourier transform of an arbitrary multi-dimensional shallow MLP (given that the ID Fourier transform of the activation function exists), which is used the bedrock in bulk of our derivations later.
4 Fourier transform of shallow MLP
Let G Rs R be MLP with single hidden layer with neurons, and a point-wise activation function a R R. Suppose we are interested in obtaining the Fourier transform & of G. Since the bias only contributes to the DC component, we formulate G as G = - w ox w (()) x). where
(2)
w (1) are rows of the fist-layer affine weight matrix, input the 7th hidden neuron is w (1) X, and w!
are the weights of the last layer. Since the Fourier transform is linear operation, it is straightforward
see that
m wi ā(w!" xx.
(2)
Thus, by obtaining the Fourier transform of f(x) := a(w xx) it is possible to derive the Fourier transform of the MLP. However, doing so using the standard definition of the multi-dimeesional Fourier transform can be infeasible in some cases. For example, consider Gaussian-MI.P Then, one
(wrse) +2 k) dx. However, note that there exists an n 1 dimensional subspace where (w . x) zero and thus, the first term inside the exponential becomes 0 in these cases. Therefore the Fourier transform does not exist as function. Nonetheless, f defines tempered distribution, and its Fourier transform if can therefore be calculated by fixing Schwartz test function &p IR" and using the identity (f.p) (ʃ, R), defining the Fourier transform of the tempered distribution f. Here (:- denotes the pairing between distributions and test functions (see Chapter 8 of Friedlander and Joshi [1999] for this background). Formally, we present the following theorem, whose proof we defer to the appendix.
can hope to calculate the Fourier transform of f(x) as f(k) fg
Theorem 4.1. The Fourier transform of f(x) = a(w x) where w, RR. is the distribution
(2x) 3 á
f(k)
k) dw (k), (k),
|w] [
where 8 (k) is the Dirac delta distribution which concentrates along the line spanned by w.
Next, using the above result, we will gather useful insights into different types of coordinate-Mi from Fourier perspective.

---

Effect of hyperparameters
In this section, we will primarily explore the effect of hyperparameters on the spectrum of coordinate- MLPs, among other key insights.
5.1
Gaussian-MLP
The Fourier transform of the Gaussian activation is ā(k) = VTroe (v2nko)), Now, using Theorem
4.1, we can obtain the Fourier transform of Gaussian-MLP with single hidden layer
( (22
(2x) (1
kr
(k).
(4)
Discussion: For fixed w; (1)s and Ø. the spectral energy is decayed as k is increased. A practitioner increase the energy of higher frequencies by decreasing Second, although decreasing can increase the energy of higher frequencies (for given weights), simultaneously suppresses the energies of lower frequencies due the term outside the exponential. Third, one can achieve the best both worlds (i.e., incorporate higher-frequencier to the spectrum while maintaining
higher energies for lower- (wweer-frquencies) by appropriately tuning w! (1).s. s. To make the above theoretical observations clearer, we demonstrate toy example in Fig. 2.
5.2 Sinusoidal-MILP
Similar to the Gaussian-MLA. the Fourier transform of sinusoidal-MIPP with single hidden-layer can be obtained as
(2T) 2 8 as (8( w a)
Op + +)))
k-a) + (1
k
(S)
2/w?")
Discussion: According Eq. 5, the only frequencies present in the spectrum are multiples aw!") (1) of the weight vectors w It follows that the magnitudes of the frequencies present in the spectrum are boundeb by the number a min |w]). Thus for given set of weights, one can add higher frequencies to the spectrum by increasing At the same time, the spectrum can be balanced by minimizing some w, accordingly.
5.3 RFF-MLPs
We consider a shallow RFF-MLP IR" + R with single hidden ReLU layer. Recall that the positional embedding scheme defined following Eq. and 2. The positional embedding layer is then followed by linear transformation A R220 + R220 yielding
which is then followed by ReLU activations and an affine transformation. ReLU activations can be approximated via linear combinations of various polynomial basis functions Ali et al. [2020], Telgarsky [2017]. One such set of polynomials Newman polynomials defined as
N (x)
II (o exp(-t/V7))-
It is important to note that although we stick to the Newman polynomials in the subsequent derivations, the obtained insights are valid for any polynomial approximation, as they only depend on the power terms of the polynomial approximation. Moving on, for m 2 5, we show in the appendix that the spectrum the RFF-MIP congentrated on frequencies
k
(2a25-1 12)-1) + (202, k22))
(6)
5

---

[w]]]
0.5-
11.5
0.2
0.3
01.2
0.5 1.0
0.5
0.8
0.4
0.6
Figure 2: Toy example (a Gaussian-MLP -> IR with single hidden layer consisting of two neurons): The behavior of the spectrum against and the weights of the network is shown. w and w( are the first and second rows of the first-layer affine weight matrix. Left: By decreasing the network can include higher frequencies the spectrum. However, as the high-frequency components are added to spectrum, energies of the low www-freuenccy components decrease. Middle: The network can still gain high energies for lower frequencies at lower a by decreasing [wo ( and [w]" However, in this case, the high-freguee freeueryyyncy components
relative
are removed from the spectrum. Right: By appropriately tuning [w6" O and [w["), I, the spectrum can include higher-frequene components while preserving the low-frequency energies. All the intensities are normalized for better comprehension.
for non-negative integers ki...., K2D adding to 2. and for all non-negative integers aj I kj for all j = 1,..., 2D. Here we recall that D is the dimension determined by the choice of RFF positional embedding.
Discussion: Recall that if V.V NN(0, o") then, Var(L + +2V = -1 12)0². Therefore the overall spectrum can be made wider by increasing the standard deviation a of the distribution from which the 1, are drawn. However, since the is constant, a larger a makes frequencies less concentrated in the lower end, reducing the overall energy of the low frequency components. Nonetheless, the spectrum can be altered via adjusting the network weights.
Thus far, we established that in all three types of coordinate-MIL both low and high frequencies can be preserved by properly tuning the weights and hyperparameters. However, coordinate-MLPP tend to suppress lower frequencies when trained without explicit regular- ization (see Fig. 5). Moreover, note that in each Fourier expression, the Fourier components cease to
a rich spectrum consisting of
exist if k is not in the direction of w due to the Dirac-delta term. Therefore, the angles between w (1) s should be increased, order add frequency components in different directions.
6 Effect of depth
This section investigates the effects of increasing depth the spectrum of coordinate-MIPPs Com- prehensive derivations of the expressions this section can be found in the Appendix. Let a stack of layers be denoted K : En Rd. Suppose we add another stack of layers n) : E - on top of x(-) to construct the MLP 17 K 12n + R. The Fourier transform of the composite function then becomes,
())
(1/2) a
(10,,((,)))
2
where, 8(k, t) fgn (((t=n(x)-ke kx), That is, the composite Fourier transform the projection of on to (()). The maximum magnitude frequency ki present in the spectrum of (1) o K) is
max (K), max(u n (x))), |u|=1
(8)
where u is the maximum frequency of along Bergner et al. [2006]. Thus, the addition of set of layers 1) with sufficiently rich spectrum tends to increase the maximum magnitude frequency expressible by the network. Next, we focus on the impact on lower frequencies by such stacking. It important to note that the remainder of the discussion in this section is not rigorous theoretical derivation, but rather, an intuitive explanation.

---

We consider the ID case for simplicity. Our intention is to explore the effect of adding more layers on (f(.), ((k, Note that 8 an integral of an oscillating function with unit magnitude. The integral of an oscillating function is non-negligible only at points where the phase is close to zero, i.e. points of stationary phase. It can be deduced that at such points the following condition needs to be satisfied,
min(k < < max(
From the previous analysis, we established that progressively adding layers increases the maximum frequency of the composite function. This causes k to have rapid fluctuations, encouraging min(x (x)) to iecrasse as the network gets deeper (note that the definition of k keeps changing as more layers are added). Note that as min(k') is increased, for smaller k's, Eq. 9 is not satisfied. On the other hand, is non-negligible only at points where Eq. 9 satisfied. Thus, according to Eq. 9, smaller k's causes the quantity (n.-), ((k,
.)) to be smaller, which encourages suppressing the energy of
the lower frequencies of (1) o k). Rigorously speaking, increasing the bandwidth of the network by adding more layers does not necessarily have to increase min min(w' (z)). However, our experimental results strongly suggest that this is the case. Summarizing our insights from Sec. 4, 5 and Sec. 6, we
state the following remarks.
Remark 6.1. In order to gain a richer spectrum, the columns of the affine weight matrices of the coordinate- MLPs need to point in different directions. Ideally, the column vectors should lie along the frequency directions in the target signal.
Remark 6.2. The spectrum of the coordinate-MLP. can be altered either by tuning the hyperparam- eters or changing the depth. In practice, when higher frequencies are added to the spectrum, the energy of lower frequencess tend to be suppressed, disrupting smooth interpolations between samples. However, better solutions exists in the parameter space, thus, an explicit prior is needed to bias the network weights towards those solutions.
In the next section, we will focus on proposing a regularization mechanism that can aid the coordinate- MLPs in finding a solution with a better spectrum.
6000
-005
02
34
8. 95
18
02
044
83
05
18
64
16
(3 200
Orouzd bruhh
wyD.mapphttttom OK
W/ rrgutarmaiio
Specimun
Figure 3: The effect of explicit regularization. When trained under conventional means, coordinate ML.ss cannot generalize well at both higher and lower ends of the spectrum. This hinders the generalization performance of coordinate-MI. MLPPs when the target signal comprises regions with different spectral properties. When the network has insufficient bandwidth, the network cannot correctly capture high-frequency modes. When the network tuned to have higher bandwidth, the network fails al modeling lower frequencies. In contrast, the network can preserve a better spectrum with the proposed regularization scheme. This example uses a 4-layer sinusoid-MLL trained with 33% of the total samples.
7 Regularizing coordinate-MLPs
The spectrum of a function is inherently related to the magnitude of its derivatives. For instance, consider a function f R + defined on finite interval c. Then, it can be shown that d((x) < |2m|| S (k=(())) dkk (see Appendix). For multi-dimensions, we can encourage the spectrum to have a higher or lower frequency support by appropriately constraining the fluctu-
max.ee dz
ations along the corresponding directions. We shall now discuss how this fact may be utilized in regularizing coordinate-Min MLLPs
Let us consider a coordinate-Min MLP f R - R, which we factorise as f f= fo g, where f is the final layer. Then by the chain rule:
df
0f
0g
0 Ög ay
af
1
dx
By
0x
0x
V oy
JJJ
(10)
7

---

W/O Regularization [kf)) W/O Regularization [k]) W/ Regularization
Figure 4: Qualitative results for encoding signals with uneven sampling (zoom in for better view). A Gaussian-MI.P is trained encode an image, where the left half of the image sampled densely, and the right half sampled with 10% pixels. The reconstruction results are shown. When the bandwidth the Gaussian-MI.P is adjusted to match the sampling procedure of a particular half, the other half demonstrates poor reconstruction. In contrast, when regularized, coordinate-MLPP can preserve both low and high frequencies. giving balanced reconstruction. Contrast the green and red areas across each setting.
where J is the Jacobian of g. Let A JJ In order to minimize d it suffices to minimize the magnitude of the components of J, which is equivalent to minimizing the trace of A. Recall that (r) Σx Ake where 0S are the eigenvalues of A. One has lim [_(x)_g(xxxe)) g(xx+
VAg, where we are the eigenvectors of A. Therefore, minimizing an eigenvalue of A is equivalent to restricting the fluctuations of g along the direction of its associated eigenvector. By this logic, the ideal regularization procedure would be identify the directions in the spectrum of the target signal where the frequency support is low, and restrict '₁ 's corresponding to those directions. According to our derivations in the previous sections, this would only be possible by regularizing w, 's,, since in that allow higher bandwidth. However, this is
caa.
practice, we use hyperparameters for coordinate-MLL cumbersome task and we empirically found that there much simpler approximation for this procedure that can give equivalent results (see Appendix) as,
[x) g(x E)
C.
(11)
E
where x are randomly sampled from the coordinate space and & ~ (O, Σ) where Σ is diagonal with small values. The total loss function for the coordinate-MLP then becomes Liotel LSE sC, where & is a small scalar coefficient and LMSE E is the ususal mean squared error loss. The total loss also be interpreted encouraging the networks to obtain the smoothest possible solution while perfectly fitting the training samples. By the same argument, one can apply the above regularization on an arbitrary layer, including the output, although we empirically observed that the penultimate layer performs best. Although the proposed loss has a similar goal to the popular total-variation loss, the former converges more desired solutions (see Appendix).
8 Experiments
In this section, we will show that the insights developed thus far extend well to deep networks in practice.
8.1 Encoding signals with uneven sampling
The earlier sections showed that coordinate-MLP. tend to suppress low frequencies when the network attempts to add higher frequencies to the spectrum. This can lead to poor performance when the target signal is unevenly sampled because, if the sampling is dense, the network needs to incorporate higher frequencies to properly encode the signal. Thus, at regions where the sampling is sparse, the network fails to produce smooth interpolations, resulting in noisy reconstructions. On the other hand, if the bandwidth of the network is restricted via hyperparameters or depth, the network can smoothly interpolate in sparse regions, but fails to encode information with high fidelity at dense regions. Explicit regularization can aid the network in finding properly balanced spectrum in the solution space. Fig. shows an example for encoding ID signal. Fig. 4 illustrates a qualitative example in encoding 2D image. Table depicts quantitative results on the natural dataset by Tancik
8

---

Moussian-MLP
Sinusoid MLP
RFP-MLP
10300
0..8
150
250.
10.0
1300
0.04
30
50.0
200
125
150
375
200
Froquctiy (4)
Frequrkc
Proquricy
10.0
0.0
0001
Layers
Frequancy
200
Frequency
1100
125
175
200
Figure 5: Spectra of the coordinate-MLLS against hyperparameters and depth. We train each network instance to encode sound wave with 33% sampling. The heat indicates the intensity of the corresponding frequency component after trained with the MSE loss. As illustrated, when the capacity of the network increased hyperparameters or depth, the networks tend to converge to solutions with suppressed low- frequency components.
W/O Regularization (ke J)
Type
L.PSNR
R-PSNR
T-FSNR
Gaussian ML.P
ML.P
30.12
22.47
27.72
Sinusoid-ML.
30.49
21.93
26.89
REF-MLP
29.89
22.33
W/O Regularization (kc 1)
Type
L-PSNK
R-PSNK
T-FFNK
Gaussian-
18.14
Sinusoid-MLA
32.17
19.30
23.49
RFF-MLP
32.21
19.11
22.19
Regularized
Type
L-PSNK
R-PSNK
T-FSNK
Gaussian-M1.R
33.11
23.31
30.15
Sinuscid-MLP
31.59
22.66
29.94
RPF-MLP
31.99
22.91
29.59
Table 1: Encoding images with uneven sampling. In each traiming instance, the left half of the image
when the hyperparameters are tuned to match the sampling of particular half, the reconstruction of the other half is poor. In contrast, the encoding performance is balanced when regularized. We use 4-layer networks for this experiment.
sampled densely, and the right half is sampled with 10% pixels. With unregularized coordinate-MI..
et al. [2020]. L-PSNR, R-PSNR, and T-PSNR means PSNR evaluated on the left, right and the total image, respectively.
Encoding signals with different local spectral properties
The difficulty in generalizing well across different regions in the spectrum hinders encoding natural signals such as images when the sampling is sparse, as they tend to contain both "flat" and "fluctuating" regions, and the network has prior how to interpolate in these different regions. See Fig. 6 and Table 2 for qualitative and quantitative results. As evident, the coordinate-MLA struggles to smoothly interpolate between samples under single hyperparameter setting. We also assess the performance of the regularization on NeRF style data. The results are depicted in Fig. 7 and Table. 3.
8.3 Effect of increasing capacity
Our derivations in Sec. 4 and 6 showed that shallow coordinate-MI. tend to suppress lower frequencies when the capacity of the network increased via depth or hyperparameters. Our empirical results show that this is indeed the case for deeper networks well (see Fig. 5).

---

W/O Regularization kk) W/O Regularization k!) W/ Regularization
Figure 6: A single hyperparameter setting cannot generalize well across the spectrum unless regularized (better viewed in zoom). With sparse sampling (10% of the total pixels), coordinate-MI.P exhibit inferior reconstruction performance across regions with different spectra. In comparison, regularized networks achieve the best of both worlds. Compare the highlighted areas across each setting.
STL
Type GaP
W/O Regularization
Regularized
22.11
24.17
Sinusoid-MLP
21.97
25.01
RFF-MLP
22.03
23.31
ImageNet
Gaussian-Mir -LLP
24.59
26.77
Sinusoid-Min RFF-MLP
23.81
26.90
23.11
25.94
Table 2: Encoding images with sparse sampling. We compare the performance the STL dataset Coates
al. [2011] a sub-sampled version of ImageNet with 10% sampling. Regularized coordinate-MLP. show superior performance due to better interpolation properties. We use 4- coordinate- MLPs for this experiment.
Loss
PSNR
SSIM
MSE
30.91
MSE L2
30.14
0.935
MSE
30.87
0.940
MSE-TV loss
29.11
MSE+proposed reg.
32.01
0.951
Table 3: Quantitative comparison in novel view syn- thesis on real synthetic dataset [Mildenhall et al., 2020J. The proposed regularization achieves better re- sults. In contrast, TV loss worsens the performance. Note that formance converges vanilla MSE loss the weight gets smaller. regularization improves results slightly, but still is inferior to the proposed regularizer.
Figure 7: Our regularization able achieve bet- ter reconstructions (NeRF' data). Top row: Proposed regularization suppresses the unnecessary higher fre- quencies. Bottom row: by increasing the frequeney support of the positional embedding layer and then applying the regularization. Cropped and zoomed in for better viewing.
TV loss with 0.001 weight, and its per-
obtain sharper results
9 Limitations
Currently, our work theoretically analyzes coordinate-ne networks within a constrained shallow setting. Although the insights we gather from this restricted setting can be empirically extrapolated to deeper networks (as have shown), future research should focus deriving rigorous formulae analyze practical, deeper coordinate networks. Further, the current approach involves manually tuning the weights of the loss components, which requires fair amount of domain knowledge. Perhaps, more in-depth study can reveal general guidelines for tuning these parameters.
10 Conclusion
We show that the traditional implicit regularization assumptions do not hold in the context of coordinate-MLPs We focus on establishing plausible reasoning for this phenomenon from Fourier angle and discover that coordinate-MLPS tend suppress lower frequencies when the capacity is increased unless explicitly regularized. We further show that the developed insights are valid in practice.
10

---

References
Ramy E Ali, Jinhyun So, and A Salman Avestimehr. On polynomial approximations for privacy- preserving and verifiable relu networks. arXiv preprint arXiv:2011.055300 2020.
Abol Basher, Muhammad Sarmad. and Jani Boutellier. Lightsal: Lightweight sign agnostic learning for implicit surface representation. arXiv preprint arXiv:2 /4273, 2021.
Steven Bergner, Torsten Moller, Daniel Weiskopf, and David J Muraki. A spectral analysis of function composition and its implications for sampling in direct volume visualization. IEEE transactions on visualization and computer graphics, 12(5):1353-135) 2006.
Christopher M Bishop. Regularization and complexity control in ned-forwwrd networks. 1995.
Christopher Bishop. Pattern recognition and machine learning (information science and statistics), 2007.
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034. PMLR, 2020.
Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint 1912vv1912.01199 2019.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5939-5948, 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
Peter Craven and Grace Wahba. Smoothing noisy data with spline functions. Numerische mathematik. 31(4):377-433 1978.
Boyang Deng, John P Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Nasa neural articulated shape approximation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pages 612-628. Springer, 2020.
G. Friedlander and M. Joshi. Introduction to the Theory of Distributions. Cambridge University Press, 1999.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Samaa, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages
2020.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. arXiv preprint arXiv: /3262, 2019.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Jakob Heiss, Josef Teichmann, and Hanna Wutte. How implicit regularization of neural networks affects the learned function part i. arXiv, pages 1911-02903, 2019.
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Learning neural 3d texture space from 2d exemplars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8356-8364, 2020.
George S Kimeldorf and Grace Wahba. A correspondence between bayesian estimation on stochastic processes and smoothing by splines. The Annals of Mathematical Statistics, 41(2):495-502, 1970.
Masayoshi Kubo, Ryotaro Banno, Hidetaka Manabe, and Masataka Minoji. Implicit regularization in over-parameterized neural networks. arXiv preprint arXiv: 1rXIv:1903.01997, 2019.

---

Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data, arXiv preprint 1808.0 01204, 2018.
Tao Luo, Zheng Ma, Zhi-Qin John Xu, and Yaoyu Zhang. Theory of the frequency principle for general deep neural networks. arXiv preprint arXiv: 19Xi:::9006002355. 2019.
Tao Luo, Zheng Ma, Zhi-Qin John Xu, and Yaoyu Zhang. On the exact computation of linear frequency principle dynamics and its generalization. arXiv preprint arXiv:2010.08/53. 2020.
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly, Gradient descent quantizes relu network features. arXiv preprint arXiv:1803.0 100:: 083677. 2018.
Ricardo
Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recognition, pages 7210-7219, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation. arXiv preprint 0074552004.07645, 2021.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv: 1412.6614, 2014.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5504-3515, 2020.
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531-4540, 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 164,174, 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven Scitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
Tomaso Poggio, Qianli Liao, Brando Miranda, Andrzej Banburski, Xavier Boix, and Jack Hidary. Theory iibb Generalization in deep networks. arXiv preprint arXiv: /806. / 1179, 2018.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps. arXiv preprint arXiv:2 2021.
Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14153-14161, 2021.
Daniel Ruderman. The statistics of natural images. Network: computation in neural systems, 5(4): 517, 1994.
12

---

Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 2304-2311 2019.
Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory, pages 2667-2690. PMLR, 2019.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous -d-strucctre-aware neural scene representations. arXiv preprint arXiv:/906.01661 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson. Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822-2878, 2018.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.1077 2020.
Matus Telgarsky. Neural networks and rational functions. International Conference on Machine Learning, pages 2387-33923 PMLR, 2017.
Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll. Neural- gint Neural gen- eralized implicit functions for animating people in clothing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11708-11718, 2021.
Grace Wahba. Smoothing noisy data with spline functions. Numerische mathematik, 24(5):383-393, 1975.
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv: 102i:22102.07044, 2021.
Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold- Geoffroy, Kalyan Sunkavalli, and Hao Su.
Neutex: Neural texture mapping for volumetric neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7119-7128, 2021.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In International Conference on Neural Information Processing, pages 264-271 244-274. Springer, 2019.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization (2016). arXiv preprint arXiv:1611.03530, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Yaoyu Zhang. Zhi-Qin John Xu, Tao Luo, and Zheng Ma. Explicitizing implicit bias of the frequency principle in two-layer neural networks. arXiv preprint arXiv:/90S, 10264, 2019.
Jianqiao Zheng, Sameera Ramasinghe, and Simon Lucey. Rethinking positional encoding. arXiv preprint wtX.h::10770056/1 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "u02FjINSmpN", "reviewer": "Reviewer_Vdaq", "summary": "This paper studies coordinate-MLPs, a class of multi-layer perceptrons that are trained to parametrize high-frequency signals over a low-dimensional domain.\n\nThe paper analyzes three main classes of coordinate-MLP, respectively based on random Fourier features, sinusoidal activations, and Gaussian activations. By proposing a new way of computing the Fourier transform of these MLPs (in the two layers case), the authors observe the effect of the main hyperparameters of the models on the spectrum.\nSimilarly, the authors study the effect of increasing the depth of the MLPs. \n\nThe authors conclude that, in all three cases, changing hyperparameters to allow the model to represent higher frequencies has the effect of decreasing the spectral energy in the lower frequencies.\nThe authors use this result to propose a regularization scheme based on minimizing the trace of $\\mathbf{J}\\mathbf{J}^\\top$, where $\\mathbf{J}$ is the Jacobian of the penultimate layer. This has the effect of limiting the high-frequency components of the MLP associated with low-energy regions of the target signal. \n\nHowever, the authors also note that one can avoid computing the Jacobian by minimizing a much simpler loss designed to make the output of the penultimate layer smoother along randomly sampled directions. \n\nThe paper is concluded with a series of experiments showing that the regularized MLPs can learn to generalize better than their unregularized counterparts, on signals sampled at variable intervals and on signals with specific spectral features (e.g., Fig. 7).", "questions": "- In Section 5.2, it is not clear why increasing $a$ means decreasing the energy of the low-frequency components. Can the authors comment on this?", "limitations": "The limitations of the work are not really discussed. There's probably something to say about the fact that the theoretical analysis is conducted on 2-layer MLPs and the analysis is assumed to hold on deeper networks.\nAlso, the authors should comment on the empirical differences between the regularization based on the Jacobian and the simplified regularization.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I really enjoyed the paper and I can safely recommend its acceptance. \nThe contributions are entirely novel and the paper is very well-written and explained. \nThe paper is likely to have a significant impact on the growing community of implicit neural representations/neural fields and gives non-trivial insights into well-known problems. \n\nI apologize that I cannot write a longer review, but I only have a few minor comments on the paper: \n\n- The numbers reported in Table 1 should be explained in the text or the caption. \n- The reasoning in lines 252-257 is not immediate to follow. I suggest the authors spend a few more words to facilitate the intuition of the result beyond the empirical observation. \n- The comparisons in Figure 4 are difficult to see, I suggest either increasing the size of the images or the resolution (or zooming in on the cutouts).\n- Figure 6 could be improved by turning it into parallel line plots instead of scatter plots. \n- The authors should comment on the empirical differences between the regularization based on the Jacobian and the simplified regularization. An experiment showing the difference in practice (computational cost, quality of the learned signal, etc.) would be a valuable addition to the paper. \n- Typos: \n    - Line 47: Biship -> Bishop\n    - Line 162: Fourier transform the activation -> Fourier transform of the activation\n    - Line 288: Much more simpler -> Much simpler", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TN19qPCgLw", "reviewer": "Reviewer_F1QE", "summary": "The paper studies implicit regularization in implicit neural representation approaches, primarily coordinate-MLPs. The authors analyze failure modes of coordinate-MLPs by carrying out an analysis of shallow MLP models, investigating the effect of depth and hyperparameters on their spectral bias. A smoothness regularization method is introduced and evaluated on image encoding.", "questions": "* Coordinate-MLPs are finding application even outside of implicit representation of data. It is common to provide \"coordinates\" as additional inputs to neural network architectures, or to use periodic activation functions (see for example current architectures for diffusion models (1), or neural operators (2, 3) ). I am curious if the authors think the proposed regularized could be helpful even in general deep learning tasks.\n\n(1) Ho et al., Denoising Diffusion Probabilistic Models\n\n(2) Li et al., Fourier Neural Operator for Parametric Partial Differential Equations\n\n(3) Poli et al., Transform Once: Efficient Operator Learning in Frequency Domain", "limitations": "The authors provided a satisfactory discussion on limitations and comparisons with similar approaches (A.6 and A.7).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The authors carry out an extensive analysis of the behavior of various implicit neural architectures (sinusoidal, gaussian and RFFs) commonly used. Curated examples are provided in support of the theoretical investigation. \n* The proposed regularizer is conceptually simple and appears to improve PSNR across evaluations. \n\n**Weaknesses**\n\n* Only images are considered. The paper would be stronger if the proposed regularized was tested on different modalities, such as encoding of audio samples. The experimental evaluation is generally limited - only 500 images from ImageNet (unclear how these are chosen), a subset of STL, and some synthetic data.  \n* No code is provided. The experimental details provided are not sufficient for reproducibility.\n\n\nSome minor typos and comments:\n* l 162: Fourier transform OF the activation exists\n* l 339: Bishop rather than Biship.\n* l 528 (appendix), missing reference (??)\n* 576 (appendix) erilest -> earliest\n* Readability of Lemma A.1 proof could be improved by always specifying the integration domain, particularly the steps where Fubini's theorem is invoked.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "71IfhW6B2Q9", "reviewer": "Reviewer_4Raw", "summary": "One-liner: Authors study the spectral bias of INRs, and propose a regularizer to fix it.\n\nThis paper argues that popular neural network architectures for INRs---or coordinate-MLPs, as authors call it---tend to learn functions with \"suppressed\" low frequency components. To support this claim, the paper provides (i) Fourier transforms of the function expressed by (the two-layer versions of) these architectures which hints the effect of depth and hyperparameters on low-and-high frequency components, and (ii) empirical results showing that the spectra of the functions learned by these architectures depend on the hyperparameters and depth. Based on this observation, authors propose a (randomized) regularizer which aims to amplify the low-frequency component of the learned signal.", "questions": "Here is an additional question, that I do not necessarily view as a weakness but very important to answer nevertheless.\n\n- One of the critical architectural factor has not been discussed---width! From my understanding, width is one of the most critical factors about the implicit regularizations of overparameterized neural networks (not confined to INR context). In fact, I believe that width is often treated as a more important factor for generalization than depth. Indeed, I did observe from my own experimental results that widening the networks tend to have beneficial impact on the super-resolution capabilities of INRs. I wonder if the authors' analytical tools can be useful in studying the effects of width. This should help strengthening the main message of the paper.", "limitations": "I couldn't find any explicit discussions on limitations or potential negative impact, but I do not see this as a big drawback, because I also could not see a very clear potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- A strength of this paper is its section 7, where the authors design a regularizer motivated by the spectral understanding of the network. Although I do not agree to some points (e.g., the tightness of the lower bound, as being sharp is different from being tight), I do find the approach original and interesting.\n \n- Another thing that I like about this paper is the quality of writing. I appreciate the authors' effort on distinguishing what can be shown rigorously and what cannot be, e.g., as in Line 255.\n \n- Also, the message conveyed in the paper is potentially very meaningful: \"Conventional way of boosting the expressivity of INRs may be at odds with boosting the generalization performance, unlike as in conventional model architectures.\"\n\n- A weak point is that some of the main claims are not properly formalized or ill-supported. As an example, authors argue that 'vanilla INRs are incapable of simultaneously generalizing well at both lower and higher ends of the spectrum,' or 'INRs do not naturally converge to solutions with rich spectrums in both low and high frequencies.' Here, I am not really sure what authors mean---I do not understand the ideal property the authors are looking for. From figure 1 (and later figures), I think that authors are actually looking for architectural bias that 'prefers low-frequency solutions whenever a perfect fit is possible,' instead of one that is frequency-neutral. This might be the reason why I was a bit puzzled when I saw Figure 6, where the authors claim that INRs have \"suppressed\" low freq. From the figure, I cannot really tell---the color distribution seems to be quite uniform on SIREN and RFF, given deep enough depth and large enough hyperparameters? I cannot imagine the \"ideal heatmap\" the authors have in mind, so that the SIREN/RFF has a relatively smaller low freq components. This part should be clear.\n \n- In my opinion, theoretical contribution of this paper is rather weak. The paper does a nice job in presenting two-layer Fourier transforms of INRs, but I do not think they directly imply the main message. Especially, the arguments in section 6 (depth) is quite hand-wavy and not persuasive. In this aspect, I suggest that authors put more emphasis on describing the empirical observations and contributions.\n \n- As theoretical analysis does not give any decisive conclusion, I think that the the experimental section needs to be more emphasized. Section 8 lacks a proper baseline---one should test the effectiveness of the proposed regularizer by comparing against existing regularizers, e.g., weight decay. In fact, it seems very likely that simply performing the weight decay may help boosting the low-frequency component of the learned signal, as model weights are what is being used as \"frequency,\" especially in SIRENs.\n\n- A nitpick is that it is very difficult to read the text on most figures if I print the paper.\n\n- Nitpick#2: Line 75: \"this effect is common to all the types of coordinate-MLPs.\" This cannot be true, as there are much more different architectures for INRs, e.g., multiplicative filter networks.\n\n- Nitpick#3: I do not agree to the authors' point that previous mainstream works focused more on the contribution of SGD on the model bias than the architectural bias. In fact, while the term \"implicit bias\" or \"implicit regularization\" is being almost exclusively used for describing the work about how GD/SGD shapes the learned function, there are still many works on the impact of model architectures. For instance, there are plenty of works regarding ConvNet bias vs. Transformer bias, e.g., [ConViT](https://arxiv.org/abs/2103.10697). I do not even think that Zhang et al.'s main message was that implicit bias is \"not\" tied to the architecture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-whu1d_SZG", "reviewer": "Reviewer_h9m2", "summary": "The paper shows that coordinate MLPs lose some of their implicit regularisation properties when they are adapted to better represent high frequency signals. This suggests that implicit regularisation properties of neural networks are not necessarily a consequence of using SGD, but instead related to the network architecture. The authors suggest a new regularisation technique that allows coordinate MLPs to maintain high quality in their representation of low frequency signals, while still allowing improved representation of high frequency signals. The performance of the suggested explicit regularisation is demonstrated by multiple experiments.", "questions": "### Q1\n\nIn Table1 what is difference of L-PSNR, R-PSNR and T-PSNR? I think I can roughly deduct what they are, but explicitly stating these would be helpful.\n\n### Typos and other minor things\nLine 147 typo: positonal\n\nLine 162 typo: transform _of_ the activation?\n\nFigure 2 caption overlaps with x-axis titles\n\nLine 210 typo: follwing\n\nFigure 4 caption typo: densly\n\nFig 7 caption: is able _to_ achieve\n", "limitations": "The authors comment on which of their results are novel, but the limitations of the proposed regularisation method are not included. It could be useful to include a comment on whether using the proposed method makes training the coordinate MLPs more difficult or requires tuning some training related hyperparameters. Potential societal impact of the work is not discussed, but this doesn't seem very relevant or necessary for this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\nLoss of implicit generalisation in coordinate MLPs has been observed before, but this paper provides some theoretical reasoning to why this happens. The paper also suggests a new method to help resolve this problem.\n\n### Quality\n\nExtensive references to prior work, consistent story throughout the paper, theoretical result of why implicit regularisation of coordinate MLPs deteriorates are mostly supported by proper derivations. For some findings, proper proofs are not provided but intuitive reasoning supported by maths is still given.\n\n### Clarity\n\nVery clear explanations of all relevant concepts, well readable without going into references also for a reader with less expertise in the specific topic. Experiments help to clarify the effects of the proposed method as well as the existing issues with coordinate MLPs. Contributions are very clearly written out.\n\n### Significance\n\nNew method proposed with practical use in a field that has recently been popular.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept