Fast Training of Sinusoidal Neural Fields via Scaling Initialization
Authors: Taesun Yeom
Sangyoon Lee
Jaeho Lee
Venue: ICLR 2025 Poster (2025)
Keywords: sinusoidal neural fields
fast training
initializations
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a simple way to accelerate the training of sinusoidal neural fields.

Abstract: Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have a high training cost, which prevents a broader adoption. In this paper, we focus on a popular family of neural fields, called sinusoidal neural fields (SNFs), and study how it should be initialized to maximize the training speed. We find that the standard initialization scheme for SNFs---designed based on the signal propagation principle---is suboptimal. In particular, we show that by simply multiplying each weight (except for the last layer) by a constant, we can accelerate SNF training by 10$\times$. This method, coined _weight scaling_, consistently provides a significant speedup over various data domains, allowing the SNFs to train faster than more recently proposed architectures. To understand why the weight scaling works well, we conduct extensive theoretical and empirical analyses which reveal that the weight scaling not only resolves the spectral bias quite effectively but also enjoys a well-conditioned optimization trajectory.

Published as a conference paper at ICLR 2025
FAST TRAINING OF SINUSOIDAL NEURAL FIELDS VIA SCALING INITIALIZATION
Taesun Yeom, Sangyoon Lee" & Jaeho Leef Pohang University of Science and Technology (POSTECH) (tayeom, sangyoon. lee, jaeho. lee (ee)@postech..o ac.
ABSTRACT
Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have high training cost, which prevents broader adoption. In this paper, we focus on a popular family of neural fields, called sinusoidal neural fields (SNFs), and study how it should be initialized maximize the training speed. We find that the standard initialization scheme for SNFs-designed based on the signal propagation principle-is suboptimal. In particular, we show that by simply mul- tiplying each weight (except for the last layer) by constant, we can accelerate SNF training by 10x. This method, coined weight scaling, consistently provides a significant speedup over various data domains, allowing the SNFs to train faster than more recently proposed architectures. To understand why the weight scaling works well, we conduct extensive theoretical and empirical analyses which reveal that the weight scaling not only resolves the spectral bias quite effectively but also enjoys a well-conditioned optimization trajectory.
INTRODUCTION
Neural field (NF) is a special family of neural networks designed to represent single datum (Xie et al., 2022). Precisely, NFs parametrize each datum with the weights of a neural net which is trained fit the mapping from spatiotemporal coordinates corresponding signal values. Thanks to their versatility and low memory footprint, NFs have been rapidly adopted in various domains, including high-dimensional computer vision and graphics (Sitzmann et al., 2019; Mildenhall al., 2020; Poole et al., 2023), physics-informed machine learning (Wang et al., 2021; Serrano et al., 2023), robotics (Simeonov et al., 2022), and non- -on-Euclidean signal processing (Rebain et al., 2024).
For neural fields, the training speed is of vital importance. Representing each datum requires tedious training of a neural network, which can take up to several hours of GPU-based training (Mildenhall et al., 2020). This computational burden becomes critical obstacle toward adoption to tasks that involve large-scale data, such as NF-based inference or generation (Ma et al., 2024; Papa al., 2024). To address this issue, many research have been taken to accelerate NF training, including fast-trainable NF architectures (Sitzmann et al., 2020b; Müller et al., 2022), meta- learning (Sitzmann et al., 2020a; Chen & Wang, 2022), and data transformations (Seo et al., 2024).
NF
Despite such efforts, a critical aspect of NF training remains understudied: How should we initialize NF's for the fastest training? While the importance of good initialization has been much contem- plated in classic deep learning literature (Glorot & Bengio, 2010; Zhang et al., 2019), such under- standings do not immediately answer our question, for many reasons. First, conventional works mostly focus on how initialization affects the signal propagation properties of deep networks with hundreds of layers, while most NFs use shallow models with only a few layers (Mildenhall et al., 2020). Second, conventional works aim to maximize the model quality at convergence, while in NFs is often more important how fast the network achieves certain fidelity criterion (Müller et al., 2022). Lastly, NFs put great emphasis on how the model performs seen inputs (i.e., training loss) and leaves how well it interpolates between coordinates (i.e., generalizability) as a secondary issue, while conventional neural networks care exclusively on the test performance.
Equua contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
50%
Pr
Pr -
or
Figure 1. A simple weight scaling accelerates training. The proposed weight scaling scales up the initial weights of SNF by the factor of except for the last layer (left panel). The weight scaled SNF significantly speeds up training across a variety of methods (right panel: train PSNR curve for single Kodak image).
To fill this gap, in this paper, we investigate how the initialization affects the training efficiency of neural fields. In particular, we focus on initializing sinusoidal neural field (SNF), i.e, multilayer perceptrons with sinusoidal activation functions (Sitzmann et al., 2020b). SNF is one of the most widely used NFs, due to its versatility and parameter-efficiency: the architecture works as a strong baseline in wide range of data modalities (Grattarola & Vandergheynst, 2022; Kim et al., 2024; RuBwurm et al., 2024) and is used as an indispensable building block of many state-of-the-an neural field frameworks (Guo al., 2023; Schwarz al., 2023; Pal al., 2024).
Contribution. We first discover that the current practice of SNF initialization is highly suboptimal in terms of the training speed (Sitzmann al., 2020b). Precisely, we find that one can accelerate the training speed of SNFs by up to 10x, simply by scaling up the initial weights of all layers by well-chosen factor. This extremely simple tuning, coined weight sealing (WS), provides much greater speedup than the conventional SNF tuning strategy, i.e., scaling up the frequency multiplier of the activation functions (called wg Sitzmann et al. (2020h)). Furthermore, this speedup can be achieved without any degradation in the generalization capabilities of the model, if we keep the scaling factor at a moderate level. Interestingly, experiments suggest that the optimal scaling factor, which strikes the balance between speedup and generalization, mainly depends the physical scale of the NF workload (e.g., data resolution and model size) and remains similar over different data.
To demystify why the weight scaling accelerates the SNF training, conduct in in-depth analysis. In particular, we focus on understanding how the effects of WS on SNFs are similar yet different from ReLU nets, where the so-called lazy training phenomenon takes place (Chizat et al., 2019)- weight-scaled ReI.U nets behave similarly to kernels, enjoying exponential convergence in theory for shallow nets but failing to achieve good accuracy at practical scales due ill-conditioned tra- jectories. Our theoretical and empirical analyses highlight several key differences that sinusoidal activations bring, such as: (I) Weight- SNFs also preserve activation distributions over lay- ers, facilitating the gradient propagation when training a deep model (Proposition 1), (2) Weight scaling increases not only the frequency of each basis, but also increases the relative magnitude of higher-order harmonics, which helps fit the high-frequene components abundant in natural signals (Lemma 3 and Section 4.2). (3) Weight-scaled SNFs enjoy even better-conditioned trajectories than the unscaled models, captured by the eigenspectrum analysis (Section 4.3).
In sense, our study constitutes a call for rethinking the role the initialization for neural fields, by demonstrating how dramatically suboptimal the current schemes be in key performance metrics for NFs. We hope that our work will further inspire more sophisticated initialization algorithms.
2 RELATED WORK
Training efficiency neural fields. There are two prevailing strategies to accelerate NF training- The first approach is to adopt advanced NF architectures (Fathony et al., 2021; Dou et al., 2023), which mitigate the spectral bias neural nets that hinders fitting high-frequency signals (Rahaman et al., 2019; Basri et al., 2019; Xu et al., 2020; Cao et al., 2021; Choraria et al., 2022). Popular examples include utilizing specially designed positional encodings (Tancik et al., 2020; Müller et al., 2022) activation functions (Sitzmann et al., 2020b; Ramasinghe & Lucey, 2022; Liu et al., 2024; Saratchandran et al., 2024c). Another strategy is to amortize the training cost via meta- e.g., by training a fast-trainable initialization (Sitzmann al., 2020a; Tancik et al., 2021) constructing
2

---

Published as a conference paper at ICLR 2025
an auxiliary model that predicts NF parameters from the datum (Chen & Wang, 2022). This work aims to accelerate NF training through a better initialization. Unlike meta-learned initializations, however, our aim is to understand how the initialization (its scale, specifically) can facilitate training by analyzing how it affects optimization dynamics, without meta-learning involved.
Neural network initialization. Early works network initialization focus preserving signals through layers via principled weight scaling (Glorot & Bengio, 2010; He et al., 2015) or orthogo- nalization (Xiao al., 2018). Recent studies explore its impact on the implicit bias of SGD (Vardi, 2023), where initialization scale determines whether training occurs in the rich or kernel regime (Ja- cot al., 2018; Woodworth et al., 2020; Varre al., 2023). While aligned with this line of research, we primarily investigate how initialization affects training speed, particularly in neural fields.
Initializing sinusoidal neural fields. There is limited number of works that study the initializa- tion for SNFs. Sitzmann et al. (2020b) proposes an initialization scheme for SNF, designed in the signal propagation perspective. Tancik et al. (2021) develops a meta-learning algorithm learn initializations for SNFs. Lee et al. (2021) extends the algorithm for sparse networks. Most related to our work, Saratchandran et al. (2024b) proposes an initialization scheme that aims to facilitate the parameter-efficiency of SNFs. Different from this work, our work primarily focuses on the training efficiency, and achieves much faster training speed empirically.
Sinusoidal activation functions in ML. Beyond their well-know use in NF literature (Sitzmann et al., 2020b; Ashkenazi & Treister, 2024), sinusoidal activation functions have also been studied in other domains. Examples include Fourier neural networks (Gashler & Ashmore, 2014), Bayesian neural networks and out-of-distribution detection (Meronen et al., 2021), low-rank learning (Ji et al., 2025), neural compression (Thrash al., 2025), and generative models (Chan al., 2021).
NEURAL FIELD INITIALIZATION VIA WEIGHT SCALING
In this section, we formally describe the weight scaling for a sinusoidal neural field initialization, and briefly discuss the theoretical and empirical properties of the method.
3.1 NETWORK SETUP AND NOTATIONS
We begin by introducing the necessary notations and problem setup. A sinusoidal neural field is multilayer perceptron (MLP) with sinusoidal activation functions (Sitzmann et al., 2020b). More concretely, suppose that have input X R where N is the number of dn-dimensional coordinates. Then, an l-layer SNF can be characterized recursively via:
S(X; W) = ( z + 6
z (W ((-1))((000)1)1)))) + b0), i -]1 (I)
parametrized by W (W b (1))) 6()), where each ( E Rdxxd-11 denotes the weight matrix, ((8) E Rdd denotes the bias, and 2 E Rd denotes the activation of the ith layer. Here, we let ( = X, and the activation function entrywise, with some frequency multiplier w. The standard practice to use a different frequency multiplier for the first layer. Precisely, we apply the activation function d (x) = sin(wp x)) for the first layer, and (x) = sin(wn x) for all other layers, i.e., +f
RR4, - Rd4 are the sinusoidal activations applied
The SNF is trained using the gradient descent, with the mean-squared error (MSE) as a loss function. We assume that the input coordinates of the training data (i.e., the column vectors of the input data) have been scaled to lie inside the hypercube [-1, +1]d, as in Sitzmann al. (2020b).
3.2 STANDARD INITIALIZATION
The standard initialization scheme for the sinusoidal neural fields independently draws each weight entry from random distribution (Sitzmann al., 2020b). Precisely, the weight matrices are initial- ized according to the scaled uniform distribution as
(1) iid. Unif w j.b
(-1 1 (-d dp dy
(( dule
(5) fid Unif
V6
V6
(2)
Wh / wh v /d
where j denotes the (j, -)tth entry of the ith layer weight matrix W()
3

---

Published as a conference paper at ICLR 2025
40
-00
Teel PSWn
Toss PSMR
-40
10
10
Frequaccy Turing
Waight Seaiig (0uss
1.5
2.5
0.3
1.0
10.0
40.0
80.0
Acceleration
(a) Weight scaling with various scaling factors
(b) Tradcoff curve of WS frequency tuning
Figure 2. Scaling factors and the speed-generalization tradeoff. (a) As we increase the scaling factor a, the training speed (o) tends to become faster while the interpolation performance (o) gets lower. Notably, however, there exists some range of where enjoy acceleration with negligible degradation in test PSNR. (h) Comparing with the frequency tuning (r), the weight scaling (o) achieves better tradeoff. Further experimental details about tradeoff curve is provided in Appendix F.1.
Importantly, the range of the uniform initialization (Eq. (2)) is determined based on the distribution preserving principle, which aims to make the distribution of the activation Z, similar throughout all layers. In particular, Sitzmann et al. (2020b, Theorem 1.8) formally proves that the choice of the numerator (i.e., v6) ensures the activation distribution to be constantly aresin(-1, throughout all hidden layers of the initialized sinusoidal neural field.
Frequency tuning. A popular way to tune the initialization (Eq. (2)) for the given data is by mod- ifying the frequency multiplier of the first layer wn. This method, which we call frequency tuning, affects the frequency spectrum of the initial SNE, enhancing the fitting of signals within certain fre- quency range. is not typical to tune wh, as changing its value does not change the initial SNF; the effects cancelled out by the scaling factors in the initialization scheme.
3.3 INITIALIZATION WITH WEIGIIT SCALING
In this paper, we develop a simple generalization of the standard initialization scheme. In particular, we consider weight scaling, which multiplies certain scaling factor 2 to the initialized weights all layers except for the last layer. More concretely, the initialization scheme is
(- dn dg
av6
av6
w j.k Unif
(1) iid.
(i) Lid Unif ji
(3)
wh Wh /d:
where we let a 1 for the last layer; this design is due to the fact that any scaling in the last layer weight leads direct scaling of the initial function f f af, which is known to be harmful for the generalizability, according to the theoretical work of Chizat et al. (2019). Empirically, we observe that this weight scaling is much more effective in speeding up the training than a recently proposed variant which scales up only the last layer of the NF (Saratchandran et al., 2024b); see Section 5.
There are two notable characteristics of the weight-scaled initializations.
(1) Distribution preservation. An attractive property of the proposed weight scaling is that, for SNFs, any a 2 also preserves the activation distribution over the layers.' 1 In particular, we provide the following result, which shows that the activation distributions will still be an arcsin(-1, Proposition 1 (Informal; extension of Sitzmann et al. (2020b, Theorem 1.8)). Consider an l-layer SNF initialized with the weight scaling (Eq. (3)). For any 2 1, we have, in an approximate sense,
z arcsin( arcsin(-1,1)
Vi E (2....1--1}.
(4)
The detailed statement of this proposition and the numerical derivation will be given in Appendix A.
The proposition implies that concerns regarding preserving the forward propagated signal. Thus, we can safely adopt the weight scaling for training very deep sinusoidal neural field.
tune the scaling factor within the range [1, oo) without any
'This point has also been noted in Sitzmann et al. (2020b), but has not heen discussed quantitatively for We make this point concrete via numerical analyses the large case.
4

---

Published as a conference paper at ICLR 2025
(2) Speed vs. generalization. Empirically, observe that the weight scaling (Eq. (3)) allows to explore for the "sweet spot" region, where we can enjoy the accelerated training, while not suffering from any degradation in the generalization performance unseen coordinates.
The Fig. 2 illustrates this phenomenon for the case of image regression, where we train an SNF to fit downsampled image until some desired peak signal-to-noke ratio (PSNR) is met, and then the model to interpolate between seen pixel coordinates. We observe that by increasing a from 1, the loss on unseen coordinates increase, and the training speed increases as well. Notably, there exists some or > 1 where the test PSNR remains similar while the training speed has strictly increased.
3.4 THE PARADIGM OF SFFFCENCY--OIENTER INITIALIZATION
The latter phenomenon motivates us to develop an alternative perspective on goodness of neural field initialization, which may replace or be used jointly with the distribution preservation principle. More specifically, we consider an optimization problem
max speed(a)/speed[] 021
subject to (estLoss(a) (estLoss(1) S
(5)
where speed(a) denotes the training speed (e.g., a reciprocal the number of steps required) SNF when initialized with weight scaling, and TestLoss(a) denotes the loss the unseen coordinates when trained from the initialization. Given this objective, the key questions we try to address are:
Why does the weight scaling accelerate the training of SNFs?
See Section 4
Is weight scaling generally applicable for other data modalities?
See Section 5.1
How can we select a good a, without expensive per-datum tuning?
D See Section 5.2
UNDERSTANDING THE EFFECTS OF THE WEIGHT SCALING
We now take a closer look at the weight scaling to understand how it speeds up the training of sinusoidal neural fields. Before diving deeper into our analyses, we first discuss why conventional understanding does not fully explain the phenomenon we observed for the case of SNF.
Comparison with lazy training. At the first glance, is connected to the "lazy training" (Chizat et al., 2019; Woodworth et al., 2020): Scaling the initial weights (W + a W) amplifies the initial model functional (f + a' f) (Taheri et al., 202 1), which in turn leads to the model to be linearized; then, one can show that the SGD converges at an exponential rate, f.e., the fast training, with the trained parameters being very close to the initial one. However, there is significant gap between this theoretical understanding and our observations for the SNFs:
tempting to argue that this phenomenon
The proposed weight scaling does not amplify the initial functional, as sinusoidal activations are not positively homogeneous (i.e., (ar) * a (z)) even for positive z), unlike ReLU, In fact, WS keeps the last layer parameters unscaled, making it further from any functional amplification, and empirically works better than direct amplification of functional (Saratchandran et al., 2024b).
At a practical scale, naive scaling initial weights for ReLU networks leads to both high training and test losses, due to a high condition number (Chizat et al., 2019, Appendix C). In contrast, the training loss of the WS remains very small even at the practical tasks.
In light of these differences, we provide alternative theoretical and empirical analyses that can help explain why the WS accelerates training, unlike in ReLU nets. In particular, we reveal that:
WS SNFs has two distinctive effects from ReLU nets, in terms of the initial functional and the layerwise learning rate; the former the major contributor to the acceleration (Section 4.
At initialization, WS increases both (1) the frequency of each basis, and (2) the relative power of the high-frequency bases (Section 4.2).
WS also results in a better-conditioned optimization trajectory (Section 4.3).
4.1 TWo EFFECTS OF WEIGHT SCALING ON SINUSOIDAL NETWORKS
There are major differences in how the weight scaling impacts SNF training compared to its effect on conventional neural networks with
activation functions, e.g., ReLU. (1) Initial functional: Larger weights lead higher frequency (low-level) features in SNE, whereas
5

---

Published as a conference paper at ICLR 2025
sompeeorr Teet PON4
10
10
(a) Weight scaling (as whole) (b) Initial functional only
(c) Early layer gradients only
Figure 3. Decoupling the effects of weight scaling on SNE. We decouple the effect of weight scaling on how it amplifies the early layer gradients, from the effects of having higher-frequency initial functional. We observe that the initial functional itself plays the key role, resulting in much greater acceleration.
in ReLU nets scaling up does not change the neuronal directions (more discussions in Section 4.2). (2) Larger early layer gradients: Unlike ReLU nets, scaling up SNF weights has a disproportionate impact the layerwise gradients; earlier layers receive larger gradients (see remarks below).
Motivated by this insight, we compare which of these two elements play a dominant role in speeding up the SNF training by decoupling these elements (Fig. 3). In particular, we analyze how changing only the initial weights-will the layerwise learning rates that are scaled down to match the rates of unscaled SNF-affects the gpeed-generalization tradeoff; we also compare with the version where the initial function stays the same default, but the learning rates on early layers are amplified. From the results, we observe that the effect of having a good initial functional, with rich high- frequency functions, play dominant role on speeding up the SNF training. The disproportionate learning rate also provides a mild speedup, while requiring less sacrifice the generalizability.
How are the early layer gradients amplified? Essentially, this is also due to the lack of positive- homogeneity the activation function. To see this, consider a toy two-layer neural net with width one f(x) (((2) (2) The loss gradient for each layer weight is proportional to the layerwise gradients of the function, which is W (w (w())x), (w)))))). For ReLU activations, both entries of V grow linearly we increase the scale of both weights. For sin activations, on the other hand, the first entry scales linearly, but the scale of the second entry remains the same. Following a similar line of reasoning, we can show that the first layer of a depth-/ sinusoidal network has a gradient that scales as a', while the last layer gradient scales as 1. We provide a more detailed discussion on the a-dependency of layerwise gradients in Appendix B.3.
4.2 FREQUENCY SPECTRUM OF THE INITIAL FUNCTIONAL
We have seen that the initial functional has a profound impact on the training speed, but how does the weight scaling affect the functional? The answer is easy for two-layer nets; the weight scaling amplifies the frequency of each hidden layer neuron, enabling the model to cover wider frequency range with hardly updating the features generated by the first layer.
How about for deeper networks? Our theoretical and empirical analyses reveal that, for deeper SNFs, the weight scaling also increases the relative power of the higher-frequena bases. Thus, the weight-scaled networks may suffer less from the spectral bias, and can easily represent signals with large high-frequency components (Rahaman et al., 2019).
Theoretical analysis. We show theoretically that the weight scaling increases the relative power of the nigher-frequene bases. For simplicity, consider again a three-layer bias-free SNF
f(;; W) (3) (2) sin(W(Dx))).
(6)
with sinusoidal activation functions. Then, it can be shown that the overall frequency spectrum of the function f(: W) can be re-expressed as the multiples of the first layer frequencies, via Fourier series expansion and the Jacobi-Anger identity (Yūce et al., 2022).
Lemma 2 (Corollary of Yüce et al. (2022)). The network Eq. (6) can be rewritten as
f(x; W) = 2w (3)
Σ J(((()) (i)((W
(7)
BE
where Je(.) denotes the Bessel function (of the first kind) with order &.

---

Published as a conference paper at ICLR 2025
Given the Bessel form (7), let proceed to analyze the effect of weight scaling. By scaling the first layer weight, i.e., W(1) + aW(()) we are increasing the frequencies of each sinusoidal basis by the factor of a. That is, each frequency basis becomes a
sinusoid.
Scaling the second layer weight, on the other hand, acts by affecting the magnitude of each sinu- soidal basis, i.e., Jz(W(()). For this quantity, we can show that the rate that the Bessel function coefficients evolve as we consider higher order harmonies (i.e., larger E) scales at the speed pro- portional to a². That is, the relative power of the higher-order harmonics becomes much greater as we consider larger scaling factor a. To formalize this point, let us further simplify the model (6) to have width one. Then, we can prove the following sandwich bound:
Lemma (Scaling of harmonics). For any nonzero W(2) E (-x/2, 7/2), we have:
(W(2))2
(e+2(W(2))
(W(2))2
(2C + 2) 4) Je(W(2))
(2E + () +3)
In other words, replacing Wx(2) + a ((2) increases the growth rate Je+2/J Jr by a². We note that there exists a restriction in the range W(2) which prevents from considering very large a. However, empirically, we observe that such amplification indeed takes place for the practical ranges of a where the generalization performance remains similar.
Summing up. the weight scaling has two effects on SNFs with more than two layers. First, the WS increases the fre- quencies of each sinusoidal basis by scaling the first layer weights. Second, the WS increases the relative weight of higher-frequency sinusoidal bases, by scaling the weight matrices of the intermediate layers.
5.0
15
10-
Empirical analysis. To demonstrate that the increase of coefficients in the higher-order harmonics takes place in- deed, we conduct spectral analysis of an initialized SNF for natural image (Fig. 4). From the plot, we observe that the initialized SNFs with larger scaling factor has richer higher frequency spectrum. Moreover, the decay rate of the higher-frequency components, as we consider increase the frequency level, is much smaller for larger or as a consequence, weight-scaled SNFs tend have non-zero coefficients for very high frequency ranges, enabling a faster fitting to the natural audio signals (gray) with rich high- figh-frequency components. Another
10
100
20
110
Figure 4. Spectrum of initialized SNFs. ID-FFT of an initialized 5-layer SNF, with various levels of weight scaling factors.
4.3 OPTIMIZATION TRAJECTORIES OF WEIGHT-SCALED NETWORKS
Now, we study the optimization properties of SNFs through the lens of the empirical neural tangent kernel (cNTK). We empirically analyze two properties of the eNTK cigenspectrum of the SNF, which are relevant to the training speed: (1) condition number, and (2) kernel-task alignment.
Condition number. Roughly, the condition number denotes the ratio Amax/ Ammin of the largest and the smallest eigenvalues of the eNTK; this quantity determines the exponent of the exponential con- vergence rates of the kernelized models, with larger condition number meaning the slower training (Chizat et al., 2019). Fig. 5a, we plot how the condition numbers of the eNTK evolve during the training for SNFs, under weight scaling or frequency tuning; for numerical stability, we take an aver- age of top-5 and bottom-5 eigenvalues and take their ratio. We observe that the weight-scaled SNFs tend to have much smaller condition number at initialization, which becomes even smaller during training. On the other hand, the SNF with the standard initialization suffers from a high condition number, which gets even higher during training. More discussion be found in Appendix C.3.
We note that this behavior is in stark contrast with ReLU neural networks, where the weight scaling leads to a higher condition number, leading to a poor convergence (Chizat al., 2019).
Kernel-task alignment. The kernel-task alignment-also known as the energy concentration- is the cumulative sum of the (normalized) dot products
E(t)=
Σ
(0) /eel2
(9)
7

---

Published as a conference paper at ICLR 2025
Log (Conditio Number)
17.5
Aligmment
0.6
12.5
0.4
7.5
0.2-
Standard Freaserey (10x)
5.0
Scalinn
1000 1400
Iterations
Normaliedd Elgenvalues
(a) Condition number
(b) Kernel-task alignment
Figure 5. Eigenanalyses with eNTK. (a) Weight scaling improves the conditioning of the SNF optimization all SGD steps, greatly reducing the condition number. (b) Weight-scaled SNF enjoys better kernel-task alignment throughout the training; darker lines indicate later itcrations.
Table 1.
Weight scaling in various data domains. We compare the training speed of the weight-scaled SNF against other baselines various data domains. To evaluate the training speed, train for fixed number steps and compare the training loss achieved. Bold denotes hest option, and underlined denotes the runner-up. We have experimented with five random seeds, and report the mean and the standard deviation.
Image (PSNR)
Occ. Field (loU)
Spherical (PSNR)
Andio (PSNR)
Activation
KODAK
DIV2K
Lucy
ERAS
Bach
Uniform Bengio, 2010)
Sinusoidal
0.39+0.10
0.0000-0.000
0.22
ReLU P.E. (Mildenhull et al,, 2020)
ReLUU
18.6010.08
16.72-00.08
0.9896_00.0003
33.30.10.54
24.98.10.19
FFN (Tancik 2020)
ReI.I
20.52-00.6
19.81-0.48
0.9843-0.0020
38.69-0.27
16.66+00.28
SIREN init. (Sitzmann et al., 2020b)
Sinusoidal
24.58.10.05
22.86-00.05
0.992510.0000
38.72-10.07
37.37.1..11
GaussNet (Ramasinghe & Lucey, 2022)
Gaussion
21.94-2.48
19,22-0.144
0.9914-0
38.56-0.51
27.47+22100
(Fathony al.,, 2021)
Wavelet
0.9847
WIRE (Saragidam al., 2023)
Wavelet
28.9410.21
28.20-00.13
0.9912=000005
31.2710.53
16.83-1:.885
NFSL. (Saratchamärun
20248)
Sinusoidal
24.93-00.07
23.39=0.09
0.9925 0.0001
38.92-0.07
37.17442.88
Sinc (Saratchundran et al., 2024c)
Sine
27.73.10.27
26.42-00.24
0.9936.100000
36.15.10.51
23.03-10.40
Weight scaling (ours)
Sinusoidal
42.03 0.44
0.9941 0.,002
45.28- 00,03
45.04+3.23
where d; denotes the i-th eigenvector of the kernel and e = f(x((x denotes the vector of prediction errors in the pixels (Kopitkov & Indelman, 2020). This quantity measures how much of the residual signal is concentrated in the directions of large eigenvalues, which is theoretically easier be expressed optimized by kernel learning (Baratin et al., 2021; Yüce et al., 2022). In Fig. 5b, we compare the kernel-task alignments during the SNF training trajectories. We observe that the weight scaling gives a highly optimizable kernel throughout all stages of training.
EXPERIMENTS
In this section, first address whether the weight scaling effective in other data domains (Sec- tion 5.1); our answer is positive. Then, we discuss the factors that determine the optimal value of scaling factor for the given target task (Section 5.2); we find that the optimal value does not depend much on the nature of each datum, but rather relies on the structural properties of the workload.
5.1 MAIN EXPERIMENTS
We validate effectiveness of WS in different data domains by comparing against various neural fields. To compare the training speed, we compare the training accuracy for equivalent steps. In par- ticular, we consider the following tasks and basclines. Other details can be found in Appendix F.4.
Task: Image regression. The network is trained approximate the signal intensity c, for each given normalized 2D pixel coordinates (x, y). For our experiments, we use Kodak (Kodak, 1999) and DIV2K (Agustsson & Timofte, 2017) datasets. Each image is resized in grayscale, following Lindell et al. (2022); Seo et al. (2024). We report the training PSNR after a full-batch training for 150 iterations. For all NFs,
resolution of 512
five layers with width 512.
Task: Occupancy field. The network trained to approximate the occupancy field of a 3D shape, ie., predict '1' for occupied coordinates and '0' for empty space. We use the voxel grid of size 512x
8

---

Published as a conference paper at ICLR 2025
512 x 512 following Saragadam et al. (2023). For evaluation, measure the training intersection- over-union (IoU) after 50 iterations on the 'Lucy' data from the 'Standard 3D Scanning Repository,' with the batch size 100k. We use NFs with five layers and width 256.
Spherical data. We use 10 randomly selected samples from the ERA5 dataset, which con- tains temperature values corresponding to grid of latitude & and longitude 6, using the geographic coordinate system (GCS). Following Dupont et al. (2022), the network inputs are lifted to 3D coor- dinates, i.e., transforming (6. 0) to (cos(Ø) cos (0), cos(o) sin(O), sin())). We report the training PSNR after 5k iterations of full-batch training. For NFs, we use five layers with width 256.
Task:
Task: Audio data. Audio is 1D temporal signal, and the network is trained to approximate the amplitude of the audio at given timestamp. We use the first 7 seconds of "Bach's Cello Suite No. 1," following Sitzmann al. (2020b). We report the training PSNR after Ik iterations of full-batch training. For NFs, we use five layers with width 256.
Baselines. We compare the weight scaling against seven baselines; we have selected the versatile neural field methods, rather than highly specialized and heavy ones, such as Instant-NGP (Müller et al., 2022). More concretely, we use the following baselines: (1) Xavier uniform: the method by (Glorot & Bengio, 2010) applied on SNF, (2) SIREN init.: SNF using the standard initialization of SIREN (Sitzmann et al., 2020b),
(3) NFSL: A recent initialization scheme motivated by scaling law (Saratchandran et al., 2024b). (4) ReLU++.E.: ReLU nets with positional encodings (Mildenhall et al., 2020). (5) FFN: Fourier feature networks (Tancik et al., 2020). (6) MFN: multiplicative filter networks (Fathony
al., 2021). (7) GaussNer: MLP with Gaussian activations (Ramasinghe & Lucey, 2022). (8) WIRE: MLP with Gabor anvelet-based activations (Saragadam et al., 2023). (9) Sinc NF: MLP with sinc (i.e., sin(wz)) (wr) activations (Saratchandran et al., 2024c). The baselines (1-3) use SNFs, while (4-9) uses other architectures with different activation functions.
Result. Table 1 reports the comprehensive results across various data domains. We observe that the weight scaling significantly outperforms all baselines throughout all tasks considered. In Ap- pendix F.8. we provide further qualitative results on various modalities, where we find that the weight scaling enhances capturing the high frequency details. These results support our hypoth- esis that the richer frequency spectrum provided by the weight scaling mitigates the spectral bias (Section 4.2), thereby enabling a faster training of neural fields.
Additional experiments. We have also conducted experiments on the NF-based dataset construc- tion tasks (Papa et al., 2024), novel view synthesis via neural radiance fields (NeRFs) (Mildenhall et al., 2020), and solving partial differential equations. We report the results and details for each experiment in Appendix F.5, Appendix F.6, and Appendix F.7 respectively.
5.2 ON SELECTING THE OPTIMAL SCALING FACTOR
Now, we consider the problem of selecting an appropriate scaling factor without having to carefully tune it for each given datum. In particular, we interested in finding the right that maximizes the training speed yet incurs only negligible degradations in generalizability. More concretely, consider the task of Kodak image regression and seek to select the a that approximately solves
maximize a
sa)
subject to TestPSNR(a) 2 0.95 TestPSNR(1).
(10)
where (again) the speed is the reciprocal of the number of steps required until the model meets the desired level of training loss, which we set to be the PSNR 50dB. Note that this optimization is similar to what has been discussed in Section 3.4.
In a nutshell, we observe in this subsection that this optimal scaling factor is largely driven by the structural properties of the optimization workload (such as model size and data resolution), and remains relatively constant over the data. This suggests that the strategy of tuning a on small number of data and transferring it to other data in the same datasets can be quite effective.
Structural properties. In Fig. 6, we visualize how changing the data resolution (512 x 512 by default), model width (512 by default), and model depth (4 by default) affects the value of optimal scaling factor. We observe that the higher data resolution and shallower model depth lead to a need for higher scaling factor, with the width having very weak correlation. Commonly, resolution and depth are both intimately related to the frequency spectrum: higher resolution leads to need for higher frequency, and deeper networks can effectively utilize the relatively smaller
find that the
9

---

Published as a conference paper at ICLR 2025
Optra
2.2
1.22
18
14-
15-
500
(350 ongtt Image
#ofttoceer Untts
of FEDdee
(a) Data resolution
(b) Model width
(c) Model depth
Figure 6. Optimal scaling factor vs. structural properties of the optimization workload. The data resolu- tion, model width, and model depth have positive, no, and negative correlations with the optimal scaling factor.
40
Acsoleraton
Text
10
(a) Acceleration
(b) Generalization
Figure 7. Scaling factor vs. the first 8 Kodak images. The behavior of cach datum with respect to the scaling factor quite similar, in both acceleration and generalization performances.
scaling factors to represent high-frequent signals. In a similar context, we provide an analysis of the impact of the learning rate in Appendix E.3.
Data instances. In Fig. 7, we observe that both curves are strikingly quite similar shape, having saturation and inflection points the similar scaling factor. This may be due the fact that natural images tend to exhibit similar frequency distribution.
CONCLUSION
In this paper, we revisit the traditional initialization scheme of sinusoidal neural fields, and propose method called weight scaling (WS), which scales the initialized weights by a specific factor. In practice, WS significantly accelerates the training speed of SNF across various data modalities, even outperforming recent neural field methods. We have discovered that the difference between the traditional and WS initialization comes from both the change in the frequency spectrum of the initial functional and the change the layerwise learning rates. Through extensive theoretical and empirical analyses, we have revealed that the WS positively impacts the learning dynamics by maintaining a well-conditioned optimization path throughout the training process.
Limitations and future directions. The limitations of our work lie in the fact that the analysis has been conducted for only neural fields using sinusoidal activation functions, rather than general neural field architectures: how the WS affects other architectures remains open question (see Appendix E.2). Also, from a theoretical perspective, our analyses does not provide a definitive an- swer the acceleration mechanisms, lacking explicit convergence bound. As final remark, work is closely related to the implicit bias of initialization scale (Azulay et al., 2021), which is an emerging topic in understanding the neural network training. However, there remains significant gap between theory and practice in this area, as the current interpretation is limited to shallow and homogeneous (or linear) networks (Varre al., 2023; Kunin et al., 2024). The step is to under- stand the implicit bias in networks with general activation functions, where the scale of initialization not necessarily equivalent output scaling. Another research direction is understand the ef- fect of weight scaling in general neural field architectures (Appendix F.2), which can sometimes improve performance or have the opposite effect. We leave these for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00213770),, in part by the Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea govern- ment (MSIT) (2022-0-00713), and in part by the NAVER-Intel Co-Lab; the work was conducted by POSTECH and reviewed by both NAVER and Intel.
REFERENCES
Eirikur Agustsson and Radu Timofte. NTIRE 2017 challenge on single image super-resolution: Dataset and study. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
Maor Ashkemazi and Eran Treister. Towards croppable implicit neural representations, In Advances in Neural Information Processing Systems, 2024.
Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning. 2021.
Aristide Baratin, Thomas George, César Laurent, Devon Hjelm, Guillaume Lajoie, Pascal Vin- cent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In Interna- tional Conference on Artificial Intelligence and Statistics, 2021.
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In Advances Neural Information Pro- cessing Systems, 2019.
Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, 2020.
Shannon R Bowling, Mohammad T Khasawneh, Sittichai Kaewkuckool, and Byung Rae Cho. A logistic approximation to the cumulative normal distribution. Journal of Industrial Engineering and Management, 2009.
Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. In Proceedings the Thirtieth International Joint Conference Artificial Intelligence, 2021.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
Yinbo Chen and Xiaolong Wang. Transformers as meta- leanners for implicit neural representations. In European Conference Computer Vision, 2022.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, 2019.
Moulik Choraria, Leello Tadesse Dadi, Grigorios Chrysos, Julien Mairal, and Volkan Cevher. The spectral bias of polynomial neural networks. In International Conference Learning Represen- tations, 2022.
Yishum Dou, Zhong Zheng, Qiaoqiao Jin, and Bingbing Ni. Multiplicative fourier level of detail. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, 2023.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Amaud Doucet. COIN++: Neural compression across modalities. Transactions on Machine Learning Research, 2022.
11

---

Published as a conference paper at ICLR 2025
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicatiiv filter networks. In International Conference on Learning Representations, 2021.
Michael S Gashler and Stephen C Ashmore. Training deep fourier neural networks to fit time-series data. In Intelligent Computing in Bioinformatics, 2014.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via Hessian eigenvalue density. In International Conference Machine Learning, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
Daniele Grattarola and Pierre Vandergheynst. Generalised implicit neural representations. In Ad- vances Neural Information Processing Systems, 2022.
Zongyu Guo, Gergely Flamich, Jiajun He, Zhibo Chen, and José Miguel Hernández-Lobato. Com- pression with Bayesian implicit neural representations. In Advances in Neural Information Pro- cessing Systems, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2015.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge University Press, 2012.
Evangelos K. Ifantis and Panayiotis D. Siafarikas. Inequalities involving Bessel and modified Bessel functions. Journal of mathematical analysis and applications, 1990.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, and Simon Lucey. Efficient learning with aite-activated low-rank matrices. In International Conference on Learning Repre- sentations, 2025.
Hyomin Kim, Yunhui Jang, Jaeho Lee, and Sungsoo Ahn. Hybrid neural representations for spheri- cal data. In International Conference on Machine Learning, 2024.
Diederik P Kingma. Adam: A method for stochastic optimization. In International Conference on Learning Reprexentations, 2015.
E. Kodak. Kodak dataset, 1999. URL httts::/t/tkkktt. w.tokkkus//raaa U.L.ttpps///bb...//rppppps////a
Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In Artificial Neural Networks and Machine Learning, 2020.
Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: Exact solutions reveal how unbalanced initializations promote rapid feature learning. arXiv preprint 2xXiv:2406.06/55 2024.
Jacho Lee, Jihoon Tack, Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. Advances in Neural Information Processing Systems, 2021.
David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. BACON: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2022.
Chaoyue Liu and Like Hui. Relu soothes the ntk condition number and accelerates optimization for wide neural networks. arXiv preprint arXiv:2305. 08000.0.813.... 2023.
Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. FINER:
Flexible spectral-bias tuning in implicit neural representation by auriable-priiofi acti- vation functions. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
12

---

Published as a conference paper at ICLR 2025
Qi Ma, Danda Pani Paudel, Ender Konukoglu, and Luc Van Gool. Implicit Zoo: A large- scale dataset of neural implicit functions for 2D images and 3D scenes. arXiv preprint 24X1v:2406..7438, 2024.
ZoplicitZZoo:
Harsh Mehta, Ashok Cutkosky, and Behnam Neyshabur. Extreme memorization via scale of initial- ization. In International Conference on Learning Representations, 2021.
Lassi Meronen, Martin Trapp, and Arn Solin. Periodic activation functions induce stationarity. In Advances in Neural Information Processing Systems, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. European Conference Computer Vision, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. ACM transactions on graphics (TOG), 2022.
Yurii Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in Python. In International Conference on Learning Representations, 2020.
Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfoode and Pascal Frossard. What can lin- earized neural networks actually say about generalization? In Advances in Neural Information Processing Systems, 2021.
Sourav Pal, Harshavardhan Adepu, Clinton Wang, Polina Golland, and Vikas Singh. Implicit repre- sentations via operator learning. In International Conference Machine Learning, 2024.
Samuele Papa, Riccardo Valperga, David Knigge, Miltiadis Kofinas, Phillip Lippe, Jan-Jakob Sonke, and Efstratios Gavves. How to train neural field representations: A comprehensive study and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2024.
Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In International Conference Learning Representations, 2023.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, 2007.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for activations in coo. In European Conference on Computer Vision, 2022.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. Challenges in training pinns: loss landscape perspective. In International Conference on Machine Learning, 2024.
Daniel Rebain, Soroosh Yazdani. Kwang Moo Yi, and Andrea Tagliasacchi. Neural fields distri- butions: Signal processing beyond Euclidean space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Mare RuBwurm, Konstantin Klemmer, Esther Rolf, Robin Zbinden, and Devis Tuia. Geographic location encoding with spherical harmonics and sinusoidal representation networks. Interna- fional Conference on Learning Representations, 2024.
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard Baraniuk. WIRE: Wavelet implicit neural representations. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2023.
13

---

Published as a conference paper at ICLR 2025
Hemanth Saratchandran, Shin-Fang Chng, and Simon Lucey. Analyzing the neural tangent kernel of periodically activated coordinate networks. arXiv preprint arXiv:2402.04783. 2024a.
Hemanth Saratchandran, Sameera Ramasinghe, and Simon Lucey. From activation to initialization: Scaling insights for optimizing neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, and Simon Lucey. sampling theory perspective on activations for implicit neural representations. In Inter- national Conference on Machine Learning, 2024c.
Jonathan Richard Schwarz, Jihoon Tack, Yee Whye Teh, Jaeho Lee, and Jinwoo Shin. Modality- agnostic variational compression of implicit neural representations. International Conference Machine Learning. 2023.
Junwon Seo, Sangyoon Lee, Kwang In Kim, and Jaeho Lee. In search of a data transformation that accelerates neural field training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Louis Serrano, Lise Le Boudec, Armand Kassai Koupai, Thomas X Wang. Yuan Yin, Jean-Noël Vittaut, and Patrick Gallinari. Operator learning with neural fields: Tackling PDEs on general geometries. In Advances in Neural Information Processing Systems, 2023.
Zenglin Shi, Pascal Mettes, Subhransu Maji, and Cees G M Snoek. On measuring and controlling the spectral bias of the deep image prior. International Journal of Computer Vision, 2022.
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: SE(3)eqquvvarian object rep- resentations for manipulation. In International Conference on Robotics and Automation, 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous 3D-structure-avw aware neural scene representations. In Advances in Neural Information Pro- cessing Systems, 2019.
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. MetaSDF: Meta-leaming signed distance functions. In Advances in Neural Information Processing Systems, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, 2020b.
Mahsa Taheri, Fang Xie, and Johannes Lederer. Statistical guarantees for regularized neural net- works. Neural Networks, 142: 161, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Pro- cessing Systems, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-base based neural representa- tions. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2021.
Terence Tao. Topics in random matrix theory, volume 132. American Mathematical Soc., 2012.
Chayne Thrash, Reed Andreas, Ali Abbasi, Parsa Nooralinejad, Soroush Abbasi Koohpayegani, Hamed Pirsiavash, and Soheil Kolouri. MCNC: Manifold-constraineed reparameterization for neural compression. In International Conference on Learning Representations, 2025.
Gal Vardi. On the implicit bias in deep-learning algorithms. Communications of the ACM, 2023.
14

---

Published as a conference paper at ICLR 2025
Aditya Vardhan Varre, Maria-Luiza Vladarean, Loucas Pillaud- Vivien, and Nicolas Flammarion. On the spectral bias of two-layer linear networks. In Advances in Neural Information Processing Systems, 2023.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of Fourier feature net- works: From regression to solving multi-scale PDEs with physics-informed neural networks. Computer Methods Applied Mechanics and Engineering, 2021.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, 2020.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of cnns: How train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, 2018.
Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and general- ization deep neural networks. In International Conference on Machine Learning, pp. 10462- 10472. PMLR, 2020.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, 2022.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 2020.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. PyHessian: Neural networks through the lens of the Hessian. In JEEE international conference on big data, 2020.
Gizem Yúce, Guillermo Ortiz-Jiménn Beril Besbinar, and Pascal Frossard. A structured dictionary perspective on implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization, In International Conference on Learning Representations, 2019.
15

---

Published as a conference paper at ICLR 2025
Appendix
Detailed statement and the derivation of Proposition
17
A.1 Proof of Lemma 4
17
A.2 Proof of Lemma
17
B Proofs in Section 4
21
B.1 Proof of Lemma
B.2 Proof of Lemma
21
B.3 Details of Section 4.1
Details of NTK Analysis and beyond
23
C.1
Preliminaries
23
C.2 Kernel regression with gradient flow
................................
23
C.3 Insights into eNTK eigenvalue
25
C.4 Non-kernel approach: analysis via Hessian eigenvalues
................................
27
D Frequency spectrum of 2D images
28
More discussion
E.1 The effect of weight scaling in mini-batch training
29
................................
E,2 On the effect of weight scaling on other activation functions
................................
E.3 The optimal scaling factor according to the learning rate
31
Experimental details and qualitative results
32
F.1
Experimental details about Fig. 2b
32
F2
Experimental details about Fig. Experimental details about Eq. (10)
32
F3
................................
32
F44
Training settings and baselines
................................
32
F.5
Neural dataset experiments
34
F66
Neural radiance fields experiments
35
F.7
Solving differential equations with partial observations
36
F8
Qualitative results
37
F.9
Training for significantly larger iterations
43
16

---

Published as a conference paper at ICLR 2025
A DETAILED STATEMENT AND THE DERIVATION OF PROPOSITION
In this section, we provide more detailed statement and derivation of Proposition where we have extended the iatributioonpreserviii claims of Sitzmann al. (2020b) on 1 to the cases of 2 1, thus covering the initialization with weight scaling.
We note that we generally follow the approach of Sitzmann et al. (2020b) in terms of the formalisms; we provide rough approximation guarantees (Lemmas 4 and 5), one which relies on a numerical analysis of function that is difficult to be expressed in a closed form (Lemma 5).
In particular, we are concerned with showing the following two points:
Activations to pre-activations. We show that, for > 1, the pre-activation of a layer with weight-scaled uniform activations and aresin input distributions are approximately distributed as Gaussian distribution with variance scaled by (Lemma 4).
Pre-activations to activations. Then, we show that, for 2 post-activation distribution of the Gaussian distribution with o-scaled variance is distributed as an arcsin distribution (Lemma 5).
More formally, our lemma are follows.
Lemma 4 (Arcsin to Gaussian distributions). Let the weights Wi of the 1-th layer be sampled i.dd from Unif(-c/ Vn, c/ VT), and the activation of the (I - 1)-th layer Z1-1 follow the i.i.d. arcsin(-1, Then, the distribution of the pre-activation of the 1-th layer Yi converges in distribu- tion Gaussian distribution N(0, e² /6) the number of hidden units n tends to infinity.
Lemma 5 (Gaussian to aresin distributions, with numerical proof). Let the pre-activation distri- bution X has Gaussian distribution (N a²) for some 1. Then the activation distribution () is distributed as arcsin(- 1,1).
Note that, by plugging in c av6 to the Lemma 4 as in the proposed weight scaling, we get the normal distribution N (0. a²) as an output distribution.
We provide the proofs of these lemma Appendices A.1 and A.2, respectively. For all derivations, we simply ignore the issue of the frequency scaling term wo, and simply let it equal to 1.
A. PROOF OF LEMMA 4
For simplicity, let us denote a pre-activation of a neuron in the Ith layer by y. Let the weight vector connected to this output neuron as w R", with each entries drawn independently from Unif(-c/ /n, c/ /n). Also, let the corresponding input activation from the (1 - 1)th layer be de- noted by 2 E IR" with each entry drawn independently from arcsin(-1, 1). Now, it suffices to show that y = w z converges in distribution to (( c² /6) as n goes to infinity.
To show this point, we simply invoke the central limit theorem for the i.i.d. case. For CLT to hold, we only need to check that each summand (w4z) of the dot product w z Σ wis; has a finite mean and variance.
W z
(11)
has a finite mean and variance T z has zero mean and a finite variance. We know that [[ = 0. as two random variables are independent and E[w:]] = 0. To show that the variance is finite, we can proceed as
21 2 6n
2 (ar(w,)Var(()) 3n
(12)
where the first equality holds as each variables are independent and zero-mean. Summing indices, we get what we want.
A.2 PROOF OF LEMMA 5
By the 30 rule, 99.7% of the mass of the Gaussian distribution (/aiibbuttioooOO....... (0, a²) lies within the finite support 1-3,, 3o.) The CDF of Y can be written as
Fv P(Y < y) P(sin(X) v)-
(13)
17

---

Published as a conference paper at ICLR 2025
Case
Case 2
Case
Figure Visualization the cases Lemma 5. = sin(æ:) line 3a Black line x
Different from the settings of Sitzmann al. (2020b), the sinusoidal activation not a bijection in
general within the support [-3ox, 3a), i.e. not directly invertible. Thus, we break it down into three cases (Fig. 8); Case 1, where the endpoint 3a less than π 7/2, Case 2. where 3a is in the increasing region of sin(-), and Case where 3a is in the decreasing region. We study each case:
Case (a < π 7/6). Here, we have
1 Fv-(v) = P(X < arcsin v) Fx (arcsin y) ~ 2 2 tanh (2 arcciny)
(14)
where the last approximation invokes the Gaussian CDF approximation (Bowling et al., 2009). In this case, the CDF Y does not approximate the CDF of arcsin(-1, 1, That is, the second-order derivative of F decreases in [-1, 0] and increases in [0, 1J. while the CDF of the arcsin distribution behaves in an opposite way.
Now, suppose that sufficiently large (a 2 7/6), i.e., case 2 or case 3. Here, will consider the nearest peak/valley that falls outside the [-3a, 3a] interval. More concretely, let y be defined as the period index such peak/valley, i.e.,
7 = min n 72);
where
(15)
(4n + 1)TT
(4n 1)TT
71 min
2
30
=+1
(peak; case
(16)
(4n 1)x
(4n
1)T
72 min
n E
3x
sin
(valley; case 3)
(17)
2
By considering the range up until such peaks/valleys, we can ensure that the probability of lying within this range is greater than 0.997.
Now, let's proceed to analyze the case 2; the case can be handled similarly.
Case 2. We break down the CDF of Y
Fy (2) Fy (v) Fy;ine (y.dec
(18)
Here, Fy.oemer(y) denotes the CDF of the bijective region, i.e., E [-T/2, x x/2], and Fy.inc (v) denotes the region where the sin(-) is increasing and (y) denotes where sin(.) is decreasing. Then, each of these components can be written as²:
Fy, (v) = P(- X < arcsin y) = Fx (arcsin v) Fx(-m/22
(19)
Fy.ie =
Fx (arcsin - A - Fx ((4n 1)TT/2)
+ Fx (arcsin y 4n (2) Fx (-(4n + 1)mr/2]]
(20)
Fy.dce (y) =
Fx ((4n 1)mT/) - Fx aresin y + (4n - 2)xx22)
Fx (4m 3)m Fx arcsin y (4n /2/]
(21)
For notational brevity,
just for moment. It will appear again in Eq. (23).
18

---

Published as a conference paper at ICLR 2025
3.6
#phal1
Intogral of Offizens
0.3
400
500
Figure 9. Approximated CDF with increasing a. line indicates the reference error. When a goes larger, the error converges to limit near zero.
Combining these terms, get:
(y) = Fx (aresin y) - Fx (-T/2)
+
Fx (4)-14433)772 - Fx(-(4n + 1)m/2)
Fx (arcsin y 4nm/2) Fx (arcsin y 4n 22)
Fx aresin y (4n 2) 22) Fx aresin y - -4n - 2)x/2))
(22)
Using the logistic approximation and Taylor's 's first-order expansion at = arcsin y i.e., f(2) 2 f'(=) (- 0) f(0), this can be further approximated by
8 NO(
71
4n,8T
(4n - 2)
(v)
+
Σ
h'
+ h
2a
a
2a
2a
n=1
1
BT
m
1
(4n
3)
1 h
(4n + 1)
h
h
(23)
2
2a
2a
2
2a
where h is the tanh function. The above equation has variables 71 and a. Thus, we can rewrite it
as:
(v f(a)
arcssn y + g(a),
(24)
B (0) +
8 71
4n,3T
(4n - 2) (4)
].
f(x)=
Σ
h
+ h1
(25)
2a
2a
2a
==1
1
BT
(4n
3)
(4n + 1)5TT
g(a)
h
h
- h
(26)
2
2a
2
2a
20x
Now, we compare the Fx (x) to ground truth CDF of aresin(-1, 11, 1), iee,, 2 arcsin closed form solution is not obtainable, we numerically compute the difference in integration over input domain:
s+1 As the
2 S Fx(0,)) arcsin
(v=r
x +
ā E
2 dx
(27)
1- Fx, aresin
2
1
dx
(28)
π
19

---

Published as a conference paper at ICLR 2025
This indicates that the RHS term represents the cumulative error from the default initialization, which is assumed to be valid for preserving the forward signal. If the LHS is smaller than the RHS for some value of å that satisfies the inequality. intuitively follows that the elements in å provide a valid scale. In Fig. 9, we empirically demonstrate that as the scaling factor a increases, the error between the ground truth and the approximated CDF decreases with some limit near zero, which implies â E (o).
20

---

Published as a conference paper at ICLR 2025
B PROOFS IN SECTION 4
B.1 PROOF OF LEMMA 2
To formally prove this lemma, we first bring the generalized form from Yúce al. (2022).
Lemma 6 (From Yúce al. (2022, Appendix A.2)). Suppose that the model be written f() W(() sin (W12) sin(W())z)). where W()) E Rd,, W(2) E Rdaxds, and W E R 1idd
Then, the model can be equivalently written as:
d2-1
d -1
d 11
f(z)= - Σ
Σ
II k=
Jk
(wu (2))
(3( sin
Σ A=0
in
(1) w
(29)
m== 61. Sh1
=-
where the lowercase w (i) denotes the k-th column of the matrix W(()
We specialize this lemma to the case of width-one SNE, i.e., d1 d2 1. We proceed as
f(x) W(3) sin(W(2) sin(W(xx))
(30)
(3)
J28+11 J28+1(W((2) sin((2k + 1))(
(31)
k=
where the second equality due to the Jacobi-Anger identity. By restricting the sum to I E Zodd. can further simplify
Mæ; W) = (3(3)
Σ
(32)
EE
B.2 PROOF LEMMA
For notational simplicity, we denote == W(2)
First, we note that the first root of the 9th order Bessel function, denoted by je,1, satisfies the in- equality je,1 > 2 for all L E N. Next, recall the results of Ifantis & Siafarikas (1990) that
JE+1(E) Je(x)
<
Ve E 10,)
(33)
2(e + 1)
28 21+1'
Similarly, for Jx))(x), we have
x
Je+2(x)) <
x 28 3
Vr E (0, 1/2).
(34)
2( + 2) Multiplying two inequalities, get the desired claim.
Je+1(z)
B.3 DETAILS OF SECTION 4.1
In this subsection, provide more precise details of gradient bound of SNFs, with simple L-layer, 1-width network. Note that the bound can be generalized to arbitrary width, with matrix product computation.
Let L-layer simple SNE, f(x), as in the main paper:
(xx = (L)) sin (p))
sin)W))).....)))
(35)
The network is trained with MSE loss function with gradient-basee based method, then the gradient of weight of each layer is computed as:
aL aL 0f @W
Of of OW()) Ro (W
for e [],
(36)
where Ro denotes the initial residual, i.e., f(x) Yime- We additionally assume that Ro in- dependent from various initalizations techniques. Then, as (5/aW(k only effects to gradients. By
21

---

Published as a conference paper at ICLR 2025
the simple property of sine function: max, sin(x) (dx,(dsin())ddx) 1 the gradients bounded with:
OL OW(R)
R
II m=1
L (L2eW(L-mtt) Gx(z)= zh-k-1 + 1, Gx(x))
(37)
L-k
R II
1 (L>xW((-mtt)) max Gk(l)) x4-k-1 l max Gx(x))
(38)
1
--
R II
1 x + LL=
(39)
m=l
L-k
R
II m=1
(1xteW(-mmt)) + +1==k)
(40)
where Gs(x)) denotes composite sinusoidal function from the gradient derivation process and is a indicator function. Since the maximum value of a composite sinusoidal function always equal to or less than 1, [=)(z)] is bounded by 1. Note that Eq. (40) is satisfied due to the bounds input domain, i.e., x E [-1,]]
22

---

Published as a conference paper at ICLR 2025
C DETAILS OF NTK ANALYSIS AND BEYOND
C.I PRELIMINARIES
The Neural Tangent Kernel (NTK) framework views neural networks as kernel machines, where the kernel is defined as the dot- pro-ppr of the gradients from two different data point (Jacot et al., 2018).
In other words, each element of NTK can be expressed as [ 2ty) (Vw f (z;), f ))- Assuming an infinitely wide network trained with an infinitesimally small learning rate (i.e., gradient flow) and using specific parameterization (i.e. NTK parameterization), the NTK converges deterministic kernel that does not evolve during training. However, in practice, neural networks are composed of finite-width layers, which introduces challenges for analysis using NTK-style theory. In this context, we study the empirical Neural Tangent Kernel (eNTK), where we do not require the assumption of stationary kernel. Recent researches (Kopitkov & Indelman, 2020; Ortiz-Jiménez et al., 2021; Baratin et al., 2021) show that, with eNTKs non-linearly evolve during training, this kernel can serve as a measure of the difficulty of learning a particular task.
C.2 KERNEL REGRESSION WITH GRADIENT FLOW
First, we provide a well-founded spectral analysis of NTK regression. Assuming the network is trained with a constant training set X and their label set Y (i.e., ((z1, V11) V1),... , (IN, yN)) 2, y E RR,, where f(x; W,) f(W,). and the loss function L is defined as MSE. We can then express the time-dependent dynamics of the function using the NTK. To begin, we start with the equation for vanilla GD:
W++1 = We - WWW(WA W++1 W. - (W+).
(41)
(42)
With infinitesimally small learning rate, i.e, gradient flow, then:
dW.
dt L(W))
W f(W.) (f(W)) - Ymue).
(43)
Eq. 43 indicates the dynamics of the weights. Using a simple chain rule, we can modify the equation to represent the dynamics of the function, with the definition of NTK (i.c., Ke NxN):
(f.) = dW. dt = (V ((W,), V ((W)) dt
(W.). dW:
Yyne)
K. (S(W,) Yinue). Note that the element-wist wise representation of NTK is can be written as K2(2,, = (Vw W.), V W.)) R. where Iy denotes k-th data point in the entire training set X. Now, let g(W,)= = ((W.) - Yime. then Eq. 44 can be rewritten as: dg. dt -K (W..)
(44)
(45)
Eq. 45 is simple ordinary differential equation (ODE) and can be solved in the form of exponential growth by assuming the NTK constant during update; K, = Ky:
g(W))= = e -tkk g(Wo
(46)
f(W:) - Yime e -t.Ki (f(W)) - Ymne)
(47)
S(W)) Yime QE Qe-MQQ (J(Wo) Yme)
(48)
Q" (fW,) Yrme) -M (f(W)) - Ymme),
(49)
where Q is matrix of concatenation of eigenfunctions, which behaves a projecting operator, and A diagonal matrix contains non-negative eigenvalues. Additionally, equality in Eq. 48 due the property of spectral decomposition, i.e., ex = Q²"
Eigenvectors compose orthonormal set in N and each residual components projected in each eigendirection. This directly indicates the convergence speed in each direction in RAA For example, we can explictly compute the remained error in direction qt. denoted as U, at timestep t:
Ut d,t 10.i
(50)
23

---

Published as a conference paper at ICLR 2025
We can observe that once the eigenspace is defined (at the initial state), the eigenvalue governs the training speed in each direction the basis. Specifically, after an arbitrary time step f. the projected error in the direction corresponding large eigenvalues decreases exponentially faster than in the case of small eigenvalues.
Another important aspect is understanding the characteristics of each direction. Empirical and theoretical evidence has shown that the NTK cigenfunctions corresponding to large eigenvalues are low-frequency functions. As result, neural network training is biased towards reducing error in low-frequency regions (Basri et al., 2019; 2020; Wang et al., 2021), providing strong evidence of spectral bias (Rahaman et al., 2019)
A natural question arises, "what happens if the eigenvalue is large but the dot product between the corresponding cigenfunction and the residual is small?" This indicates that, despite the large eigenvalue, contributes less to the reduction of the overall loss. This directly implies that should consider not only the absolute scale of the eigenvalues but also the dot product (or alignment) between the eigenfunctions and the residual. In summary, can conclude: if there is high aligument in directions with large eigenvalues, training will be faster.
Based on this analysis, let the timestep that we want to evaluate the loss, then:
(f(W,,, - Ymme) = e B (f Ymue).
(51)
We evaluate the eNTK at every discrete timestep until the training ends. Specifically, we approxi- mate the flow of error over the interval t - - E [0, 1]. This setting allows the kernel to evolve during training and enables us to further explore the evolution of its eigenspace. For computation of NTKs in our work, we use 'Neural Tangent Library' (Novak et al., 2020).
24

---

Published as a conference paper at ICLR 2025
C.3 INSIGHTS INTO ENTK EIGENVALUE
In this subsection, we focus on the eNTK of two-layer sinusoidal networks with the scaling factor a, i.e,, [(; W) W2sin(aW1z)). The resulting kernel is sum of two dot-product kernels
KNTK( (2, x²) (sin(aW1x)) sin(aW1æ")) + (a cos(o a cos
(52)
More specifically, we study the condition number and eigenvalue decay rate (EDR) of the kernel as follows.
Eigenvalue gap of eNTK. We start by approximating the sinusoidal eNTK as a Gaussian kernel (Rahimi & Recht, 2007; Mehta et al., 2021), which allows for analytic form of the spectral decomposition. Using this approximation, we can also estimate the EDR. Here, we observe that increase in the scaling factor directly leads to decrease in the EDR.
Condition number. Building on recent studies of the condition number of the NTK, which significantly affects the convergence rate of neural networks (particularly the kernel regime) (Xiao et al., 2020; Liu & Hui, 2023), we demonstrate that WS-SNF approximately achieves low condition number for the eNTK, with the scaling factor a playing a crucial role due the lower EDR.
We start with the results from Mehta et al. (2021).
Theorem 7 (Theorem in Mehta et al. (2021).). Suppose each entry of W1 is initialized with a Gaussian distribution mean 0 and variance o². Then for any and x', we have
0² f2 x 2²113
Ew, [(in(W12)) sin ))I
C1 exp
(53)
2
Theorem 7 states that the expectation over the first-layer weights can be bounded by Gaussian ker- nels, where the bandwidth is determined by the initialization variance a. The scaling factor in Eq. (52) directly affects the variance term, which can be replaced with a²o², without loss of gener- ality. Note that can be handled similarly in the case of cosine activation function.
Returning to the eNTK, we start by analyzing K. From Eq. (53), the kernel element can be approx- imated as
K(I,, x,) S Ci exp o²a00| (
x, ==1l²) = K (xi,
(54)
Assuming K then Ai 2 > holds for all eigenvalue indices where X and A; denote the i- th eigenvalues of the kernels K" and K, respectively. Next step is to compute the eigenvalues of K** (i.e., eigenvalue upper bound of the K). The eigenvalues of K* can be represented as (Rasmussen & Williams, 2006):
2
2a²0²
A A Ci 1+ 20 + v1 + 2a 1 + 2a + V1 + 4a2
(55)
and K' can be handled similarly. In Eq. (55), we assume that x (N, oz) and set a2 without loss of generality. Note that the eigenfunctions of the Gaussian kernel K* can be computed via Her- mite polynomial expansion; however, this does not guarantee characteristics of the eigenfunctions of K, so we do not take into account. In Eq. (55), we can compute the eigenvalue decay rate (EDR) from arbitrary index i to k + by examining the base number. Intuitively, we can that the larger the a, the smaller the EDR. Considering both kermels K and K', characterizing the exact eigenvalue of KNTK K K' is open problem in the field of random matrix theory (Tao, 2012). However, we analyze the spectrum indirectly through Weyl's inequality from Courant-Fischer's theorem (Horn & Johnson, 2012). Let (K) and (K') the k(5 n) n)-th largest eigenvalue of each kernel. Then:
(K*) + A1 S Ae (KNIK) S >a()') + (K).
(56)
Here, p denotes the EDR of K. In other words, the bound of A (KNK)) is determined by the EDR of K, where a lower EDR (i.e., a larger scaling factor) contributes to an increase the lower bound of the eNTK eigenvalue for each index.
25

---

Published as a conference paper at ICLR 2025
This theoretical analysis aligns with empirical observations. Specifically, shown in Fig. 5b, the 2-axis represents the distribution of normalized eigenvalues. The eigenvalues of the weight- scaled network are more densely distributed compared to the default case (Lee, weight scaling applied), indicating that the EDR the weight-scaled network is lower. Intuitively, a smaller EDR corresponds smaller condition number, further suggesting that the optimization trajectory of weight scaling is relatively well-conditioned.
Recently, Saratchandran et al. (2024a) theoretically derived the minimum eigenvalue of the eNTK in periodically activated MLPs of general depth. Under certain assumptions, the minimum eigen- value is shown be asymptotically bounded, involving the variance of the initialization distribution (specifically, larger the variance, larger the minimum eigenvalue), which also supports our results.
26

---

Published as a conference paper at ICLR 2025
C.4 NON-KERNEL APPROACH: ANALYSIS VIA HESSIAN EIGENVALUES
Understanding the structure of the Hessian is a crucial aspect in the field of optimization. The Hessian matrix represents the second-order derivatives of the loss function with respect to each parameter, and its eigenvalues provide insights into the local curvature of the model's parameter space. The condition number of the Hessian, defined as Amax/Amine directly affects the convergence rate in gradient- based convex optimization problems, which is provable (Nesterov et al., 2018).
On the other hand, in deep learning, the Hessian is also widely studied gain better under- standing of the complex optimization trajectories and characteristies of deep neural networks. For instance, Ghorbani et al. (2019) examined the outlier eigenvalues of the Hessian neural networks (i.e., eigenvalues detached from the main bulk) and demonstrated through comprehensive empirical analyses that these outliers can slow down the network's convergence.
Building on this line of research, in this subsection, we plot the condition number (CN) of the Hessian during training in three different settings. Fig. 10 illustrates that the CN of WS-SNF remains significantly lower than that of other comparison models throughout training. This suggests that WS initialization leads to a better-conditioned Hessian not only in the -pprel-approximmaee regime but also in the nonlinear regime, contributing more efficient training dynamics.
To provide deeper insights and establish connections with related work on spectral analyses of the Hessian in neural networks, plot the eigenspectrum snapshot of the Hessian during training in Fig. 11. In both networks, we observe the occurrence of 'outlier eigenvalues' during training. How- ever, the default setting exhibits much more extreme value (even initialization), which serves as a proxy for poorly conditioned optimization and slow convergence, aligning with the observations in Ghorbani et al. (2019) and Rathore et al. (2024).
12
Staraterd Omage Soaling (10x)
10,000
(2x)
8.000
Condiion
1000
Norraions
Figure 10. Condition number of Hessian. The WS initialization consistently results lower condition number compared to other models, indicating that without the approximation of kernelized networks, the opti- mization path of WS-SNFFS is well-conditioned.
'For numerical stability, similar to Section 4.3, we compute the average of the top 3 eigenvalues for Amax and the bottom 3 eigenvalues for Amin. where the scale of the eigenvalues are considered in terms of their absolute values. For Hessian computation, we 'PyHessian' (Yao al., 2020).
27

---

Published as a conference paper at ICLR 2025
Denuty (1o) Scart
lagg
10
10
200
200
106
200
383
60
30
160
20
Digenwatoe
Eipersatuue
Espernate
Default (0 step)
Default (600 step)
Default (1339 step)
Canalty (ung
Denuiyy (10g Scale
10-
10-4
10
Esperratie
Weight scaling (0 step)
Weight scaling (35 step)
Weight scaling (79 step)
Figure 11. Figenspectrum of the Hessian during training: The first column represents the eigenspectrum initialization, the second column corresponds the eigenspectrum mid-term training, and the third column shows the cigenspectrum at the training.
D FREQUENCY SPECTRUM OF 2D IMAGES
To further validate our findings, we extend our analysis to Kodak image dataset. Following the methodology of Shi al. (2022), partition the frequency map into several discrete bands and compute the mean values within each band.
10"
Standard
102_
Welgght Staling (Pe)
10
10
Figure 12. Frequency distribution of SNFs and 2D signals. We partition the frequency map into 100 sub- groups facilitate the visualization of 2D signals in ID format. Natural images consistently exhibit sim- ilar frequency distribution patterns. Notably, WS-SNF demonstrates better capability in representing high- frequeney components, even initial state.
Fig. 12 illustrates the frequency distributions across three distinct images from Kodak dataset, as well as the initial output of SNFs. Our analysis reveals that natural images exhibit consistent spectral characteristics across the dataset, providing empirical support for claim presented in Section 5.2. Moreover, we observe that the initial functional generated by weight-scaled SNF demonstrates notably higher frequency distribution, consistent with theoretical analysis.
28

---

Published as a conference paper at ICLR 2025
E MORE DISCUSSION
E. THE EFFECT OF WEIGHT SCALING IN MINI-BATCH TRAINING
Most of our primary experiments were conducted under a full-batch training to prioritize fast con- vergence. In this section, we provide supplementary experiments on mini-batch training for image regression further investigate the effect of weight scaling.
Standard
80
Weight Staling
50
PSNR
90
10
100
200
300
350
400
450
500
steps
Figure 13. PSNR curves mini-batch training. PSNR curves for default weight scaled SNFs with various batch sizes. Darker colors represent larger batch sizes.
512x512 sized single image using five different batch sizes 215 24 221) over 500 iterations. Fig. 13 demonstrates that weight scaling is also ef- fective in mini-batch training, achieving significant acceleration compared to standard SNFs. These results emphasize the robustness and applicability of the proposed method.
We trained SNFs
29

---

Published as a conference paper at ICLR 2025
E.2
ON THE EFFECT OF WEIGHT SCALING ON OTHER ACTIVATION FUNCTIONS
100
20
Teut PSai
40
100
200
300
(a) SNF
(b) SNF
20
20-
16-
Texe PSNN
12-
10-
500
400
(c) ReLU + P.E.
(d) ReLU P.E.
30-
Text PONE
20
10
100
202
300
400
600
100
200
300
13200
stops
(c) GaussNet
(f) GaussNet
40
80
Teat PONT
12-
10-
8-
100
202
300
400
500
100
200
300
stece
steps
(g) WIRE
(h) WIRE
30-
Text PSNN
20-4
50
10-
10-
-00
200
300
20
820
100
200
300
400
800
stece
staps
Sine NF
ii) Sinc NF
Figure 14. Weight scaling in various architectures. WS substantially affects training of the neural field with periodic activations. We present curves PSNR (first column), as well as Test PSNR (sccond column).
30

---

Published as a conference paper at ICLR 2025
We extend our investigation to examine the effect of weight scaling different activation functions. Specifically, we evaluate ReLU, the most commonly used homogeneous activation function, along with other periodic activations beyond sinusoidal. Fig. 14 depicts the train and test PSNR curves over 500 iterations with varying values of a. As expected, neural networks with ReL.U activations are minimally affected by the scale of weight initialization. In contrast, architectures, such as GaussNet, WIRE, and Sinc NF are significantly influenced by weight scaling. This is because the weight distribution in non-homogeneous networks has substantial impact their performance, primarily through its effect on the functional frequency, as discussed in our manuscript.
We observe that weight- GaussNet and Sine NF also enjoy better performance in terms of the training PSNR, but their test PSNR degrades more severely than in SNFs. Moreover, WIRE shows a degraded performance with weight scaling. While our analysis primarily focused on sinusoidal neural fields, we emphasize that initialization for other periodic activations also impacts acceleration and generalization of neural field training. We leave this as future work, aiming to provide valuable insights into the role of weight initialization in enabling faster convergence.
E.3 THE OPTIMAL SCALING FACTOR ACCORDING TO THE LEARNING RATE
2.6
2.4
2,37
2.2
Optimala
2.0
1.8
1.6
1.4
10-6
10
Learning rate
Figure 15. Optimal scaling factor vs. learning rate.
We discussed the relationship between the optimal scaling factor and structure properties of neural fields in Section 5.2. However, the learning rate significantly influences training dynamics, poten- tially leading instability within the hyperparameter space. Fig. 15 illustrates the optimal weight scaling factor for different learning rates 1) E {10-3, 10-4 10 JJ. Our observations indicate that weight scaling consistently accelerates training across all tested learning rates. Furthermore, clear trend emerges: smaller values of a are optimal for larger 17. This trend highlights the predictability of the optimal a, consistent with other structural properties discussed our main section.
31

---

Published as a conference paper at ICLR 2025
F EXPERIMENTAL DETAILS AND QUALITATIVE RESULTS
F.1 EXPERIMENTAL DETAILS ABOUT FIG. 2B
Fig. 2b presents the tradeoff between acceleration and generalization when tuning w or a. Therefore, we choose the ranges of w and a that can help us capture the full tradeoff curve. Specifically, for frequency tuning, we explore a broad range, w [30, 6510], with intervals of 270, while also including smaller values (w 10 and 20) in the search space. On the other hand, weight scaling requires a narrower search range, with a E [0.4, 40] with the grid 0.2.
F.2 EXPERIMENTAL DETAILS ABOUT FIG. 3
For the k-th layer of an l-layer SNF, weight scaling increases the gradient magnitude by a'- com- pared to standard setting. By scaling down the learning rate of each layer by a/= the gradient update magnitudes become similar to those of the default while preserving the initial functional of weight scaling (Fig. 3b). In contrast, for Fig. 3c, the learning rate of ktth layer is amplified by 1) a to only utilize the effect of larger gradients. This is inspired by our analytic derivations at the end of Section 4.1 (and Appendix B.3).
F.3 EXPERIMENTAL DETAILS ABOUT EQ. (10)
We clarify that the optimization Eq. (10) is solved by simple grid search. That is, we have trained SNFs with many different values of and selected the value which maximizes the training speed and meets the constraint. We have tried all a within the range [1.0, 4.0] with the grid size 0.2.
F.4 TRAINING SETTINGS AND BASELINES
In this section, we provide detailed information about our experiments. For data fitting tasks, we used the Adam optimizer (Kingma, 2015) with learning rate of le-04 (except spherical data, in this case we use learning rate of le-05), without any learning rate scheduler, except for the occupancy field experiments (in which case, we use PyTorch learning rate scheduler). Each experiment was conducted with 5 different seeds, and we report both the average and standard deviation of the eval- uation metric. We note that all baselines in this work are MLP architectures, with slight modifica- tions to the input features activation functions. We used NVIDIA RTX 3090/4090//5000//6000 GPUs for all experiments. Additionally, we set the hyperparameters of each network using grid search, shown in T.
ReLU + P.E. (Mildenhall et al., 2020): For this architecture, we use fixed positional encoding applied to the input x, with K frequencies that lift the dimension of the inputs, i..., 7(x) (sin(29T .r), cos(22 sin...,,in(2K-1))) H x)), cos(22 .)))
FFN (Tancik et al., 2020): For this architecture, we use Gaussian positional encoding applied the input z, i.e., 7x) [sin(2nr z), cos(22 =)], where G defines the variance of the Gaussian distribution.
MEN (Fathony et al., 2021): For this architecture, we use MLP with multiple Gabor filters, which achieves superior results than Fourier filter networks (Fathony et al., 2021).
the Gaussian activation fune- tion, i.c., (Z) = exp (-(s z)²). where Z denotes the pre-activation matrix and s a hyperpa- rameter that defines the bandwidth of the Gaussian function.
GaussNet (Ramasinghe & Lucey, 2022): For this architecture,
WIRE (Saragadam et al., 2023): For this architecture, we use a Gabor wavelet-based activation function, i.e. (Z) exp(jw Z) exp(-(s ?) where and are hyperparameters.
SIREN (Sitzmann et al., 2020b): We use the sine activation function (2) = sin(w Z), where w in the first layer is a tunable hyperparameter.
Neural field scaling law (NFSL) (Saratchandran et al., 2024b): We modify the initialization distri-
bution of the last layer of SIREN, i... ((1) u (-V6/ (n .V6 n 3/4
(==-w)) where n
denotes the width of the (T. 1)-th layer.
32

---

Published as a conference paper at ICLR 2025
Sine NF (Saratchandran et al., 2024c): In this case. activation function sinc function, L.., o (Z) = sin(wz)/wr, where w is the fixed parameter. We use w = for all experiments, except for audio fitting (in this case, use 16).
Bengio, 2010): In this case, wC use standard Xavier uniform initialization for the SNF architecture, i.e. W (--v6/ / + R1, /6/vnu111 + E []. We note that the Xavier uniform ini- tialization in SNF does not yield reasonable results. However, the purpose presenting to highlight the initialization principle of SNFs (Appendix A), which guarantees stable training.
Xavier uniform initialization (Glorot
Table 2. Detailed information about hyperparameters. We provide the exact hyperparameter settings for each domain the table below. 'default' in ReLU+P.E. denotes the using nyquist sampling methods (Sara- gadam et al., 2023).
ReLU+PE
FFN
SIREN
GaussNet
WIRE
Weight scaling
k
Image Occ. Field
10
10.
256
30
10
20
10
2.37
default
2
10
10
10
10
40
3.7
Spherical Data Audio
default
2
256
30
10
10
20
2.5
default
20
20
3000
100
60
10
2.0
33

---

Published as a conference paper at ICLR 2025
F.5 NEURAL DATASET EXPERIMENTS
In recent research on constructing large-scale NF datasets (Ma et al., 2024; Papa et al., 2024), there has been growing interest in this area. However, no studies have investigated how initialization schemes affect the overall process, from constructing SNF-based neural datasets to their application downstream tasks. To address this gap. we first measure the time required to fit the entire dataset to NFs and then report the classification accuracy on the neural dataset while varying the scale factor a. The results are shown in Table
Analysis. Interestingly, despite the efficiency of WS-SNFs in terms of training time (i.e., the time required to fit each datum into an individual NF until reaching the target PSNR), the test accuracy (i.e., classification accuracy the test set SNFs), specifically for the case of
1.5, does not significantly decrease. Moreover, we observed that, in the case of a = 2.0, test accuracy dropped by average of 5 percentage points across all PSNR values. This mirrors similar phenomenon discussed in Papa et al. (2024), where constraining the weight space of NFs was identified as a key factor in downstream task performance. More precisely, scaling initialization distributions causes the neural functions to lie in a larger weight space, making it harder to learn features from the weights. Furthermore, we propose interesting research direction: without compromising the performance of downstream tasks and without incurring additional cost?
create neural dataset quickly
Experimental details. For training, used 'fit-a-net (Papa al., 2024), JAX-based library for fast construction of large-scale neural field datasets. We used the entire dataset for NF dataset generation (i.e., 60,000 images for MNIST and CIFAR-10, respectively). Additionally, we used an MLP-based classifier targeted at classifying each NF's class, which consists of 4 layers and width of 256, and was trained with cross- coos-eyn loss for 10 epochs.
Table 3. Neural dataset experiments. Each neural dataset is trained until reaches the batch-averaged target PSNR. The training time refers to the total time required to construct the entire INR dataset. classifier then trained on the neural dataset, and the resulting test accuracy is reported for each configuration. We use SIREN as the NF architecture, which is widely used baseline in this area.
Neural MNIST
Neural CIFAR-10
Target
Training
accuracy
Training time
accuracy
35
14m 23s
97.95=t0.00
19m 14s
47.98=0.01
SIREN
18m 20s
97.76:10.00
29m 45s
48.46=0.01
45
23m 23s
97,53=t0.01
47m 54s
48.09=0.01
7m 22s
97.56-t0.00
10m 52s
47.15-0.02
(S 1.5)
8m 22s
97.81-1-.000
17m 21s
47.72-0.01
10m 41s
97.02:40.01
25m 52s
46.08-0003
25s
97.48:20.01
26s
43.30=0.02
WS (x 2.0)
40
5m 32s
97.06
9m 20s
4..86-0.022
41s
97.49:10.00
42.70=0.01
34

---

Published as a conference paper at ICLR 2025
F.6 NEURAL RADIANCE FIELDS EXPERIMENTS
In this subsection, we provide details and quantitative results of novel view synthesis via neural radiance fields (NeRFs) (Mildenhall al., 2020). Neural radiance fields (NeRFs) represent scenes using neural fields. More specifically, NeRF employs neural fields (i.e. MLPs) with finite amount of training data to capture the scene, aiming to recover the scene in continuous manner. At the end, we input unseen coordinates into the model and evaluate the test PSNR.
Datasets. We use the 'Lego' and 'Drums' data from the 'NeRF-synthetic" dataset (Mildenhall al., 2020), which is publicly available online. Each dataset contains 100 training images, 100 validation images, and 200 test images, along with their corresponding camera directions.
Implementation details. We mainly follow the settings of WIRE (Saragadam et al., 2023), using the 'torch-ngp' codebase. Specifically, we use a 4-layer volume density network and 4-layer color network, each with a width of 182, except for WIRE. This exception is due to the complex-valued weights of WIRE, where we use width 128 for fair comparison. For the WS network, we apply weight scaling only to the volume density network.
Task training with 800 800 images. this case, we use wn = 15 and wr = 5 for the SNF family, frequency lifting factor & = 10 for ReLU + P.E., 20 for GaussNet, and 5, 10 for WIRE. Additionally, we use positional encoding scheme only for ReLU networks. We use learning rate 1e 02 for ReLU+P.E., 1e 03 for FFN, 3e 04 for SNF family, 1e 03 for GaussNet, and 6e - 04 for WIRE. We train all models until 300 epochs. We train the model with 100 images and evaluate the test PSNR with 100 untrained images.
Task 2: training with 200 200 images, with longer epochs. Additionally, as in Saragadam et al.
(2023), train model with 200 200 images, and trained until further epochs (600 epochs). In this case, We use W/ = 15 and w/ = 3 for the SNF family, frequency lifting factor k = 10 for ReLU + P.E.. s 20 for GaussNet, and only for ReLU networks. We use learning rate 6e 03 for ReLU + P.E., 3e 03 for FFN, 9e 04 for SNF family, le 03 for GaussNet, and 9e 04 for WIRE, We train the model with 100 images and evaluate the test PSNR with 100 untrained images.
20. 40 for WIRE. Also, we use positional encoding scheme
We report the test PSNR and learning curve in Table 4 and Fig. 16 respectively. For both datasets and both training resolutions, WS achieves a higher test PSNR compared to other baselines. We emphasize that WS primarily focuses on fast training neural fields, but here we also demonstrate its good generalization quality. Qualitative results can be found in Fig. 21.
Results.
800x800 (300 Epochs)
200x (600 Epochs)
Method
Lego
Drums
Lego
Drums
ReLU + P.E. (Mildenhall et al., 2020)
26.02
20.97
31.57
26.21
SIREN (Sitzmann et al., 2020b)
27.68
24.05
32.02
27.73
FFN (Tancik et al., 2020)
26.96
22.10
30.40
24.68
GaussNet (Ramasinghe & Lucey, 2022)
25.95
22.42
30.61
26.24
WIRE (Saragadam et al., 2023)
26.71
23.62
32.09
27.73
NFSL (Saratchandran et al., 2024b)
27.79
24.04
32.10
27.64
Weight Scaling (ours)
28.17
24.10
32.48
27.86
Table 4. Quantitative results: test PSNR for NeRF experiments.
35

---

Published as a conference paper at ICLR 2025
22-
PSNR
PSNR
20
R P.E.
FALU F.E.
20-
SETEN
SIEEN
18-
16
16
50
100
100
200
200
300
50
100
150
200
250
300
stops
stopa
(a) Lego (800 x 800 images, 300 epochs)
(b) Drums (800 x 800 images, 300 epochs)
28-
30
28-
Teat PBN4
18-
10-
15
200
900
420
500
800
100
200
500
400
500
Epocte
(c) Lego (200 x 200 images, 600 epochs)
(d) Drums (200 200 images, 600 epochs)
Figure 16. Test PSNR curves for NeRF experiments under different settings. The first row represents experi- ments conducted with 800 x 800 images over 300 epochs, while the second row shows experiments conducted with 200 200 images 600 epochs. WS initialization consistently achieves better test PSNR compared to baselines.
F.7 SOLVING DIFFERENTIAL EQUATIONS WITH PARTIAL OBSERVATIONS
In this subsection, we provide details about solving wave equation: type of second order time- dependent partial differential equation (PDE) describing wave propagation in time t. The initial conditions are only the given information, neural fields aim to extrapolate the equation in timescale domain. We use the same formulation in Sitzmann et al. (2020b).
Datasets and training details. We set fax (i.e., neural network) to reconstruct and extrapolate Gaussian pulse wave equation, same setup as in Sitzmann al. (2020b). Input coordinates are 3- dimensional (i.e., 2-dimensional spatial and I-dimensional temporal timescale from & = 0 to 4), and outputs I-dimensional Gaussian pulse value for each spatiotemporal coordinate. Other hyperparam- eters are set the same as in Sitzmann et al. (2020b), except for the number of iterations (we trained for up to 40k iterations).
We present the qualitative results in Fig. 22. However, in Fig. 22, the WS network consis- tently predicts the boundary of the Gaussian, compared to other architectures". Qualitative compar- isons are provided using coordinate-wist error (i.e., MSE) at = O. It can be seen that WS-SNF not only faithfully reconstructs the given observation but also able to generalize accurately to further timesteps. For WIRE and MFN, they failed to reconstruct the pulse; therefore, we do not provide the results.
Results.
"Since the ground truth Gaussian pulses for f are unavailable, they can only be reconstructed using PDE solver utilizing finite element method.
36

---

Published as a conference paper at ICLR 2025
F.8 QUALITATIVE RESULTS
In this subsection, we provide qualitative results of experiments in Section 5.1. We do not pro- vide reconstructed results for occupancy field reconstruction with Xavier initialization due to training failure.
Ground truth
ReLU P.E.
FFN
SIREN
MFN
GaussNet
WIRE
NESI
Sinc NE
WS (ours)
Figure 17. Image reconstruction: qualitative results.
37

---

Published as a conference paper at ICLR 2025
ReLU P.F.
FFN
SIREN
Ground Truth
MFN
GaussNet
WIRE
NFSL
Sinc NF
WS (ours)
Figure 18. Occupancy field experiments: qualitative results.
38

---

Published as a conference paper at ICLR 2025
140
Ground Truth
0.3
033
0.0
15.1
-10
Ground truth
Reconstructie
Reconstuutiss
63
-0.4-
-25
-2.-
-1.0
-05
0.5
-30
1.0
-3.0
-25
0.6
355
1.0
ReLU + P.E.
FFN
SIREN
Recorstracted
60
-05
-2.
-30
-25
0.5
-25
MFN
GaussNet
WIRE
-03
-35
3.
0.5
3.0
3.0
3.0
0.5
1.0
3.5
3.5
3.5
00
30
230
-32
-24
05
-25
-3.0
NFSL
Sinc NE
WS (ours)
Figure 19, Audio reconstruction: qualitative results.
39

---

Published as a conference paper at ICLR 2025
Ground truth
ReLU + ..E.
FFN
SIREN
MFN
GaussNet
WIRE
NFSL
Sinc
WS (ours)
Figure 20. Spherical data reconstruction: qualitative results. The top figure of each index shows the recon- structed results, while the bottom figure of each index shows the error map.
40

---

Published as a conference paper at ICLR 2025
Lego (ground truth)
ReLU P.E.
SIREN
GaussNet
FFN
WIRE
NFSL
WS (ours)
Drums (ground truth)
ReLU + P.E.
SIREN
GaussNet
FFN
WIRE
NFSL
WS (ours)
Figure 21. Novel view synthesis: qualitative results (800 x 800 images).
41

---

Published as a conference paper at ICLR 2025
Ground truth (t 0. which the only one given).
ReLU P.E. (7.7c-03)
SIREN (1.8e-02)
GaussNet (4.3c-03)
NFSL (7.8c-04)
WS (2.8e-05)
Figure 22, Solving PDE from partial observation: qualitative results. Each column represents 0.07,
0.14, and 0.21 from left to right. The parentheses in the caption* represent the MSE at f = 0.
42

---

Published as a conference paper at ICLR 2025
F.9 TRAINING FOR SIGNIFICANTLY LARGER ITERATIONS
In this section, we present the learning curves for Kodak image dataset over significantly longer iterations. We report the average train and test PSNR values for 5000 training steps, in which most methods converge to saturation. The learning rate scheduler is employed, resulting in improved performance across most methods under extended training setting, and the learning rates are tuned for each architecture to work well with a scheduler.
Xavier Int. R P.E.
SIREN inil.
WIRE
GaussNet
NFSL Weight Scaling (Ours)
Train PSNR
50
0
1000
2000
3000
4000
5000
steps
(a) Train PSNR curves for Kodak dataset
Xavier
SIREN Inf. GaussNet MFN
WIRE
ReLU + PE.
NFSL
FFN
Weight Scaling (Ours)
40
30
Test PSNR
20
0
1000
2000
3000
4000
5000
steps
(b) Test PSNR for Kodak dataset
Figure 23. PSNR curves over 5000 iterations. We compared the PSNR of weight scaling to other baselines.
Weight scaling demonstrates superior train accuracy, achieving higher PSNR with substantial accel- cration. Moreover, even at saturation, weight scaling outperforms other methods, with the highest PSNR. Notably, our methods maintain its generalization ability even for extended training. Weight scaling reaches its peak test PSNR (37.45 dB), which comparable to or exceeds that of other methods, much faster, demonstrating the fastest convergence in both train and test performance.
We utilize learning rate 1e 02 for Xavier initialized SNF, 1e 03 for ReLU P.E., 1e 03 for FFN, Ie 04 for SNF family, 1e 03 for GaussNet, le 02 for MFN, and le 04 for WIRE, with Adam optimizer.
43


Reviews:
[{"review_id": "JD5zanv5NK", "reviewer": "Reviewer_3nXe", "summary": "This work proposes an alternate initialization scheme for sinusoidal neural fields. The main claim is that scaling the initialization weights by a constant value greater than one can help speed up the learning in SNFs as well as improve fitting for higher frequencies, by improving the function at initialization as well as boosting optimization trajectories. Empirical validation on various domains is presented to substantiate the arguments.", "questions": "- See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The motivation is clear, and the exposition is easy to follow.\n- Although not very surprising, the study of the theoretical behavior of weight scaling is fairly comprehensive and I appreciate the authors efforts to draw upon different theoretical tools in existing literature to provide a broad picture.", "weaknesses": "- The current choice of experiments is rather limited. Since the contribution is largely methodological, experiments on important practical applications of SNFs, specifically NeRF and solving differential equations (for instance, see Saratchandran et al., 2024), as well as larger datasets, are needed to justify the claims. \n- Experimental methodology: A few aspects of the experimental methodology are unclear and/or need revision.\n    - Baseline: The included baseline with Xavier initialization is largely irrelevant from a practical standpoint. Alternatively, Multiplicative Filter Networks (Fathony et al, 2021) is an important baseline to consider, especially since multiplicative networks, akin to weight scaling, are known to speed up learning for higher frequencies [4] and thus, directly competes with weight scaling in terms of spectral bias.\n    - Fixed Iterations: The choice of number of training iterations to report performance is arbitrary and could reflect unfairly on the baselines. The authors should consider including the PSNR curves for training until convergence (for both WS and baselines) for understanding practical feasibility, since ground truth data to stop at a target PSNR is not available in practice.\n    - Full Batch Training/Susceptibility to SGD Noise: In practice, it is often not possible to perform full batch training due to size of dataset, network size etc. How does the noise in moving to stochastic (batched) gradient descent affect the convergence of weight scaling? Experiments that reflect those settings will help understand better the limitations of the proposed framework. \n- Related work: Missing citations for Multiplicative Filter Networks (Fathony et al, 2021) as well as spectral bias literature:\n    - [2] The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies, Basri et al., NeurIPS 2019\n    - [3] Towards Understanding the Spectral Bias of Deep Learning, Cao et al., 2020\n    - [4] The Spectral Bias of Polynomial Neural Networks, Choraria et al., ICLR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmyeLKqT5l", "reviewer": "Reviewer_svcN", "summary": "This paper studies initialization schemes for the class of sinusoidal neural fields (SNFs) and proposes a simple weight scaling method to accelerate training across different data domains. The idea is to simply scale the initial weights of all layers in the neural net by a well chosen factor. Some theoretical results and experimental results are provided to support and demonstrate the proposed method.", "questions": "- In Figure 2(b), the relationships between test PSNR and acceleration plotted for both cases of frequency tuning and weight scaling do not seem to be one-to-one. Why is that so? Am I missing something there?\n- In Eq. (3), the initialization scheme involves scaling the uniform distributions. Will similar weight scaling trick achieve similar acceleration results when the distributions involved are non-uniform (say Gaussian)?\n- How does the speed-up offered by the proposed weight scaling scheme depend on/interact with the choice of optimizers (say SGD vs Adam)? Is it more effective for certain choices of optimizers? \n- For the results in Figure 3, the learning rates vary with layers and are scaled down to match the rates of unscaled SNF. Can you provide more details on how they vary with depth? Also, what if the same learning rate is used for all layers?\n- Can you provide more details on how the optimization problem (10) is solved? Can we treat $\\alpha$ as a trainable parameter instead?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Overall the paper is well written, with sufficient number of plots provided to visualize the findings.\n- The proposed weight scaling scheme is simple yet seems to be quite effective in speeding up training.", "weaknesses": "- The theoretical results are quite limited. For instance, to show that the proposed weight scaling increases the relative power\nof the higher-frequency bases, the paper considers the effects of such scaling on a 3-layer SNF only for the case when the width is one. \n- While the optimization trajectories are studied via the lens of the empirical neural tangent kernel, it would be more convincing if there are theoretical/empirical results that quantify the impact of such scaling on the gradient loss and the speed of convergence of gradient descent. \n- There are also some missing details and confusing parts in the paper that would need to be addressed in order to make the presentation stronger (see my questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Eqac4Ulrr", "reviewer": "Reviewer_bvG5", "summary": "The work introduces a method to accelerate training in sinusoidal neural fields (SNFs), focusing on a technique called weight scaling (WS). This method modifies the initialization by scaling the initial weights of SNFs (except for the final layer) by a constant factor. By improving the frequency characteristics of SNFs, WS significantly reduces training time—up to 10 times faster than the traditional approach—while maintaining generalization. The authors provide theoretical insights and empirical validation across tasks in some data domains, including images, 3D shapes, and audio signals. They compare WS with standard initialization and frequency tuning, demonstrating its advantages in both speed and stability during training. The idea behind this method is quite interesting, as it revolves around reducing time to create more efficient models; however, there are some significant weaknesses and gaps that cannot be ignored.", "questions": "* $\\textbf{Impact on Long-Term Generalization}$: How does WS affect long-term generalization on larger datasets or in cases where SNFs are used for predictive tasks (e.g., continuous function approximation or spatial data interpolation)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* $\\textbf{Effective Speed Improvement}$: WS offers a substantial acceleration in training time without degrading model performance or generalization capabilities, which is valuable for SNF applications requiring rapid processing. \n* $\\textbf{Simplicity and Practicality}$: The method requires minimal changes to existing architectures, making it straightforward to apply WS to SNFs without complex modifications. \n* $\\textbf{Well-Written}$: The paper has been written in detail.\n* $\\textbf{Theoretical Justifications}$: The paper offers a detailed theoretical framework explaining why WS is effective, especially in reducing spectral bias and improving the conditioning of the optimization trajectory. This theoretical result supports the empirical findings and strengthens the reliability of the results.", "weaknesses": "The authors acknowledge the main limitations of their work, which cannot be ignored. To interpret the results accurately, these limitations must be carefully considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tS9nDZPH7", "reviewer": "Reviewer_jBfM", "summary": "This paper presents a study on sinusoidal neural fields (SNFs) and introduces \"weight scaling,\" a method that accelerates training by adjusting the initialization of each layer, except the final one. The key idea behind weight scaling is straightforward: each layer’s weights are initialized by multiplying with a specific scale factor, which results in faster training across various neural field tasks. The authors benchmark their approach against existing methods, including the original SNF initialization by Sitzmann et al., and demonstrate that weight scaling consistently achieves faster convergence while maintaining robust training and test performance. Furthermore, the authors provide theoretical analysis explaining why weight scaling works.", "questions": "1). Your paper is entirely focused on neural fields that use a sinusoidal activation. The weight scaling method you propose scales the initialization of a sinusoidal neural field by a certain factor $\\alpha$, as per eqn (6) in your paper. This method takes the SIREN initialization as its basis. Could you explain what happens if we take a standard initialization scheme like Kaiming uniform or Xavier uniform and scale it in exactly the same way as your eqn. (6)? Does this lead to anything new? Why did you apply your weight scaling to only the SIREN initialization?\n\n2). Could you explain whether your results would go through for other activations that are used within the neural fields literature such as Gaussian [1], Wavelet [2], Sinc [4]? \n\n3). Did you try weight scaling on a NeRF task? I am very surprised there are no NeRF results as this a big experiment that requires hours to train and I noticed some of the papers you are comparing against [1], [2], [3] compare to NeRF. Could you explain whether weight scaling would or wouldn't work on NeRF and if it cannot be made to work on NeRF then why? I think it would really help your paper if you had NeRF results and you were able to show large speed ups on this large scale experiment.\n\n4). Your weight scaling essentially just scales the initialization of the inner layers of the initialization by Sitzmann et al for SIRENs. However, if you look at the denominator of eqn. (3), line 194, there is the term $\\omega_h$ which is supposed to be the frequency of the sinusoid being used on that layer. Could we not get the same effect as your initialization by using a sinusoid with frequency \n$\\frac{\\omega_h}{\\alpha}$? \n\n5). In appendix C.3 you analyse the condition number of the Hessian of the loss landscape. You mention that the condition number of the Hessian directly affects the convergence rate. However, my understanding is that this is only true for convex loss landscapes. Loss functions associated to neural networks are almost always non-convex, as they employ non-linearities, therefore in such cases the condition number of the Hessian may not be well defined, as the Hessian may have negative eigenvalues or zero as an eigenvalue. Could you comment on this? I don't understand this analysis you have provided in your paper.\n\n--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n[1] Ramasinghe et al: Beyond Periodicity: Towards a unifying framework for activations in coordinate-MLPs; ECCV 2022.    \n\n[2] Saragadam et al: WIRE: Wavelet implicit neural representations; CVPR 2023.\n\n[3] Saratchandran et al: Scaling insights for optimizing neural fields: CVPR 2024.\n\n[4] Saratchandran et al: A Sampling Theory Perspective on Activations for Implicit Neural Representations: ICML 2024.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality:** The paper introduces an original idea that I have never seen used before in the neural field literature. Furthermore, the authors take the time to give a good theoretical analysis of why weight scaling works and how it relates to other initialization methods within the literature.\n\n**Quality:** The paper has both theoretical and experimental contributions making it a good quality paper written by the authors. The theoretical contributions are sound and the experiments are compared with recent literature.\n\n**Clarity:** The paper is in general well written. The authors take the time to exposit the main ideas behind weight scaling well and explain their key contributions in a readable manner. I have checked their mathematical proofs in the appendix and they were all well written and explained well.\n\n**Significance:** The paper gives a new way to initialize sinusoidal neural fields which I believe is interesting to the community of researchers working in neural fields. The initialization of neural fields in general is not as commonly studied as other deep networks such as CNNs or Transformers and apart from a few recent works on the topic, I believe this paper offers a nice viewpoint in how sinusoidal neural fields should be initialized for efficient training.", "weaknesses": "**Novelty:** My main issue with the paper and what I feel is its main weakness is the fact that it is entirely focused on one type of neural field, namely sinusoidal neural fields. The paper does not say anything about other neural fields in general. The authors do say that their methods would not work on ReLU based neural fields due to ReLU having the positive homogeneity condition and\nwhile the authors do say that sinusoidal neural fields are used within the community I would argue that there are several papers showing that sinusoidal neural fields are not as effective as other neural fields, such as [1], [2], [3], [4].  While the paper is written well I do feel that it being entirely focused on one type of neural field means it won't be as useful to many researchers in the neural fields community. \n\n**Large Scale Experiments:** The paper compares its methodology of weight scaling on a tasks such as image regression, binary occupancy, spherical data, audio data and fit a nef (in the appendix). Yet I noticed there were no experiments on Neural Radiance Fields (NeRFs) which I found very surprising. The authors say that their method is very effective at reducing the training time of neural fields which is generally large yet many of the tasks such as image regression, audio, occupancy are not considered big tasks within the community. On the other hand NeRF is a big experiment that takes anywhere between 4 hours to 12 hours to train, depending on the fidelity the user wants. I would have liked to see how their method did on such a task that requires hours of training. I also looked at the papers the authors compare to namely [1], [2], [3] and noticed that these papers compared their methods on NeRF. In the appendix the authors do compare their method on \"fit a nef\" on a large scale dataset but these takes less than 1 hour to train even with a SIREN. Thus I feel another big weakness of the paper is that the authors did not compare their method on a task that takes hours to train such as NeRF, so that we can see how their method does on a more complicated large scale neural field.\n\n[1] Ramasinghe et al: Beyond Periodicity: Towards a unifying framework for activations in coordinate-MLPs; ECCV 2022.    \n\n[2] Saragadam et al: WIRE: Wavelet implicit neural representations; CVPR 2023.\n\n[3] Saratchandran et al: Scaling insights for optimizing neural fields: CVPR 2024.\n\n[4] Saratchandran et al: A Sampling Theory Perspective on Activations for Implicit Neural Representations: ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)