Fast Training of Sinusoidal Neural Fields via Scaling Initialization

## Abstract

Abstract Neural fields are an emerging paradigm that represent data as continuous functions parameterized by neural networks. Despite many advantages, neural fields often have a high training cost, which prevents a broader adoption. In this paper, we focus on a popular family of neural fields, called sinusoidal neural fields (SNFs), and study how it should be initialized to maximize the training speed. We find that the standard initialization scheme for SNFs—designed based on the signal propagation principle—is suboptimal. In particular, we show that by simply multiplying each weight (except for the last layer) by a constant, we can accelerate SNF training by 10 × \times × . This method, coined weight scaling , consistently provides a significant speedup over various data domains, allowing the SNFs to train faster than more recently proposed architectures. To understand why the weight scaling works well, we conduct extensive theoretical and empirical analyses which reveal that the weight scaling not only resolves the spectral bias quite effectively but also enjoys a well-conditioned optimization trajectory.

## 1 Introduction

Neural field (NF) is a special family of neural networks designed to represent a single datum (Xie et al., ). Precisely, NFs parametrize each datum with the weights of a neural net which is trained to fit the mapping from spatiotemporal coordinates to corresponding signal values. Thanks to their versatility and low memory footprint, NFs have been rapidly adopted in various domains, including high-dimensional computer vision and graphics (Sitzmann et al., ; Mildenhall et al., ; Poole et al., ), physics-informed machine learning (Wang et al., ; Serrano et al., ), robotics (Simeonov et al., ), and non-Euclidean signal processing (Rebain et al., ).

For neural fields, the training speed is of vital importance. Representing each datum as an NF requires tedious training of a neural network, which can take up to several hours of GPU-based training (Mildenhall et al., ). This computational burden becomes a critical obstacle toward adoption to tasks that involve large-scale data, such as NF-based inference or generation (Ma et al., ; Papa et al., ). To address this issue, many research efforts have been taken to accelerate NF training, including fast-trainable NF architectures (Sitzmann et al., ; Müller et al., ), meta-learning (Sitzmann et al., ; Chen & Wang, ), and data transformations (Seo et al., ).

Despite such efforts, a critical aspect of NF training remains understudied: How should we initialize NFs for the fastest training? While the importance of a good initialization has been much contemplated in classic deep learning literature (Glorot & Bengio, ; Zhang et al., ), such understandings do not immediately answer our question, for many reasons. First, conventional works mostly focus on how initialization affects the signal propagation properties of deep networks with hundreds of layers, while most NFs use shallow models with only a few layers (Mildenhall et al., ). Second, conventional works aim to maximize the model quality at convergence, while in NFs it is often more important how fast the network achieves certain fidelity criterion (Müller et al., ). Lastly, NFs put great emphasis on how the model performs on seen inputs (i.e., training loss) and leaves how well it interpolates between coordinates (i.e., generalizability) as a secondary issue, while conventional neural networks care exclusively on the test performance.

To fill this gap, in this paper, we investigate how the initialization affects the training efficiency of neural fields. In particular, we focus on initializing sinusoidal neural field (SNF) architectures, i.e., multilayer perceptrons with sinusoidal activation functions (Sitzmann et al., ). SNF is one of the most widely used NFs, due to its versatility and parameter-efficiency; the architecture works as a strong baseline in a wide range of data modalities (Grattarola & Vandergheynst, ; Kim et al., ; Rußwurm et al., ) and is used as an indispensable building block of many state-of-the-art neural field frameworks (Guo et al., ; Schwarz et al., ; Pal et al., ).

Contribution. We first discover that the current practice of SNF initialization is highly suboptimal in terms of the training speed (Sitzmann et al., ). Precisely, we find that one can accelerate the training speed of SNFs by up to 10$\times$, simply by scaling up the initial weights of all layers by a well-chosen factor. This extremely simple tuning, coined weight scaling (WS), provides a much greater speedup than the conventional SNF tuning strategy, i.e., scaling up the frequency multiplier of the activation functions (called $\omega_{0}$ in Sitzmann et al. ()). Furthermore, this speedup can be achieved without any degradation in the generalization capabilities of the model, if we keep the scaling factor at a moderate level. Interestingly, experiments suggest that the optimal scaling factor, which strikes the balance between speedup and generalization, mainly depends on the physical scale of the NF workload (e.g., data resolution and model size) and remains similar over different data.

Figure: Figure 1: A simple weight scaling accelerates training. The proposed weight scaling scales up the initial weights of an SNF by the factor of $\alpha$, except for the last layer (left panel). This scaling significantly reduces the number of steps required to achieve the desired level of training loss (right panel).
Refer to caption: x1.png

To demystify why the weight scaling accelerates the SNF training, we conduct an in-depth analysis. In particular, we focus on understanding how the effects of WS on SNFs are similar yet different from ReLU nets, where the so-called lazy training phenomenon takes place (Chizat et al., )—weight-scaled ReLU nets behave similarly to kernels, enjoying exponential convergence in theory for shallow nets but failing to achieve good accuracy at practical scales due to ill-conditioned trajectories. Our theoretical and empirical analyses highlight several key differences that sinusoidal activations bring, such as: (1) Weight-scaled SNFs also preserve activation distributions over layers, facilitating the gradient propagation when training a deep model (3.1). (2) Weight scaling increases not only the frequency of each basis, but also increases the relative magnitude of higher-order harmonics, which helps fit the high-frequency components abundant in natural signals (Lemmas 4.2 and 4.2). (3) Weight-scaled SNFs enjoy even better-conditioned trajectories than the unscaled models, as captured by the eigenspectrum analysis (Section 4.3).

In a sense, our study constitutes a call for rethinking the role of the initialization for neural fields, by demonstrating how dramatically suboptimal the current schemes can be in key performance metrics for NFs. We hope that our work will further inspire more sophisticated initialization algorithms.

## 2 Related work

Training efficiency of neural fields. There are two prevailing strategies to accelerate NF training. The first approach is to adopt advanced NF architectures, which mitigate the spectral bias of neural nets that hinders fitting high-frequency signals (Rahaman et al., ; Xu et al., ). Popular examples include utilizing specially designed positional encodings (Tancik et al., ; Müller et al., ) or activation functions (Sitzmann et al., ; Ramasinghe & Lucey, ; Liu et al., ). Another strategy is to amortize the training cost via meta-learning, e.g., by training a fast-trainable initialization (Sitzmann et al., ; Tancik et al., ) or constructing an auxiliary model that predicts NF parameters from the datum (Chen & Wang, ). This work aims to accelerate NF training through a better initialization. Unlike meta-learned initializations, however, our aim is to understand how the initialization (its scale, specifically) can facilitate training by analyzing how it affects optimization dynamics, without meta-learning involved.

Neural network initialization. Early works on network initialization focus on preserving the signals through layers to facilitate training deep networks. This is typically done via principled scaling of the initial weights (Glorot & Bengio, ; He et al., ), or by ensuring isometries in each layer through orthogonalization (Xiao et al., ). Recent works focus on the relationship between the initialization and the implicit bias of SGD (Vardi, ). In particular, the initialization scale determines whether the training takes place in the rich regime or kernel regime, which correspond to stronger and weaker influence of implicit bias, respectively (Woodworth et al., ; Varre et al., ). Empirically, weak implicit bias leads to the models trained from larger initialization to generalize poorly (Mehta et al., ). Our analyses share some spirit with the latter line of the literature. Distinct from the works, however, we put our main focus on how initialization affects the training speed, specifically targeting the neural fields.

Initializing sinusoidal neural fields. There is a limited number of works that study the initialization for SNFs. Sitzmann et al. () proposes an initialization scheme for SNF, designed in the signal propagation perspective. Tancik et al. () develops a meta-learning algorithm to learn initializations for SNFs. Lee et al. () extends the algorithm for sparse networks. Most related to our work, Saratchandran et al. () proposes an initialization scheme that aims to facilitate the parameter-efficiency of SNFs. Different from this work, our work primarily focuses on the training efficiency, and achieves much faster training speed empirically.

## 3 Neural fields initialization via weight scaling

In this section, we formally describe the weight scaling for a sinusoidal neural field initialization, and briefly discuss the theoretical and empirical properties of the method.

### 3.1 Network setup and notations

We begin by introducing the necessary notations and problem setup. A sinusoidal neural field is a multilayer perceptron (MLP) with sinusoidal activation functions (Sitzmann et al., ). More concretely, suppose that we have an input $X\in\mathbb{R}^{d_{0}\times N}$, where $N$ is the number of $d_{0}$-dimensional coordinates. Then, an $l$-layer SNF can be characterized recursively via:

$$ $\displaystyle f(X;\mathbf{W})=W^{(l)}Z^{(l-1)}+b^{(l)},\qquad Z^{(i)}=\sigma_{ i}(W^{(i)}Z^{(i-1)}+b^{(i)}),\quad i\in[l-1],$ (1) $$

parametrized by $\mathbf{W}=(W^{(1)},b^{(1)},\ldots,W^{(l)},b^{(l)})$, where each $W^{(i)}\in\mathbb{R}^{d_{i}\times d_{i-1}}$ denotes the weight matrix, $b^{(i)}\in\mathbb{R}^{d_{i}}$ denotes the bias, and $Z^{(i)}\in\mathbb{R}^{d_{i}\times N}$ denotes the activation of the $i$th layer. Here, we let $Z^{(0)}=X$, and the activation function $\sigma_{i}:\mathbb{R}^{d_{i}}\to\mathbb{R}^{d_{i}}$ are the sinusoidal activations applied entrywise, with some frequency multiplier $\omega$. The standard practice is to use a different frequency multiplier for the first layer. Precisely, we apply the activation function $\sigma_{1}(x)=\sin(\omega_{0}\cdot x)$ for the first layer, and $\sigma_{i}(x)=\sin(\omega_{h}\cdot x)$ for all other layers, i.e., $i\neq 1$.

The SNF is trained using the gradient descent, with the mean-squared error (MSE) as a loss function. We assume that the input coordinates of the training data (i.e., the column vectors of the input data) have been scaled to lie inside the hypercube $[-1,+1]^{d_{0}}$, as in Sitzmann et al. ().

### 3.2 Standard initialization

The standard initialization scheme for the sinusoidal neural fields independently draws each weight entry from a random distribution (Sitzmann et al., ). Precisely, the weight matrices are initialized according to the scaled uniform distribution as

$$ $\displaystyle w^{(1)}_{j,k}\stackrel{{\scriptstyle\textrm{i.i.d.}}}{{\sim}} \mathrm{Unif}\left(-\frac{1}{d_{0}},\frac{1}{d_{0}}\right),\qquad w^{(i)}_{j,k }\stackrel{{\scriptstyle\textrm{i.i.d.}}}{{\sim}}\mathrm{Unif}\left(-\frac{ \sqrt{6}}{\omega_{h}\sqrt{d_{i-1}}},\frac{\sqrt{6}}{\omega_{h}\sqrt{d_{i-1}}} \right),$ (2) $$

where $w^{(i)}_{j,k}$ denotes the $(j,k)$-th entry of the $i$th layer weight matrix $W^{(i)}$.

Importantly, the range of the uniform initialization (Eq. 2) is determined based on the distribution preserving principle, which aims to make the distribution of the activation $Z_{i}$ similar throughout all layers. In particular, Sitzmann et al. (, Theorem 1.8) formally proves that the choice of the numerator (i.e., $\sqrt{6}$) ensures the activation distribution to be constantly $\arcsin(-1,1)$ throughout all hidden layers of the initialized sinusoidal neural field.

Frequency tuning. A popular way to tune the initialization (Eq. 2) for the given data is by modifying the frequency multiplier of the first layer $\omega_{0}$. This method, which we call frequency tuning, affects the frequency spectrum of the initial SNF, enhancing the fitting of signals within certain frequency range. It is not typical to tune $\omega_{h}$, as changing its value does not change the initial SNF; the effects are cancelled out by the scaling factors in the initialization scheme.

### 3.3 Initialization with weight scaling

Figure: (a) Weight scaling with various scaling factors
Refer to caption: x2.png

In this paper, we develop a simple generalization of the standard initialization scheme. In particular, we consider weight scaling, which multiplies a certain scaling factor $\alpha\geq 1$ to the initialized weights in all layers except for the last layer. More concretely, the initialization scheme is

$$ $\displaystyle w^{(1)}_{j,k}\stackrel{{\scriptstyle\textrm{i.i.d.}}}{{\sim}} \mathrm{Unif}\left(-\frac{{\color[rgb]{0,0.4,0.8}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0.4,0.8}\alpha}}{d_{0}},\frac{{\color[rgb]{0,0.4,0.8} \definecolor[named]{pgfstrokecolor}{rgb}{0,0.4,0.8}\alpha}}{d_{0}}\right), \qquad w^{(i)}_{j,k}\stackrel{{\scriptstyle\textrm{i.i.d.}}}{{\sim}}\mathrm{ Unif}\left(-\frac{{\color[rgb]{0,0.4,0.8}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0.4,0.8}\alpha}\sqrt{6}}{\omega_{h}\sqrt{d_{i-1}}},\frac{{\color[rgb]{ 0,0.4,0.8}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.4,0.8}\alpha}\sqrt{6}}{ \omega_{h}\sqrt{d_{i-1}}}\right),$ (3) $$

where we let $\alpha=1$ for the last layer; this design is due to the fact that any scaling in the last layer weight leads to a direct scaling of the initial function $f\mapsto\alpha f$, which is known to be harmful for the generalizability, according to the theoretical work of Chizat et al. (). Empirically, we observe that this weight scaling is much more effective in speeding up the training than a recently proposed variant which scales up only the last layer of the NF (Saratchandran et al., ); see Section 5.

There are two notable characteristics of the weight-scaled initializations.

(1) Distribution preservation.
An attractive property of the proposed weight scaling is that, for SNFs, any $\alpha\geq 1$ also preserves the activation distribution over the layers.(^1^11This point has also been noted in Sitzmann et al. (), but has not been discussed quantitatively for $\alpha\geq 1$. We make this point concrete via numerical analyses on the large $\alpha$ case.) In particular, we provide the following result, which shows that the activation distributions will still be an $\arcsin(-1,1)$.

###### Proposition 3.1 (Informal; extension of Sitzmann et al. ( 2020b , Theorem 1.8) ) .

Consider an $l$-layer SNF initialized with the weight scaling (Eq. 3). For any $\alpha\geq 1$, we have, in an approximate sense,

$$ $\displaystyle Z^{(i)}\sim\arcsin(-1,1),\qquad\forall i\in\{2,\ldots,l-1\}.$ (4) $$

The detailed statement of this proposition and the numerical derivation will be given in Appendix A.

The proposition implies that we can tune the scaling factor $\alpha$ within the range $[1,\infty)$ without any concerns regarding preserving the forward propagation signal. Thus, we can safely adopt the weight scaling for training a very deep sinusoidal neural field.

(2) Speed vs. generalization. Empirically, we observe that the weight scaling (Eq. 3) allows us to explore for the “sweet spot” region, where we can enjoy the accelerated training, while not suffering from any degradation in the generalization performance on unseen coordinates.

The Fig. 2 illustrates this phenomenon for the case of image regression, where we train an SNF to fit a downsampled image until some desired peak signal-to-noise ratio (PSNR) is met, and then use the model to interpolate between seen pixel coordinates.

We observe that by increasing $\alpha$ from $1$, the loss on unseen coordinates increase, and the training speed increases as well. Notably, there exists some $\alpha>1$ where the test PSNR remains similar while the training speed has strictly increased.

###### Proposition 3.1 (Informal; extension of Sitzmann et al. ( 2020b , Theorem 1.8) ) .

### 3.4 The paradigm of efficiency-oriented initialization

The latter phenomenon motivates us to develop an alternative perspective on a goodness of a neural field initialization, which may replace or be used jointly with the distribution preservation principle. More specifically, we consider an optimization problem

$$ $\displaystyle\max_{\alpha\geq 1}\mathrm{speed}(\alpha)/\mathrm{speed}(1)\quad \mathrm{subject\>to}\quad\mathrm{TestLoss}(\alpha)-\mathrm{TestLoss}(1)\leq\varepsilon,$ (5) $$

where $\mathrm{speed}(\alpha)$ denotes the training speed (e.g., a reciprocal of the number of steps required) of SNF when initialized with $\alpha$ weight scaling, and $\mathrm{TestLoss}(\alpha)$ denotes the loss on the unseen coordinates when trained from the initialization. Given this objective, the key questions we try to address are:

- •
Why does the weight scaling accelerate the training of SNFs? $\triangleright$ See Section 4
- •
Is weight scaling generally applicable for other data modalities? $\triangleright$ See Section 5.1
- •
How can we select a good $\alpha$, without expensive per-datum tuning? $\triangleright$ See Section 5.2

## 4 Understanding the effects of the weight scaling

We now take a closer look at the weight scaling to understand how it speeds up the training of sinusoidal neural fields. Before diving deeper into our analyses, we first discuss why a conventional understanding does not fully explain the phenomenon we observed for the case of SNF.

Comparison with lazy training. At the first glance, it is tempting to argue that this phenomenon is connected to the “lazy training” phenomenon (Chizat et al., ; Woodworth et al., ): Scaling the initial weights ($\mathbf{W}\mapsto\alpha\mathbf{W}$) amplifies the initial model functional ($f\mapsto\alpha^{l}f$), which in turn leads to the model to be linearized; then, one can show that the SGD converges at an exponential rate, i.e., the fast training, with the trained parameters being very close to the initial one. However, there is a significant gap between this theoretical understanding and our observations for the SNFs:

- •
The proposed weight scaling does not amplify the initial functional, as sinusoidal activations are not positively homogeneous (i.e., $\sigma(ax)\neq a\cdot\sigma(x)$, even for positive $a,x$), unlike ReLU. In fact, WS keeps the last layer parameters unscaled, making it further from any functional amplification, and empirically works better than a direct amplification of functional (Saratchandran et al., ).
- •
At a practical scale, a naïve scaling of initial weights for ReLU networks leads to both high training and test losses, due to a high condition number (Chizat et al., , Appendix C). In contrast, the training loss of the WS remains very small even at the practical tasks.

In light of these differences, we provide alternative theoretical and empirical analyses that can help explain why the WS accelerates training, unlike in ReLU nets. In particular, we reveal that:

- •
WS on SNFs has two distinctive effects from ReLU nets, in terms of the initial functional and the layerwise learning rate; the former is the major contributor to the acceleration (Section 4.1).
- •
At initialization, WS increases both (1) the frequency of each basis, and (2) the relative power of the high-frequency bases (Section 4.2).
- •
WS also results in a better-conditioned optimization trajectory (Section 4.3).

### 4.1 Two effects of weight scaling on sinusoidal networks

There are two major differences in how the weight scaling impacts SNF training compared to its effect on conventional neural networks with positive-homogeneous activation functions, e.g., ReLU. (1) Initial functional: Larger weights lead to higher frequency (low-level) features in SNF, whereas in ReLU nets scaling up does not change the neuronal directions (more discussions in Section 4.2).
(2) Larger early layer gradients: Unlike ReLU nets, scaling up SNF weights has a disproportionate impact on the layerwise gradients; earlier layers receive larger gradients (see remarks below).

Motivated by this insight, we compare which of these two elements play a dominant role in speeding up the SNF training by decoupling these elements (Fig. 3). In particular, we analyze how changing only the initial weights—with the layerwise learning rates that are scaled down to match the rates of unscaled SNF—affects the speed-generalization tradeoff; we also compare with the version where the initial function stays the same as default, but the learning rates on early layers are amplified. From the results, we observe that the effect of having a good initial functional, with rich high-frequency functions, play a dominant role on speeding up the SNF training. The disproportionate learning rate also provides a mild speedup, while requiring less sacrifice in the generalizability.

Figure: (a) Weight scaling (as a whole)
Refer to caption: x4.png

How are the early layer gradients amplified? Essentially, this is also due to the lack of positive-homogeneity of the activation function. To see this, consider a toy two-layer neural net with width one $f(x)=w^{(2)}\sigma(w^{(1)}x)$. The loss gradient for each layer weight is proportional to the layerwise gradients of the function, which is $\nabla_{\mathbf{w}}f=\big{(}w^{(2)}x\sigma^{\prime}(w^{(1)}x),\sigma(w^{(1)}x)
\big{)}$. For ReLU activations, both entries of $\nabla_{\mathbf{w}}f$ grow linearly as we increase the scale of both weights. For $\sin$ activations, on the other hand, the first entry scales linearly, but the scale of the second entry remains the same. Following a similar line of reasoning, we can show that the first layer of a depth-$l$ sinusoidal network has a gradient that scales as $\alpha^{l}$, while the last layer gradient scales as $1$. We provide a more detailed discussion on the $\alpha$-dependency of layerwise gradients in Section B.3.

### 4.2 Frequency spectrum of the initial functional

We have seen that the initial functional has a profound impact on the training speed, but how does the weight scaling affect the functional? The answer is easy for two-layer nets; the weight scaling amplifies the frequency of each hidden layer neuron, enabling the model to cover wider frequency range with hardly updating the features generated by the first layer.

How about for deeper networks? Our theoretical and empirical analyses reveals that, for deeper SNFs, the weight scaling also increases the relative power of the higher-frequency bases. Thus, the weight-scaled networks may suffer less from the spectral bias, and can easily represent signals with large high-frequency components (Rahaman et al., ).

Theoretical analysis. We now show theoretically that the weight scaling increases the relative power of the higher-frequency bases. For simplicity, consider again a three-layer bias-free SNF

$$ $\displaystyle f(x;\mathbf{W})=W^{(3)}\sin(W^{(2)}\sin(W^{(1)}x)),$ (6) $$

with sinusoidal activation functions. Then, it can be shown that the overall frequency spectrum of the function $f(\cdot;\mathbf{W})$ can be re-expressed as the multiples of the first layer frequencies, via Fourier series expansion and the Jacobi-Anger identity (Yüce et al., ).

###### Lemma 4.1 (Corollary of Yüce et al. ( 2022 ) ) .

The network Eq. 6 can be rewritten as

$$ $\displaystyle f(x;\mathbf{W})=2W^{(3)}\sum_{\ell\in\mathbb{Z}_{\text{odd}}}J_{ \ell}(W^{(2)})\sin(\ell W^{(1)}x),$ (7) $$

where $J_{\ell}(\cdot)$ denotes the Bessel function (of the first kind) with order $\ell$.

Given the Bessel form (7), let us proceed to analyze the effect of weight scaling. By scaling the first layer weight, i.e., $W^{(1)}\mapsto\alpha W^{(1)}$, we are increasing the frequencies of each sinusoidal basis by the factor of $\alpha$. That is, each frequency basis becomes a higher-frequency sinusoid.

Scaling the second layer weight, on the other hand, acts by affecting the magnitude of each sinusoidal basis, i.e., $J_{\ell}(W^{(2)})$. For this quantity, we can show that the rate that the Bessel function coefficients evolve as we consider a higher order harmonics (i.e., larger $\ell$) scales at the speed proportional to $\alpha^{2}$. That is, the relative power of the higher-order harmonics becomes much greater as we consider larger scaling factor $\alpha$. To formalize this point, let us further simplify the model (6) to have width one. Then, we can prove the following sandwich bound:

###### Lemma 4.2 (Scaling of harmonics) .

For any nonzero $W^{(2)}\in(-\pi/2,\pi/2)$, we have:

$$ $\frac{(W^{(2)})^{2}}{(2\ell+2)(2\ell+4)}<\frac{J_{\ell+2}(W^{(2)})}{J_{\ell}(W ^{(2)})}<\frac{(W^{(2)})^{2}}{(2\ell+1)(2\ell+3)}.$ (8) $$

In other words, replacing $W^{(2)}\mapsto\alpha\cdot W^{(2)}$ increases the growth rate $J_{\ell+2}/J_{\ell}$ by $\alpha^{2}$. We note that there exists a restriction in the range $W^{(2)}$ which prevents us from considering very large $\alpha$. However, empirically, we observe that such amplification indeed takes place for the practical ranges of $\alpha$ where the generalization performance remains similar.

Figure: Figure 4: Spectrum of initialized SNFs. 1D-FFT of an initialized 5-layer SNF, with various levels of weight scaling factors.
Refer to caption: x7.png

Summing up, the weight scaling has two effects on SNFs with more than two layers. First, the WS increases the frequencies of each sinusoidal basis by scaling the first layer weights. Second, the WS increases the relative weight of higher-frequency sinusoidal bases, by scaling the weight matrices of the intermediate layers.

Empirical analysis. To demonstrate that the increase of coefficients in the higher-order harmonics takes place indeed, we conduct a spectral analysis of an initialized SNF for audio data (Fig. 4). From the plot, we observe that the initialized SNFs with larger scaling factor $\alpha$ has a richer higher frequency spectrum. Moreover, the decay rate of the higher-frequency components, as we consider increase the frequency level, is much smaller for larger $\alpha$; as a consequence, weight-scaled SNFs tend to have non-zero coefficients for very high frequency ranges, enabling a faster fitting to the natural audio signals (gray) with rich high-frequency components.

###### Lemma 4.1 (Corollary of Yüce et al. ( 2022 ) ) .

###### Lemma 4.2 (Scaling of harmonics) .

### 4.3 Optimization trajectories of weight-scaled networks

Now, we study the optimization properties of SNFs through the lens of the empirical neural tangent kernel (eNTK). We empirically analyze two properties of the eNTK eigenspectrum of the SNF, which are relevant to the training speed: (1) condition number, and (2) kernel-task alignment.

Condition number. Roughly, the condition number denotes the ratio $\lambda_{\max}/\lambda_{\min}$ of the largest and the smallest eigenvalues of the eNTK; this quantity determines the exponent of the exponential convergence rates of the kernelized models, with larger condition number meaning the slower training (Chizat et al., ). In Fig. 5(a), we plot how the condition numbers of the eNTK evolve during the training for SNFs, under weight scaling or frequency tuning; for numerical stability, we take an average of top-5 and bottom-5 eigenvalues and take their ratio. We observe that the weight-scaled SNFs tend to have a much smaller condition number at initialization, which becomes even smaller during the training. On the other hand, the SNF initialized with the standard initialization scheme suffers from a very high condition number, which gets even higher during the training.

We note that this behavior is in stark contrast with ReLU neural networks, where the weight scaling leads to a higher condition number, leading to a poor convergence (Chizat et al., ).

Kernel-task alignment. The kernel-task alignment—also known as the energy concentration—is the cumulative sum of the (normalized) dot products

$$ $\displaystyle\mathcal{E}(t)=\sum_{i:\lambda_{i}/\lambda_{0}\geq t}(\phi_{i}^{ \top}\mathbf{e})^{2}/\|\mathbf{e}\|_{2}^{2}$ (9) $$

where $\phi_{i}$ denotes the $i$-th eigenvector of the kernel and $\mathbf{e}=Y-f(X)$ denotes the vector of prediction errors in the pixels (Kopitkov & Indelman, ). This quantity measures how much of the residual signal is concentrated in the directions of large eigenvalues, which is theoretically easier to be expressed or optimized by kernel learning (Baratin et al., ; Yüce et al., ). In Fig. 5(b), we compare the kernel-task alignments during the SNF training trajectories. We observe that the weight scaling gives a highly optimizable kernel throughout all stages of training.

Figure: (a) Condition number
Refer to caption: x8.png

## 5 Experiments

In this section, we first address whether the weight scaling is effective in other data domains (Section 5.1); our answer is positive. Then, we discuss the factors that determine the optimal value of scaling factor $\alpha$ for the given target task (Section 5.2); we find that the optimal value does not depend much on the nature of each datum, but rather relies on the structural properties of the workload.

### 5.1 Main experiments

**Table 1: Weight scaling in various data domains. We compare the training speed of the weight-scaled SNF against other baselines in various data domains. To evaluate the training speed, we train for a fixed number of steps and compare the training loss achieved. Bold denotes the best option, and underlined denotes the runner-up. We have experimented with five random seeds, and report the mean and the standard deviation.**
|  |  | Image (PSNR) | Occ. Field (IoU) | Spherical (PSNR) | Audio (PSNR) |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Activation | KODAK | DIV2K | Lucy | ERA5 | Bach |
| Xavier Uniform (Glorot & Bengio, ) | Sinusoidal | 0.46$\pm$0.10 | 0.39$\pm$0.10 | 0.0000$\pm$0.0000 | 4.11$\pm$0.66 | 7.77$\pm$0.20 |
| ReLU + P.E. (Mildenhall et al., ) | ReLU | 18.60$\pm$0.08 | 16.72$\pm$0.08 | 0.9896$\pm$0.0003 | 33.30$\pm$0.54 | 24.98$\pm$0.19 |
| FFN (Tancik et al., ) | ReLU | 20.52$\pm$0.60 | 19.81$\pm$0.48 | 0.9843$\pm$0.0020 | 38.69$\pm$0.27 | 16.66$\pm$0.28 |
| SIREN init. (Sitzmann et al., ) | Sinusoidal | 24.58$\pm$0.05 | 22.86$\pm$0.06 | 0.9925$\pm$0.0001 | 38.72$\pm$0.07 | 37.37$\pm$3.11 |
| GaussNet (Ramasinghe & Lucey, ) | Gaussian | 21.94$\pm$2.48 | 19.22$\pm$0.14 | 0.9914$\pm$0.0005 | 38.56$\pm$0.51 | 27.47$\pm$2.10 |
| WIRE (Saragadam et al., ) | Wavelet | 28.94$\pm$0.21 | 28.20$\pm$0.13 | 0.9912$\pm$0.0005 | 31.27$\pm$0.53 | 16.83$\pm$1.85 |
| NFSL (Saratchandran et al., ) | Sinusoidal | 24.93$\pm$0.07 | 23.39$\pm$0.09 | 0.9925$\pm$0.0001 | 38.92$\pm$0.07 | 37.17$\pm$2.88 |
| Weight scaling (ours) | Sinusoidal | 42.83$\pm$0.35 | 42.03$\pm$0.41 | 0.9941$\pm$0.0002 | 45.28$\pm$0.03 | 45.04$\pm$3.23 |

We validate the effectiveness of WS in different data domains by comparing against various neural fields. To compare the training speed, we fix the number of training steps and compare the training accuracy. In particular, we consider the following tasks and baselines.

Task: Image regression. The network is trained to approximate the signal intensity $c$, for each given normalized 2D pixel coordinates $(x,y)$. For our experiments, we use Kodak (Kodak, ) and DIV2K (Agustsson & Timofte, ) datasets. Each image is resized to a resolution of $512\times 512$ in grayscale, following Lindell et al. (); Seo et al. (). We report the training PSNR after a full-batch training for 150 iterations. For all NFs, we use five layers with width 512.

Task: Occupancy field. The network is trained to approximate the occupancy field of a 3D shape, i.e., predict ‘1’ for occupied coordinates and ‘0’ for empty space. We use the voxel grid of size $512\times 512\times 512$ following Saragadam et al. (). For evaluation, we measure the training intersection-over-union (IoU) after 50 iterations on the ‘Lucy’ data from the ’Standard 3D Scanning Repository,’ with the batch size 100k. We use NFs with five layers and width 256.

Task: Spherical data. We use 10 randomly selected samples from the ERA5 dataset, which contains temperature values corresponding to a grid of latitude $\phi$ and longitude $\theta$, using the geographic coordinate system (GCS). Following Dupont et al. (), the network inputs are lifted to 3D coordinates, i.e., transforming $(\phi,\theta)$ to $(\cos(\phi)\cdot\cos(\theta),\cos(\phi)\cdot\sin(\theta),\sin(\phi))$. We report the training PSNR after 5k iterations of full-batch training. For NFs, we use five layers with width 256.

Task: Audio data. Audio is a 1D temporal signal, and the network is trained to approximate the amplitude of the audio at a given timestamp. We use the first 7 seconds of “Bach’s Cello Suite No. 1,” following Sitzmann et al. (). We report the training PSNR after 1k iterations of full-batch training. For NFs, we use five layers with width 256.

Baselines. We compare the weight scaling against seven baselines; we have selected the memory-efficient and versatile neural fields, rather than highly specialized and heavy ones, such as Instant-NGP (Müller et al., ). More concretely, we use the following baselines:
(1) Xavier uniform: the method by (Glorot & Bengio, ) applied on SNF, (2) SIREN init.: SNF using the standard initialization of SIREN (Sitzmann et al., ), (3) NFSL: A recent initialization scheme motivated by scaling law (Saratchandran et al., ). (4) ReLU+P.E.: ReLU nets with positional encodings (Mildenhall et al., ). (5) FFN: Fourier feature networks Tancik et al. (). (6) GaussNet: MLP with Gaussian activations (Ramasinghe & Lucey, ). (7) WIRE: MLP with Gabor wavelet-based activations (Saragadam et al., ). The baselines (1-3) use SNFs as the backbone architecture, while (4-7) uses other architectures with different activation functions.

Other details. For selecting the hyperparameters for each NFs (including $\alpha$), we have conducted a grid search. We use the Adam optimizer (Kingma, ) with a fixed learning rate specific to each data domain. Other experimental details are provided in Appendix E.

Result. Table 1 reports the comprehensive results across various data domains. We observe that the weight scaling significantly outperforms all baselines throughout all tasks considered. In Section E.4, we provide further qualitative results on various modalities, where we find that the weight scaling enhances capturing the high frequency details. These results support our hypothesis that the richer frequency spectrum provided by the weight scaling mitigates the spectral bias (Section 4.2), thereby enabling a faster training of neural fields.

Additional experiments. We have also conducted experiments on the NF-based dataset construction tasks (Papa et al., ). We report the results in Section E.2.

### 5.2 On selecting the optimal scaling factor

Now, we consider the problem of selecting an appropriate scaling factor without having to carefully tune it for each given datum. In particular, we are interested in finding the right $\alpha$ that maximizes the training speed yet incurs only negligible degradations in generalizability. More concretely, consider the task of Kodak image regression and seek to select the $\alpha$ that approximately solves

$$ $\displaystyle\underset{\alpha}{\text{maximize}}\quad$ $\displaystyle\mathrm{speed}(\alpha)\qquad\text{subject to}\quad\mathrm{ TestPSNR}(\alpha)\geq 0.95\cdot\mathrm{TestPSNR}(1).$ (10) $$

where (again) the $\mathrm{speed}$ is the reciprocal of the number of steps required until the model meets the desired level of training loss, which we set to be the PSNR 50dB. Note that this optimization is similar to what has been discussed in Section 3.4.

In a nutshell, we observe in this subsection that this optimal scaling factor is largely driven by the structural properties of the optimization workload (such as model size and data resolution), and remains relatively constant over the data. This
suggests that the strategy of tuning $\alpha$ on a small number of data and transferring it to other data in the same datasets can be quite effective.

Structural properties. In Fig. 6, we visualize how changing the data resolution ($512\times 512$ by default), model width ($512$ by default), and model depth ($4$ by default) affects the value of optimal scaling factor. We observe that the higher data resolution and shallower model depth lead to a need for a higher scaling factor, with the width having very weak correlation. Commonly, we find that the resolution and depth are both intimately related to the frequency spectrum; higher resolution leads to a need for a higher frequency, and deeper networks can effectively utilize the relatively smaller scaling factors to represent high-frequency signals.

Data instances. In Fig. 7, we observe that both curves are strikingly quite similar in shape, having saturation and inflection points at the similar scaling factor. This may be due to the fact that natural images tend to exhibit a similar frequency distribution.

Figure: (a) Data resolution
Refer to caption: x10.png

Figure: (a) Acceleration
Refer to caption: x13.png

## 6 Conclusion

In this paper, we revisit the traditional initialization scheme of sinusoidal neural fields, and propose a new method called weight scaling, which scales the initialized weights by a specific factor. In practice, WS significantly accelerates the training speed of SNF across various data modalities, even outperforming recent neural field training methods. We have discovered that the difference between the traditional and WS initialization comes from both the change in the frequency spectrum of the initial functional and the change in the layerwise learning rates, where the former plays a more crucial role in acceleration. Through extensive theoretical and empirical analyses, we have revealed that the WS positively impacts the learning dynamics by maintaining a well-conditioned optimization path throughout the training process.

Limitations and future direction.
The limitations of our work lie in the fact that the analysis has been conducted for only neural fields using sinusoidal activation functions, rather than general neural field architectures; whether the weight scaling is effective on other architectures remains an open question. Also, from a theoretical perspective, our analyses does not provide a definitive answer on the acceleration mechanisms, lacking an explicit convergence bound. As a final remark, our work is closely related to the *implicit bias of initialization scale* (Azulay et al., ), which is an emerging topic in understanding the mechanics of neural network training. However, there remains a significant gap between theory and practice in this area, as the current interpretation is limited to shallow, homogeneous, and linear networks (Varre et al., ; Kunin et al., ), rather than general nonlinear networks. The next step is to understand the implicit bias in networks with general activation functions, where the scale of initialization is not necessarily equivalent to output scaling. We leave this as future work.

#### Reproducibility statement

Most experiments in our work are based on widely used neural field baselines, whose official source codes are publicly available online. The only new part, i.e., our proposed algorithm, is rather simple and only requires multiplying constants to the initial weights. While the implementation is quite simple, we will also release our code for our algorithm and the experiments as a public repository, upon the preparation of the camera-ready version of the paper.

## Appendix A Detailed statement and the derivation of 3.1

In this section, we provide a more detailed statement and derivation of 3.1, where we have extended the distribution-preserving claims of Sitzmann et al. () on $\alpha=1$ to the cases of $\alpha\geq 1$, thus covering the initialization with weight scaling.

We note that we generally follow the approach of Sitzmann et al. () in terms of the formalisms; we provide rough approximation guarantees (Lemmas A.1 and A.2), one of which relies on a numerical analysis of a function that is difficult to be expressed in a closed form (Lemma A.2).

In particular, we are concerned with showing the following two points:

- •
Activations to pre-activations. We show that, for $\alpha\geq 1$, the pre-activation of a layer with weight-scaled uniform activations and $\arcsin$ input distributions are approximately distributed as a Gaussian distribution with variance scaled by $\alpha$ (Lemma A.1).
- •
Pre-activations to activations. Then, we show that, for $\alpha\geq 1$, post-activation distribution of the Gaussian distribution with $\alpha$-scaled variance is distributed as an $\arcsin$ distribution (Lemma A.2).

More formally, our lemma are as follows.

###### Lemma A.1 (Arcsin to Gaussian distributions) .

Let the weights $W_{l}$ of the $l$-th layer be sampled i.i.d. from $\mathrm{Unif}(-c/\sqrt{n},c/\sqrt{n})$, and the activation of the $(l-1)$-th layer $Z_{l-1}$ follow the i.i.d. $\mathrm{arcsin}(-1,1)$. Then, the distribution of the pre-activation of the $l$-th layer $Y_{l}$ converges in distribution to a Gaussian distribution $\mathcal{N}(0,c^{2}/6)$ as the number of hidden units $n$ tends to infinity.

###### Lemma A.2 (Gaussian to arcsin distributions, with numerical proof) .

Let the pre-activation distribution $X$ has a Gaussian distribution $\mathcal{N}(0,\alpha^{2})$ for some $\alpha\geq 1$. Then the activation distribution $Y=\sin(X)$ is distributed as $\mathrm{arcsin}(-1,1)$.

Note that, by plugging in $c=\alpha\sqrt{6}$ to the Lemma A.1 as in the proposed weight scaling, we get the normal distribution $\mathcal{N}(0,\alpha^{2})$ as an output distribution.

We provide the proofs of these lemma in Sections A.1 and A.2, respectively. For all derivations, we simply ignore the issue of the frequency scaling term $\omega_{0}$, and simply let it equal to $1$.

###### Lemma A.1 (Arcsin to Gaussian distributions) .

###### Lemma A.2 (Gaussian to arcsin distributions, with numerical proof) .

### A.1 Proof of Lemma A.1

For simplicity, let us denote a pre-activation of a neuron in the $l$th layer by $y$. Let the weight vector connected to this output neuron as $\mathbf{w}\in\mathbb{R}^{n}$, with each entries drawn independently from $\mathrm{Unif}(-c/\sqrt{n},c/\sqrt{n})$. Also, let the corresponding input activation from the $(l-1)$th layer be denoted by $\mathbf{z}\in\mathbb{R}^{n}$, with each entry drawn independently from $\arcsin(-1,1)$. Now, it suffices to show that $y=\mathbf{w}^{\top}\mathbf{z}$ converges in distribution to $\mathcal{N}(0,c^{2}/6)$ as $n$ goes to infinity.

To show this point, we simply invoke the central limit theorem for the i.i.d. case. For CLT to hold, we only need to check that each summand ($w_{i}z_{i}$) of the dot product $\mathbf{w}^{\top}\mathbf{z}=\sum w_{i}z_{i}$ has a finite mean and variance.

$$ $\displaystyle\mathbf{w}^{\top}\mathbf{z}=\sum_{i=1}^{d}w_{i}z_{i}$ (11) $$

has a finite mean and variance
$\mathbf{w}^{\top}\mathbf{z}$ has a zero mean and a finite variance. We know that $\mathbb{E}[w_{i}z_{i}]=0$, as two random variables are independent and $\mathbb{E}[w_{i}]=0$. To show that the variance is finite, we can proceed as

$$ $\displaystyle\mathrm{Var}(w_{i}z_{i})=\mathrm{Var}(w_{i})\mathrm{Var}(z_{i})= \frac{c^{2}}{3n}\frac{1}{2}=\frac{c^{2}}{6n},$ (12) $$

where the first equality holds as each variables are independent and zero-mean. Summing over $n$ indices, we get what we want.

### A.2 Proof of Lemma A.2

By the 3$\sigma$ rule, $99.7\%$ of the mass of the Gaussian distribution $\mathcal{N}(0,\alpha^{2})$ lies within the finite support $[-3\alpha,3\alpha]$. The CDF of $Y$ can be written as

$$ $F_{Y}(y)=P(Y\leq y)=P(\sin(X)\leq y).$ (13) $$

Different from the settings of Sitzmann et al. (), the sinusoidal activation is not a bijection in general within the support $[-3\alpha,3\alpha]$, i.e., not directly invertible. Thus, we break it down into three cases (Fig. 8): Case 1, where the endpoint $3\alpha$ is less than $\pi/2$, Case 2, where $3\alpha$ is in the increasing region of $\sin(\cdot)$, and Case 3, where $3\alpha$ is in the decreasing region. We study each case:

Case 1 ($\alpha<\pi/6$). Here, we have

$$ $\displaystyle F_{Y}(y)$ $\displaystyle=P(X\leq\arcsin y)=F_{X}\left(\arcsin y\right)\approx\frac{1}{2}+ \frac{1}{2}\tanh\left(\frac{\beta}{\alpha}\arcsin y\right),$ (14) $$

where the last approximation invokes the Gaussian CDF approximation (Bowling et al., ). In this case, the CDF of $Y$ does not approximate the CDF of $\arcsin(-1,1)$. That is, the second-order derivative of $F$ decreases in $[-1,0]$ and increases in $[0,1]$, while the CDF of the $\arcsin$ distribution behaves in an opposite way.

Figure: Figure 8: Visualization of the cases in Lemma A.2. In $y=\sin(x)$, Red line : $x=3\alpha$ and Black line : $x=\frac{\pi}{2}$
Refer to caption: extracted/5906275/assets/fig/distribution/example.jpg

Now, suppose that $\alpha$ is sufficiently large ($\alpha\geq\pi/6$), i.e., case 2 or case 3. Here, we will consider the nearest peak/valley that falls outside the $[-3\alpha,3\alpha]$ interval. More concretely, let $\gamma$ be defined as the period index of such peak/valley, i.e.,

$$ $\displaystyle\gamma$ $\displaystyle=\min\{\gamma_{1},\gamma_{2}\},\qquad\text{where}$ (15) $\displaystyle\gamma_{1}$ $\displaystyle=\min\left\{n\in\mathbb{N}\mid\frac{(4n+1)\pi}{2}\geq 3\alpha~{} \cap~{}\sin\left(\frac{(4n+1)\pi}{2}\right)=+1\right\}\qquad{\color[rgb]{ .5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5} \pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\text{(peak; case 2)}}$ (16) $\displaystyle\gamma_{2}$ $\displaystyle=\min\left\{n\in\mathbb{N}\mid\frac{(4n-1)\pi}{2}\geq 3\alpha~{} \cap~{}\sin\left(\frac{(4n-1)\pi}{2}\right)=-1\right\}\qquad{\color[rgb]{ .5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5} \pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\text{(valley; case 3)}}$ (17) $$

By considering the range up until such peaks/valleys, we can ensure that the probability of $X$ lying within this range is greater than 0.997.

Now, let’s proceed to analyze the case 2; the case 3 can be handled similarly.

Case 2. We break down the CDF of $Y$ as

$$ $\displaystyle F_{Y}(y)=F_{Y,\text{center}}(y)+F_{Y,\text{inc}}(y)+F_{Y,\text{ dec}}(y).$ (18) $$

Here, $F_{Y,\text{center}}(y)$ denotes the CDF of the bijective region, i.e., $x\in[-\pi/2,\pi/2]$, and $F_{Y,\text{inc}}(y)$ denotes the region where the $\sin(\cdot)$ is increasing and $F_{Y,\text{dec}}(y)$ denotes where $\sin(\cdot)$ is decreasing. Then, each of these components can be written as:

$$ $\displaystyle F_{Y,\text{center}}(y)$ $\displaystyle=P(-\pi/2\leq X\leq\arcsin y)=F_{X}(\arcsin y)-F_{X}(-\pi/2)$ (19) $\displaystyle F_{Y,\text{inc}}(y)$ $\displaystyle=\sum_{n=1}^{\gamma_{1}}\big{[}F_{X}(\arcsin y+4n\pi/2)-F_{X}((4n -1)\pi/2)$ $\displaystyle\qquad+F_{X}(\arcsin y-4n\pi/2)-F_{X}\left(-(4n+1)\pi/2\right) \big{]}$ (20) $\displaystyle F_{Y,\text{dec}}(y)$ $\displaystyle=\sum_{n=1}^{\gamma_{1}}\big{[}F_{X}((4n-1)\pi/2)-F_{X}(-\arcsin y +(4n-2)\pi/2)$ $\displaystyle\qquad+F_{X}(-(4n-3)\pi/2)-F_{X}(-\arcsin y-(4n-2)\pi/2)\big{]}$ (21) $$

Combining these terms, we get:

$$ $\displaystyle F_{Y}(y)$ $\displaystyle=F_{X}\left(\arcsin y\right)-F_{X}\left(-\pi/2\right)$ $\displaystyle\quad+\sum_{n=1}^{\gamma_{1}}\big{[}F_{X}(-(4n-3)\pi/2)-F_{X}(-(4 n+1)\pi/2)$ $\displaystyle\qquad+F_{X}(\arcsin y+4n\pi/2)+F_{X}(\arcsin y-4n\pi/2)$ $\displaystyle\qquad-F_{X}(-\arcsin y+(4n-2)\pi/2)-F_{X}(-\arcsin y-(4n-2)\pi/2 )\big{]}$ (22) $$

Using the logistic approximation and Taylor’s first-order expansion at $z=\arcsin y=0$, i.e., $f(z)\approx\frac{f^{\prime}(z)}{1!}(z-0)+f(0)$, this can be further approximated by

$$ $\displaystyle F_{Y}(y)$ $\displaystyle=\Bigg{[}\frac{\beta}{2\alpha}h^{\prime}(0)+\sum_{n=1}^{\gamma_{1 }}\Bigg{(}\frac{\beta}{\alpha}\bigg{(}h^{\prime}\left(\frac{4n\beta\pi}{2 \alpha}\right)+h^{\prime}\left(\frac{(4n-2)\beta\pi}{2\alpha}\right)\bigg{)} \Bigg{)}\Bigg{]}z$ $\displaystyle\quad-\frac{1}{2}h\left(-\frac{\beta\pi}{2\alpha}\right)+\sum_{n= 1}^{\gamma_{1}}\Bigg{[}\frac{1}{2}h\left(-\frac{(4n-3)\beta\pi}{2\alpha}\right )-\frac{1}{2}h\left(-\frac{(4n+1)\beta\pi}{2\alpha}\right)\Bigg{]},$ (23) $$

where $h$ is the $\tanh$ function. The above equation has variables $\gamma_{1}$ and $\alpha$. Thus, we can rewrite it as:

$$ $\displaystyle F_{Y}(y)=f(\alpha)\cdot\arcsin y+g(\alpha),$ (24) $\displaystyle f(\alpha)=\frac{\beta}{2\alpha}h^{\prime}(0)+\frac{\beta}{\alpha }\sum_{n=1}^{\gamma_{1}}\Bigg{[}h^{\prime}\left(\frac{4n\beta\pi}{2\alpha} \right)+h^{\prime}\left(\frac{(4n-2)\beta\pi}{2\alpha}\right)\Bigg{]},$ (25) $\displaystyle g(\alpha)=-\frac{1}{2}h\left(-\frac{\beta\pi}{2\alpha}\right)+ \frac{1}{2}\sum_{n=1}^{\gamma_{1}}\Bigg{[}h\left(-\frac{(4n-3)\beta\pi}{2 \alpha}\right)-h\left(-\frac{(4n+1)\beta\pi}{2\alpha}\right)\Bigg{]}.$ (26) $$

Figure: Figure 9: Approximated CDF with increasing in $\alpha$. Red line indicates the reference error. When $\alpha$ goes larger, the error converges to limit near zero.
Refer to caption: extracted/5906275/assets/fig/distribution/activation.jpg

Now, we compare the $F_{X}(x)$ to ground truth CDF of $\mathrm{arcsin}(-1,1)$, i.e., $\frac{2}{\pi}\arcsin\sqrt{\frac{x+1}{2}}$. As the closed form solution is not obtainable, we numerically compute the difference in integration over input domain:

$$ $\displaystyle\hat{\alpha}\in\left\{\alpha\ \middle|\ \int_{-1}^{1}\left|F_{X}( x,\alpha)-\frac{2}{\pi}\arcsin\left(\sqrt{\frac{x+1}{2}}\right)\right|\,dx\right.$ (27) $\displaystyle\left.\leq\int_{-1}^{1}\left|F_{X}(x,1)-\frac{2}{\pi}\arcsin\left (\sqrt{\frac{x+1}{2}}\right)\right|\,dx\right\}.$ (28) $$

This indicates that the RHS term represents the cumulative error from the default initialization, which is assumed to be valid for preserving the forward signal. If the LHS is smaller than the RHS for some value of $\hat{\alpha}$ that satisfies the inequality, it intuitively follows that the elements in $\hat{\alpha}$ provide a valid scale. In Fig. 9, we empirically demonstrate that as the scaling factor $\alpha$ increases, the error between the ground truth and the approximated CDF decreases with some limit near zero, which implies $\hat{\alpha}\in[1,\infty)$.

## Appendix B Proofs in Section 4

### B.1 Proof of Lemma 4.1

To formally prove this lemma, we first bring the generalized form from Yüce et al. ().

###### Lemma B.1 (From Yüce et al. ( 2022 , Appendix A.2) ) .

Suppose that the model can be written as $f(x)=W^{(3)}\sin\left(W^{(2)}\sin(W^{(1)}x)\right)$, where $W^{(1)}\in\mathbb{R}^{d_{1}}$, $W^{(2)}\in\mathbb{R}^{d_{2}\times d_{1}}$, and $W^{(3)}\in\mathbb{R}^{1\times d_{2}}$. Then, the model can be equivalently written as:

$$ $f(x)=\sum^{d_{2}-1}_{m=0}\sum^{\infty}_{i_{1},\cdots,i_{d_{1}}=-\infty}\left( \prod^{d_{1}-1}_{k=0}J_{k}\left(w^{(2)}_{k}\right)\right)w_{m}^{(3)}\sin\left( \left(\sum_{k=0}^{d_{1}-1}i_{k}\cdot w^{(1)}_{k}\right)x\right),$ (29) $$

where the lowercase $w_{k}^{(i)}$ denotes the $k$-th column of the matrix $W^{(i)}$.

We specialize this lemma to the case of width-one SNF, i.e., $d_{1}=d_{2}=1$. We proceed as

$$ $\displaystyle f(x)$ $\displaystyle=W^{(3)}\sin(W^{(2)}\sin(W^{(1)}x))$ (30) $\displaystyle=2\sum_{k=0}^{\infty}J_{2k+1}(W^{(2)})\sin((2k+1)W^{(1)}x),$ (31) $$

where the second equality is due to the Jacobi-Anger identity. By restricting the sum to $l\in\mathbb{Z}_{\text{odd}}$, we can further simplify as

$$ $f(x;\mathbf{W})=2\sum_{\ell\in\mathbb{Z}_{\text{odd}}}J_{\ell}(W^{(2)})\sin( \ell W^{(1)}x).$ (32) $$

###### Lemma B.1 (From Yüce et al. ( 2022 , Appendix A.2) ) .

### B.2 Proof of Lemma 4.2

For notational simplicity, we denote $x:=W^{(2)}$.

First, we note that the first root of the $\ell$th order Bessel function, denoted by $j_{\ell,1}$, satisfies the inequality $j_{\ell,1}>\pi/2$ for all $\ell\in\mathbb{N}$. Next, recall the results of Ifantis & Siafarikas () that

$$ $\frac{x}{2(\ell+1)}<\frac{J_{\ell+1}(x)}{J_{\ell}(x)}<\frac{x}{2\ell+1},\qquad \forall x\in(0,\pi/2).$ (33) $$

Similarly, for $J_{\ell+2}(x)$, we have

$$ $\frac{x}{2(\ell+2)}<\frac{J_{\ell+2}(x)}{J_{\ell+1}(x)}<\frac{x}{2\ell+3}, \qquad\forall x\in(0,\pi/2).$ (34) $$

Multiplying two inequalities, we get the desired claim.

### B.3 Details of Section 4.1

In this subsection, we provide more precise details of gradient bound of SNFs, with simple L-layer, 1-width network. Note that the bound can be generalized to arbitrary width, with matrix product computation.

Let $L$-layer simple SNF, $f(x)$, as in the main paper:

$$ $f(x)=W^{(L)}\sin(W^{(L-1)}\sin(\cdots\sin(W^{(1)}x)\cdots)).$ (35) $$

The network is trained with MSE loss function with gradient-based method, then the gradient of weight of each layer is computed as:

$$ $\frac{\partial L}{\partial W^{(k)}}=\frac{\partial L}{\partial f}\cdot\frac{ \partial f}{\partial W^{(k)}}=R_{0}\cdot\frac{\partial f}{\partial W^{(k)}}, \quad\text{for}\quad k\in[1,L],$ (36) $$

where $R_{0}$ denotes the initial residual, i.e., $f(x)-Y_{\text{true}}$. We additionally assume that $R_{0}$ is independent from various initalizations techniques. Then, $\partial f/\partial W^{(k)}$ only effects to gradients. By the simple property of sine function: $\max_{x}\sin(x)=\max_{x}(d\sin(x)/dx)=1$ , the gradients are bounded with:

$$ $\displaystyle\left|\frac{\partial L}{\partial W^{(k)}}\right|$ $\displaystyle=R_{t=0}\prod_{m=1}^{L-k}\left(\mathbbm{1}_{L>k}W^{(L-m+1)}\cdot G _{k}(x)\cdot x^{L-k-1}+\mathbbm{1}_{L=k}G_{k}(x)\right)$ (37) $\displaystyle\leq R_{t=0}\prod_{m=1}^{L-k}\left(\mathbbm{1}_{L>k}W^{(L-m+1)} \cdot\max_{x}G_{k}(x)\cdot x^{L-k-1}+\mathbbm{1}_{L=k}\max_{x}G_{k}(x)\right)$ (38) $\displaystyle=R_{t=0}\prod_{m=1}^{L-k}\left(\mathbbm{1}_{L>k}W^{(L-m+1)}\cdot x ^{L-k-1}+\mathbbm{1}_{L=k}\right)$ (39) $\displaystyle\leq R_{t=0}\prod_{m=1}^{L-k}\left(\mathbbm{1}_{L>k}W^{(L-m+1)}+ \mathbbm{1}_{L=k}\right),$ (40) $$

where $G_{k}(x)$ denotes a composite sinusoidal function from the gradient derivation process and $\mathbbm{1}$ is a indicator function. Since the maximum value of a composite sinusoidal function is always equal to or less than 1, $|G_{k}(x)|$ is bounded by 1. Note that Eq. 40 is satisfied due to the bounds on input domain, i.e., $x\in[-1,1]$.

## Appendix C Details of NTK Analysis and beyond

### C.1 Preliminaries

The Neural Tangent Kernel (NTK) framework views neural networks as kernel machines, where the kernel is defined as the dot-product of the gradients from two different data point (Jacot et al., ). In other words, each element of NTK can be expressed as $K(x_{i},x_{j})=\langle\nabla_{\mathbf{W}}f(x_{i}),\nabla_{\mathbf{W}}f(x_{j})\rangle$. Assuming an infinitely wide network trained with an infinitesimally small learning rate (i.e., gradient flow) and using a specific parameterization (i.e., NTK parameterization), the NTK converges to a deterministic kernel that does not evolve during training. However, in practice, neural networks are composed of finite-width layers, which introduces challenges for analysis using NTK-style theory. In this context, we study the empirical Neural Tangent Kernel (eNTK), where we do not require the assumption of a stationary kernel. Recent researches (Kopitkov & Indelman, ; Ortiz-Jiménez et al., ; Baratin et al., ) show that, with eNTKs non-linearly evolve during training, this kernel can serve as a measure of the difficulty of learning a particular task.

### C.2 Kernel regression with gradient flow

First, we provide a well-founded spectral analysis of NTK regression. Assuming the network is trained with a constant training set $X$ and their label set $Y$ (i.e., $\left((x_{1},y_{1}),\cdots,(x_{N},y_{N})\right),~{}x,y\in\mathbb{R}$), where $f(X;W_{t})=f(W_{t})$, and the loss function $L$ is defined as MSE. We can then express the time-dependent dynamics of the function using the NTK. To begin, we start with the equation for vanilla GD:

$$ $\displaystyle W_{t+1}=W_{t}-\eta\nabla_{W}L(W_{k})$ (41) $\displaystyle\frac{W_{t+1}-W_{t}}{\eta}=-\nabla_{W}L\left(W_{k}\right).$ (42) $$

With infinitesimally small learning rate, i.e., gradient flow, then:

$$ $\displaystyle\frac{dW_{t}}{dt}$ $\displaystyle=-\nabla_{W}L(W_{t})$ $\displaystyle=-\nabla_{W}f(W_{t})\cdot(f(W_{t})-Y_{\text{true}}).$ (43) $$

Eq. 43 indicates the dynamics of the weights. Using a simple chain rule, we can modify the equation to represent the dynamics of the function, with the definition of NTK (i.e., $K_{t}\in\mathbb{R}^{N\times N}$):

$$ $\displaystyle\frac{df(W_{t})}{dt}=\frac{df(W_{t})}{dW_{t}}\cdot\frac{dW_{t}}{dt}$ $\displaystyle=-\langle\nabla_{W}f(W_{t}),\nabla_{W}f(W_{t})\rangle(f(W_{t})-Y_ {\text{true}})$ $\displaystyle=-K_{t}\cdot(f(W_{t})-Y_{\text{true}}).$ (44) $$

Note that the element-wise representation of NTK is can be written as $K_{t}(x_{i},x_{j})=\langle\nabla_{W}f(x_{i},W_{t}),\nabla_{W}f(x_{j},W_{t})
\rangle\in\mathbb{R}$, where $x_{k}$ denotes $k$-th data point in the entire training set $X$. Now, let $g(W_{t}):=f(W_{t})-Y_{\text{true}}$, then Eq. 44 can be rewritten as:

$$ $\displaystyle\frac{dg(W_{t})}{dt}=-K_{t}\cdot g(W_{t}).$ (45) $$

Eq. 45 is a simple ordinary differential equation (ODE) and can be solved in the form of exponential growth by assuming the NTK is constant during update; $K_{t}=K_{0}$:

$$ $\displaystyle g(W_{t})=e^{-t\cdot K_{t}}\cdot g(W_{0})$ (46) $\displaystyle f(W_{t})-Y_{\text{true}}=e^{-t\cdot K_{t}}\cdot(f(W_{0})-Y_{ \text{true}})$ (47) $\displaystyle f(W_{t})-Y_{\text{true}}=Qe^{-\Lambda t}Q^{T}\cdot(f(W_{0})-Y_{ \text{true}})$ (48) $\displaystyle Q^{T}(f(W_{t})-Y_{\text{true}})=e^{-\Lambda t}Q^{T}\cdot(f(W_{0} )-Y_{\text{true}}),$ (49) $$

where $Q$ is matrix of concatenation of eigenfunctions, which behaves a projecting operator, and $\Lambda$ is diagonal matrix contains non-negative eigenvalues. Additionally, equality in Eq. 48 is due to the property of spectral decomposition, i.e., $e^{K_{t}}=Qe^{\Lambda}Q^{T}$.

Eigenvectors compose orthonormal set in $\mathbb{R}^{N}$, and each residual components projected in each eigendirection. This directly indicates the convergence speed in each direction in $\mathbb{R}^{N}$. For example, we can explictly compute the remained error in direction $q_{i}$, denoted as $v_{i}$, at timestep $t$:

$$ $\displaystyle v_{t,i}=e^{-\lambda_{i}t}\cdot v_{0,i}$ (50) $$

- •
We can observe that once the eigenspace is defined (at the initial state), the eigenvalue governs the training speed in each direction of the basis. Specifically, after an arbitrary time step $t$, the projected error in the direction corresponding to large eigenvalues decreases exponentially faster than in the case of small eigenvalues.
- •
Another important aspect is understanding the characteristics of each direction. Empirical and theoretical evidence has shown that the NTK eigenfunctions corresponding to large eigenvalues are low-frequency functions. As a result, neural network training is biased towards reducing error in low-frequency regions (Ronen et al., ; Basri et al., ; Wang et al., ), providing strong evidence of *spectral bias* (Rahaman et al., ).
- •
A natural question arises, “what happens if the eigenvalue is large but the dot product between the corresponding eigenfunction and the residual is small?” This indicates that, despite the large eigenvalue, it contributes less to the reduction of the overall loss. This directly implies that we should consider not only the absolute scale of the eigenvalues but also the dot product (or alignment) between the eigenfunctions and the residual. In summary, we can conclude: *if there is high alignment in directions with large eigenvalues, training will be faster*.

Based on this analysis, let the timestep $\hat{t}$ that we want to evaluate the loss, then:

$$ $\hat{Q}^{T}(f(W_{t-\hat{t}})-Y_{\text{true}})=e^{-\hat{\Lambda}(t-\hat{t})} \hat{Q}^{T}(f(W_{\hat{t}})-Y_{\text{true}}).$ (51) $$

We evaluate the eNTK at every discrete timestep until the training ends. Specifically, we approximate the flow of error over the interval $t-\hat{t}\in[0,1]$. This setting allows the kernel to evolve during training and enables us to further explore the evolution of its eigenspace. For computation of NTKs in our work, we use ‘Neural Tangent Library’ (Novak et al., ).

### C.3 Non-kernel approach: analysis via Hessian eigenvalues

Understanding the structure of the Hessian is a crucial aspect in the field of optimization. The Hessian matrix represents the second-order derivatives of the loss function with respect to each parameter, and its eigenvalues provide insights into the local curvature of the model’s parameter space. The condition number of the Hessian, defined as $\lambda_{\max}/\lambda_{\min}$, directly affects the convergence rate and guarantees of quadratic optimization models. In deep learning, the Hessian is widely studied to better understand the complex optimization trajectories of deep neural networks. For instance, Ghorbani et al. () investigated outlier eigenvalues of the Hessian in neural networks (i.e., eigenvalues detached from the main bulk) and demonstrated through empirical and theoretical analysis that such outliers can slow down the convergence of the network.

Building on this line of research, in this section, we present the condition number (CN) of the Hessian during training in three different settings. Fig. 10 illustrates that the CN(^2^22For numerical stability, similar to Section 4.3, we compute the average of the top 3 eigenvalues for $\lambda_{\max}$ and the bottom 3 eigenvalues for $\lambda_{\min}$, where the scale of the eigenvalues are considered in terms of their absolute values. For Hessian computation, we use ‘PyHessian’ (Yao et al., ).) of WS-SNF remains significantly lower than that of other comparison models throughout training. This suggests that WS initialization leads to a better-conditioned Hessian not only in the kernel-approximated regime but also in the nonlinear regime, contributing to more efficient training dynamics.

Figure: Figure 10: Condition number of Hessian. The WS initialization consistently results in a lower condition number compared to other models, indicating that without the approximation of kernelized networks, the optimization path of WS-SNF is well-conditioned.
Refer to caption: x15.png

## Appendix D Frequency spectrum of 2D images

Fig. 4 presents a comparative analysis of the frequency spectrum between weight-scaled SNF and 1D audio signals. To further validate our findings, we extend our analysis to Kodak image dataset. Following the methodology of Shi et al. (), we partition the frequency map into several discrete bands and compute the mean values within each band.

Figure: Figure 11: Frequency distribution of SNFs and 2D signals. We partition the frequency map into 100 subgroups to facilitate the visualization of 2D signals in a 1D format. Natural images consistently exhibit similar frequency distribution patterns. Notably, WS-SNF demonstrates a better capability in representing high-frequency components, even in its initial state.
Refer to caption: x16.png

Fig. 11 illustrates the frequency distributions across three distinct images from Kodak dataset, as well as the initial output of SNFs. Our analysis reveals that natural images exhibit consistent spectral characteristics across the dataset, providing empirical support for our claim presented in Section 5.2. Moreover, we observe that the initial functional generated by weight-scaled SNF demonstrates a notably higher frequency distribution, consistent with our theoretical analysis.

## Appendix E Experimental details and qualitative results

### E.1 Training settings and baselines

In this section, we provide detailed information about our experiments. For data fitting tasks, we used the Adam optimizer (Kingma, ) with a learning rate of 0.0001, without any learning rate scheduler, except for the occupancy field experiments (in which case, we use PyTorch learning rate scheduler). Each experiment was conducted with 5 different seeds, and we report both the average and standard deviation of the evaluation metric. We note that all baselines in this work are MLP architectures, with slight modifications to the input features or activation functions. We used NVIDIA RTX 3090/4090/A5000/A6000 GPUs for all experiments. Additionally, we set the hyperparameters of each network using a grid search, as shown in LABEL:tab:details.

ReLU + P.E. (Mildenhall et al., ): For this architecture, we use fixed positional encoding applied to the input $x$, with $K$ frequencies that lift the dimension of the inputs, i.e., $\gamma(x)=\left(\sin(2^{0}\pi\cdot x),\cos(2^{0}\pi\cdot x),\ldots,\sin(2^{K-1
}\pi\cdot x),\cos(2^{K-1}\pi\cdot x)\right)$.

FFN (Tancik et al., ): For this architecture, we use Gaussian positional encoding applied to the input $x$, i.e., $\gamma(x)=[\sin(2\pi G\cdot x),\cos(2\pi G\cdot x)]$, where $G$ defines the variance of the Gaussian distribution.

GaussNet (Ramasinghe & Lucey, ): For this architecture, we use the Gaussian activation function, i.e., $\sigma(Z)=\exp\left(-(s\cdot Z)^{2}\right)$, where $Z$ denotes the pre-activation matrix and $s$ is a hyperparameter that defines the bandwidth of the Gaussian function.

WIRE (Saragadam et al., ): For this architecture, we use a Gabor wavelet-based activation function, i.e., $\sigma(Z)=\exp(j\omega\cdot Z)\exp(-(s\cdot Z)^{2})$, where $\omega$ and $s$ are hyperparameters.

SIREN (Sitzmann et al., ): We use the sine activation function $\sigma(z)=\sin(\omega\cdot Z)$, where $\omega$ in the first layer is a tunable hyperparameter.

Neural field scaling law (Saratchandran et al., ): We modify the initialization distribution of the last layer of SIREN, i.e., $W^{(L)}\sim\mathcal{U}\left(-\sqrt{6}/\left(n^{3/4}_{L-1}\omega\right),\sqrt{6
}/\left(n^{3/4}_{L-1}\omega\right)\right)$, where $n_{L-1}$ denotes the width of the $(L-1)$-th layer.

Xavier uniform initialization (Glorot & Bengio, ): In this case, we use the standard Xavier uniform initialization for the SNF architecture, i.e., $W^{(l)}\sim\mathcal{U}\left(-\sqrt{6}/\sqrt{n_{l-1}+n_{l}},\sqrt{6}/\sqrt{n_{l
-1}+n_{l}}\right),\quad l\in[1,L]$.

**Table 2: Detailed information about hyperparameters. We provide the exact hyperparameter settings for each domain in the table below. ‘default’ in ReLU+P.E. denotes the using nyquist sampling methods (Saragadam et al., ).**
|  | ReLU+P.E. | FFN | SIREN | GaussNet | WIRE | Weight scaling |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | k | $\sigma$ | m | $\omega$ | $s$ | $\omega$ | $s$ | $\alpha$ |
| Image | 10 | 10 | 256 | 30 | 10 | 20 | 10 | 2.37 |
| Occ. Field | default | 2 | 10 | 10 | 10 | 10 | 40 | 3.7 |
| Spherical Data | default | 2 | 256 | 30 | 10 | 10 | 20 | 2.5 |
| Audio | default | 20 | 20 | 3000 | 100 | 60 | 10 | 2.0 |

### E.2 Neural dataset experiements

In recent research on constructing large-scale NF datasets (Ma et al., ; Papa et al., ), there has been growing interest in this area. However, no studies have investigated how initialization schemes affect the overall process, from constructing SNF-based neural datasets to their application in downstream tasks. To address this gap, we first measure the time required to fit the entire dataset to NFs and then report the classification accuracy on the neural dataset while varying the scale factor $\alpha$. The results are shown in Table 3.

Analysis. Interestingly, despite the efficiency of WS-SNFs in terms of training time (i.e., the time required to fit each datum into an individual NF until reaching the target PSNR), the test accuracy (i.e., classification accuracy on the test set SNFs), specifically for the case of $\alpha=1.5$, does not significantly decrease. Moreover, we observed that, in the case of $\alpha=2.0$, test accuracy dropped by an average of 5 percentage points across all PSNR values. This mirrors a similar phenomenon discussed in Papa et al. (), where constraining the weight space of NFs was identified as a key factor in downstream task performance. More precisely, scaling initialization distributions causes the neural functions to lie in a larger weight space, making it harder to learn features from the weights. Furthermore, we propose an interesting research direction: *can we create a neural dataset quickly without compromising the performance of downstream tasks and without incurring additional cost?*

Experimental details. For training, we used ‘fit-a-nef’ (Papa et al., ), a JAX-based library for fast construction of large-scale neural field datasets. We used the entire dataset for NF dataset generation (i.e., 60,000 images for MNIST and CIFAR-10, respectively). Additionally, we used an MLP-based classifier targeted at classifying each NF’s class, which consists of 4 layers and a width of 256, and was trained with cross-entropy loss for 10 epochs.

**Table 3: Neural dataset experiments. Each neural dataset is trained until it reaches the batch-averaged target PSNR. The training time refers to the total time required to construct the entire INR dataset. A classifier is then trained on the neural dataset, and the resulting test accuracy is reported for each configuration. We use SIREN as the NF architecture, which is a widely used baseline in this area.**
|  |  | Neural MNIST | Neural CIFAR-10 |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Target PSNR | Training time | Test accuracy | Training time | Test accuracy |
| SIREN | 35 | 14m 23s | 97.95$\pm$0.00 | 19m 14s | 47.98$\pm$0.01 |
| 40 | 18m 20s | 97.76$\pm$0.00 | 29m 45s | 48.46$\pm$0.01 |  |
| 45 | 23m 23s | 97.53$\pm$0.01 | 47m 54s | 48.09$\pm$0.01 |  |
| WS ($\times$ 1.5) | 35 | 7m 22s | 97.56$\pm$0.00 | 10m 52s | 47.15$\pm$0.02 |
| 40 | 8m 22s | 97.81$\pm$0.00 | 17m 21s | 47.72$\pm$0.01 |  |
| 45 | 10m 41s | 97.02$\pm$0.01 | 25m 52s | 46.08$\pm$0.03 |  |
| WS ($\times$ 2.0) | 35 | 5m 25s | 97.48$\pm$0.01 | 8m 26s | 43.30$\pm$0.02 |
| 40 | 5m 32s | 97.06$\pm$0.01 | 9m 20s | 42.86$\pm$0.02 |  |
| 45 | 5m 41s | 97.49$\pm$0.00 | 10m 51s | 42.70$\pm$0.01 |  |

### E.3 Training for additional iterations

Weight scaling initialization primarily focuses on the training efficiency of neural fields. In other words, the experiments in the main paper were conducted over a limited number of epochs. In this subsection, we provide qualitative results from models trained for additional iterations. As shown in Table 4, networks with WS initialization continue to outperform other networks, suggesting that its effectiveness is consistently preserved throughout the training process.

**Table 4: Quantitative results: training for additional iterations.**
|  | Image (300 iterations) | Occ. Field (100 iterations) | Spherical Data (15k iterations) | Audio (5k iterations) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Activation | KODAK | DIV2K | Lucy | ERA5 | Bach |
| Xavier Uniform (Glorot & Bengio, ) | Sinusoidal | 2.31 | 2.20 | 0.0706 | 7.10$\pm$1.02 | 16.16 |
| ReLU + P.E. (Mildenhall et al., ) | ReLU | 21.77 | 19.60 | 0.9920 | 41.32$\pm$0.16 | 32.40 |
| FFN (Tancik et al., ) | ReLU | 26.51 | 25.62 | 0.9874 | 45.27$\pm$0.12 | 17.54 |
| SIREN init. (Sitzmann et al., ) | Sinusoidal | 26.01 | 24.76 | 0.9926 | 44.42$\pm$0.32 | 46.78 |
| GaussNet (Ramasinghe & Lucey, ) | Gaussian | 21.90 | 20.22 | 0.9935 | 44.15$\pm$0.12 | 34.65 |
| WIRE (Saragadam et al., ) | Wavelet | 31.47 | 30.89 | 0.9939 | 45.13$\pm$0.42 | 20.17 |
| NFSL (Saratchandran et al., ) | Sinusoidal | 26.91 | 25.97 | 0.9933 | 45.10$\pm$0.08 | 45.70 |
| Weight scaling (ours) | Sinusoidal | 52.73 | 53.10 | 0.9950 | 49.63$\pm$0.14 | 54.04 |

### E.4 Qualitative results

In this subsection, we provide qualitative results of experiments in Section 5.1. We do not provide reconstructed results for occupancy field reconstruction with Xavier initialization case due to training failure.

Figure: (a) Ground truth
Refer to caption: extracted/5906275/assets/fig/image_recon/kodim05_512_grayscale.png

Figure: (a) Ground truth
Refer to caption: extracted/5906275/assets/fig/occ_field/occ_gt.jpg

Figure: (a) Ground truth
Refer to caption: extracted/5906275/assets/fig/audio/gt.jpg

Figure: (a) Ground truth
Refer to caption: extracted/5906275/assets/fig/era5/gt.jpg

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - See weaknesses
soundness: 2
strengths: - The motivation is clear, and the exposition is easy to follow.
- Although not very surprising, the study of the theoretical behavior of weight scaling is fairly comprehensive and I appreciate the authors efforts to draw upon different theoretical tools in existing literature to provide a broad picture.
summary: This work proposes an alternate initialization scheme for sinusoidal neural fields. The main claim is that scaling the initialization weights by a constant value greater than one can help speed up the learning in SNFs as well as improve fitting for higher frequencies, by improving the function at initialization as well as boosting optimization trajectories. Empirical validation on various domains is presented to substantiate the arguments.
weaknesses: - The current choice of experiments is rather limited. Since the contribution is largely methodological, experiments on important practical applications of SNFs, specifically NeRF and solving differential equations (for instance, see Saratchandran et al., 2024), as well as larger datasets, are needed to justify the claims. 
- Experimental methodology: A few aspects of the experimental methodology are unclear and/or need revision.
    - Baseline: The included baseline with Xavier initialization is largely irrelevant from a practical standpoint. Alternatively, Multiplicative Filter Networks (Fathony et al, 2021) is an important baseline to consider, especially since multiplicative networks, akin to weight scaling, are known to speed up learning for higher frequencies [4] and thus, directly competes with weight scaling in terms of spectral bias.
    - Fixed Iterations: The choice of number of training iterations to report performance is arbitrary and could reflect unfairly on the baselines. The authors should consider including the PSNR curves for training until convergence (for both WS and baselines) for understanding practical feasibility, since ground truth data to stop at a target PSNR is not available in practice.
    - Full Batch Training/Susceptibility to SGD Noise: In practice, it is often not possible to perform full batch training due to size of dataset, network size etc. How does the noise in moving to stochastic (batched) gradient descent affect the convergence of weight scaling? Experiments that reflect those settings will help understand better the limitations of the proposed framework. 
- Related work: Missing citations for Multiplicative Filter Networks (Fathony et al, 2021) as well as spectral bias literature:
    - [2] The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies, Basri et al., NeurIPS 2019
    - [3] Towards Understanding the Spectral Bias of Deep Learning, Cao et al., 2020
    - [4] The Spectral Bias of Polynomial Neural Networks, Choraria et al., ICLR 2022

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: - In Figure 2(b), the relationships between test PSNR and acceleration plotted for both cases of frequency tuning and weight scaling do not seem to be one-to-one. Why is that so? Am I missing something there?
- In Eq. (3), the initialization scheme involves scaling the uniform distributions. Will similar weight scaling trick achieve similar acceleration results when the distributions involved are non-uniform (say Gaussian)?
- How does the speed-up offered by the proposed weight scaling scheme depend on/interact with the choice of optimizers (say SGD vs Adam)? Is it more effective for certain choices of optimizers? 
- For the results in Figure 3, the learning rates vary with layers and are scaled down to match the rates of unscaled SNF. Can you provide more details on how they vary with depth? Also, what if the same learning rate is used for all layers?
- Can you provide more details on how the optimization problem (10) is solved? Can we treat $\alpha$ as a trainable parameter instead?
soundness: 3
strengths: - Overall the paper is well written, with sufficient number of plots provided to visualize the findings.
- The proposed weight scaling scheme is simple yet seems to be quite effective in speeding up training.
summary: This paper studies initialization schemes for the class of sinusoidal neural fields (SNFs) and proposes a simple weight scaling method to accelerate training across different data domains. The idea is to simply scale the initial weights of all layers in the neural net by a well chosen factor. Some theoretical results and experimental results are provided to support and demonstrate the proposed method.
weaknesses: - The theoretical results are quite limited. For instance, to show that the proposed weight scaling increases the relative power
of the higher-frequency bases, the paper considers the effects of such scaling on a 3-layer SNF only for the case when the width is one. 
- While the optimization trajectories are studied via the lens of the empirical neural tangent kernel, it would be more convincing if there are theoretical/empirical results that quantify the impact of such scaling on the gradient loss and the speed of convergence of gradient descent. 
- There are also some missing details and confusing parts in the paper that would need to be addressed in order to make the presentation stronger (see my questions below).

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 4
questions: * $\textbf{Impact on Long-Term Generalization}$: How does WS affect long-term generalization on larger datasets or in cases where SNFs are used for predictive tasks (e.g., continuous function approximation or spatial data interpolation)?
soundness: 3
strengths: * $\textbf{Effective Speed Improvement}$: WS offers a substantial acceleration in training time without degrading model performance or generalization capabilities, which is valuable for SNF applications requiring rapid processing. 
* $\textbf{Simplicity and Practicality}$: The method requires minimal changes to existing architectures, making it straightforward to apply WS to SNFs without complex modifications. 
* $\textbf{Well-Written}$: The paper has been written in detail.
* $\textbf{Theoretical Justifications}$: The paper offers a detailed theoretical framework explaining why WS is effective, especially in reducing spectral bias and improving the conditioning of the optimization trajectory. This theoretical result supports the empirical findings and strengthens the reliability of the results.
summary: The work introduces a method to accelerate training in sinusoidal neural fields (SNFs), focusing on a technique called weight scaling (WS). This method modifies the initialization by scaling the initial weights of SNFs (except for the final layer) by a constant factor. By improving the frequency characteristics of SNFs, WS significantly reduces training time—up to 10 times faster than the traditional approach—while maintaining generalization. The authors provide theoretical insights and empirical validation across tasks in some data domains, including images, 3D shapes, and audio signals. They compare WS with standard initialization and frequency tuning, demonstrating its advantages in both speed and stability during training. The idea behind this method is quite interesting, as it revolves around reducing time to create more efficient models; however, there are some significant weaknesses and gaps that cannot be ignored.
weaknesses: The authors acknowledge the main limitations of their work, which cannot be ignored. To interpret the results accurately, these limitations must be carefully considered.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1). Your paper is entirely focused on neural fields that use a sinusoidal activation. The weight scaling method you propose scales the initialization of a sinusoidal neural field by a certain factor $\alpha$, as per eqn (6) in your paper. This method takes the SIREN initialization as its basis. Could you explain what happens if we take a standard initialization scheme like Kaiming uniform or Xavier uniform and scale it in exactly the same way as your eqn. (6)? Does this lead to anything new? Why did you apply your weight scaling to only the SIREN initialization?

2). Could you explain whether your results would go through for other activations that are used within the neural fields literature such as Gaussian [1], Wavelet [2], Sinc [4]? 

3). Did you try weight scaling on a NeRF task? I am very surprised there are no NeRF results as this a big experiment that requires hours to train and I noticed some of the papers you are comparing against [1], [2], [3] compare to NeRF. Could you explain whether weight scaling would or wouldn't work on NeRF and if it cannot be made to work on NeRF then why? I think it would really help your paper if you had NeRF results and you were able to show large speed ups on this large scale experiment.

4). Your weight scaling essentially just scales the initialization of the inner layers of the initialization by Sitzmann et al for SIRENs. However, if you look at the denominator of eqn. (3), line 194, there is the term $\omega_h$ which is supposed to be the frequency of the sinusoid being used on that layer. Could we not get the same effect as your initialization by using a sinusoid with frequency 
$\frac{\omega_h}{\alpha}$? 

5). In appendix C.3 you analyse the condition number of the Hessian of the loss landscape. You mention that the condition number of the Hessian directly affects the convergence rate. However, my understanding is that this is only true for convex loss landscapes. Loss functions associated to neural networks are almost always non-convex, as they employ non-linearities, therefore in such cases the condition number of the Hessian may not be well defined, as the Hessian may have negative eigenvalues or zero as an eigenvalue. Could you comment on this? I don't understand this analysis you have provided in your paper.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[1] Ramasinghe et al: Beyond Periodicity: Towards a unifying framework for activations in coordinate-MLPs; ECCV 2022.    

[2] Saragadam et al: WIRE: Wavelet implicit neural representations; CVPR 2023.

[3] Saratchandran et al: Scaling insights for optimizing neural fields: CVPR 2024.

[4] Saratchandran et al: A Sampling Theory Perspective on Activations for Implicit Neural Representations: ICML 2024.
soundness: 3
strengths: **Originality:** The paper introduces an original idea that I have never seen used before in the neural field literature. Furthermore, the authors take the time to give a good theoretical analysis of why weight scaling works and how it relates to other initialization methods within the literature.

**Quality:** The paper has both theoretical and experimental contributions making it a good quality paper written by the authors. The theoretical contributions are sound and the experiments are compared with recent literature.

**Clarity:** The paper is in general well written. The authors take the time to exposit the main ideas behind weight scaling well and explain their key contributions in a readable manner. I have checked their mathematical proofs in the appendix and they were all well written and explained well.

**Significance:** The paper gives a new way to initialize sinusoidal neural fields which I believe is interesting to the community of researchers working in neural fields. The initialization of neural fields in general is not as commonly studied as other deep networks such as CNNs or Transformers and apart from a few recent works on the topic, I believe this paper offers a nice viewpoint in how sinusoidal neural fields should be initialized for efficient training.
summary: This paper presents a study on sinusoidal neural fields (SNFs) and introduces "weight scaling," a method that accelerates training by adjusting the initialization of each layer, except the final one. The key idea behind weight scaling is straightforward: each layer’s weights are initialized by multiplying with a specific scale factor, which results in faster training across various neural field tasks. The authors benchmark their approach against existing methods, including the original SNF initialization by Sitzmann et al., and demonstrate that weight scaling consistently achieves faster convergence while maintaining robust training and test performance. Furthermore, the authors provide theoretical analysis explaining why weight scaling works.
weaknesses: **Novelty:** My main issue with the paper and what I feel is its main weakness is the fact that it is entirely focused on one type of neural field, namely sinusoidal neural fields. The paper does not say anything about other neural fields in general. The authors do say that their methods would not work on ReLU based neural fields due to ReLU having the positive homogeneity condition and
while the authors do say that sinusoidal neural fields are used within the community I would argue that there are several papers showing that sinusoidal neural fields are not as effective as other neural fields, such as [1], [2], [3], [4].  While the paper is written well I do feel that it being entirely focused on one type of neural field means it won't be as useful to many researchers in the neural fields community. 

**Large Scale Experiments:** The paper compares its methodology of weight scaling on a tasks such as image regression, binary occupancy, spherical data, audio data and fit a nef (in the appendix). Yet I noticed there were no experiments on Neural Radiance Fields (NeRFs) which I found very surprising. The authors say that their method is very effective at reducing the training time of neural fields which is generally large yet many of the tasks such as image regression, audio, occupancy are not considered big tasks within the community. On the other hand NeRF is a big experiment that takes anywhere between 4 hours to 12 hours to train, depending on the fidelity the user wants. I would have liked to see how their method did on such a task that requires hours of training. I also looked at the papers the authors compare to namely [1], [2], [3] and noticed that these papers compared their methods on NeRF. In the appendix the authors do compare their method on "fit a nef" on a large scale dataset but these takes less than 1 hour to train even with a SIREN. Thus I feel another big weakness of the paper is that the authors did not compare their method on a task that takes hours to train such as NeRF, so that we can see how their method does on a more complicated large scale neural field.

[1] Ramasinghe et al: Beyond Periodicity: Towards a unifying framework for activations in coordinate-MLPs; ECCV 2022.    

[2] Saragadam et al: WIRE: Wavelet implicit neural representations; CVPR 2023.

[3] Saratchandran et al: Scaling insights for optimizing neural fields: CVPR 2024.

[4] Saratchandran et al: A Sampling Theory Perspective on Activations for Implicit Neural Representations: ICML 2024.

Meta-review: This paper proposes a novel weight scaling initialization scheme for SNFs that speed ups training by scaling the weights of all layers by a constant factor. The authors provide a theoretical analysis and empirical evidence showing that this method improves convergence speed while maintaining or enhancing generalization. The reviewers are overall positive and identified the following strengths and weaknesses:

### **Strenghts**
- The proposed method is straightforward and can be applied without major architectural modifications.
- Demonstrates significant acceleration in training times across various tasks (image regression, 3D shapes, and audio signals) without sacrificing generalization.
- Provides a sound theoretical basis for why weight scaling improves training, including insights into spectral bias and empirical neural tangent kernel analysis.
   - Highlights how weight scaling preserves distributional properties of activations across layers.

### **Weaknesses**
- Work is focused on SNFs and some reviewers argue that SNFs are less commonly used than other neural field architectures in practice.
- Missing or inadequate baselines, such as Multiplicative Filter Networks (MFNs), were highlighted initially. The authors later added this baseline.
- The theoretical results primarily focus on a specific case

### **Reviewer Scores:**
- All reviewers rated the submission marginally above the acceptance threshold, with increased scores after the authors addressed concerns through rebuttals and additional experiments.

The proposed method demonstrates some interesting empirical and theoretical results for accelerating SNFs, but its narrow focus and limited novelty may restrict its impact. The authors addressed many reviewer concerns, including missing experiments and theoretical clarifications, strengthening the manuscript. I therefore recommend acceptance but I highlight there remains some skepticism about its broader applicability and significance, which I encourage the authors to discuss openly in the revised version of the paper.

Decision: Accept (Poster)