A New Initialization to Control Gradients in Sinusoidal Neural Networks
Authors: Andrea Combette
Nelly Pustelnik
Antoine Venaille
Venue: ICLR 2026 Poster (2026)
Keywords: Initialization Strategy
Deep Neural Networks
Sinusoidal Activations
Gradient Control
Implicit Neural Representations
Neural Tangent Kernel
Primary area: learning theory
TL;DR: We propose a closed form expression for parameters initialization in SIREN networks to control gradient

Abstract: Proper initialisation strategy is of primary importance to mitigate gradient explosion or vanishing when training neural networks. Yet, the impact of initialisation parameters still lacks a precise theoretical understanding for several well-established architectures. Here, we propose a new initialisation for networks with sinusoidal activation functions such as \texttt{SIREN}, focusing on gradients control, their scaling with network depth, their impact on training and on generalization. To achieve this, we identify a closed-form expression for the initialisation of the parameters, differing from the original \texttt{SIREN} scheme. This expression is derived from fixed points obtained through the convergence of pre-activation distribution and the variance of Jacobian sequences. Controlling both gradients  and targeting vanishing pre-activation helps preventing the emergence of inappropriate frequencies during estimation, thereby improving generalization. We further show that this initialisation strongly influences training dynamics through the Neural Tangent Kernel framework (NTK). Finally, we benchmark \texttt{SIREN} with the proposed initialisation against the original scheme and other baselines on function fitting and image reconstruction. The new initialisation consistently outperforms state-of-the-art methods across a wide range of reconstruction tasks, including those involving physics-informed neural networks.

Published as a conference paper at ICLR 2026
A NEW INITIALISATION TO CONTROL GRADIENTS IN SINUSOIDAL NEURAL NETWORK
Andrea Combette, Antoine Venaille & Nelly Pustelnik CNRS, ENS de Lyon
Laboratoire de Physique UMR5672
Lyon, 69007, France
(andrea. combette, antoine. www.oineeeenaillee.. venaille, nelly nel.ly.puutennikkeens lyon.
ABSTRACT
Proper initialisation strategy is of primary importance to mitigate gradient explo- sion or vanishing when training neural networks. Yet, the impact of initialisa- tion parameters still lacks precise theoretical understanding for several well- established architectures. Here, we propose new initialisation for networks with sinusoidal activation functions such as SIREN, focusing on gradients control, their scaling with network depth, their impact on training and on generalization. To achieve this, we identify a closed-form expression for the initialisation of the pa- rameters, differing from the original S SIREN scheme. This expression is derived from fixed points obtained through the convergence of pre-activation distribution and the variance of Jacobian sequences. Controlling both gradients and targeting vanishing pre-activation helps preventing the emergence of inappropriate frequen- cies during estimation, thereby improving generalization. We further show that this initialisation strongly influences training dynamics through the Neural Tan- gent Kernel framework (NTK). Finally, benchmark SIREN with the proposed initialisation against the original scheme and other baselines on function fitting and image reconstruction. The new initialisation consistently outperforms state- of-the-art methods across a wide range of reconstruction tasks, including those involving physics-informed neural networks.
INTRODUCTION
1.1 CONTEXT AND MOTIVATION
Implicit neural representations (NNRs) have become prevalent tool for approximating functions in diverse applications, including signal encoding (Strümpler et al. 2022 Dupont et al. (02T), signal reconstruction (Park et al. 2019 Mildenhall et al. 2020). and solutions of partial differential equa-
tions (PDEs) (Raissi et al. 2019). ). A central challenge in these neural approximations is to recover the frequency spectrum of a target signal within reasonable training time and from limited data. In this context, standard multi-layer perceptrons (MLPs) used for INRs often suffer from spectral bias, whereby low-frequency components are preferentially learned compared high- gighffreuencyy details Rahaman et al. 20199 ei al. 2024). This bias can hinder performance, either by slowing train- ing by reducing precision, when the signal of interest contains significant high-frequency content (fine textures, details,... ). To mitigate this issue, several architectures have been proposed, such as positional encoding (Tancik al. 2020) or networks with sinusoidal activation functions (SIREN, (Sizzmann et al.] 20200). which enable faster learning of high- frequeeyyyncy components. However, increasing network depth in these methods has been empirically observed introduce the re- constructed function spurious high-frequency components absent from the target one (sce, e.g., (Ma ctt al. 2025)), leading to noisy representations and degraded generalization (i.c., the ability to inter- polate the signal correctly).
In this work, we propose an initialisation strategy for SIREN that bypasses two opposing pitfalls: (i) slow training and poor recovery of high-frequency details due to spectral bias in standard MLPs, and (ii) rapid training in deeper SIREN, which comes at the cost of spurious high-frequency artifacts and degraded generalization. Finding the right balance between these extremes corresponds

---

Published as a conference paper at ICLR 2026
to locating the frontier between vanishing- aanishing-gradient and exploding-gradient regimes. Operating in this regime, where gradients remain stable, is often referred to as computing at the edge of chaos Yang & Schoenholz 2017; Scleznova & Kutyniok| 2022), concept from dynamical systems the- ory (Kelso et al. 1986, Langton 1986p. Building on this idea, we introduce an explicit initialisation scheme for STREN. Our method ensures that inputs and parameters gradients neither vanish nor explode with depth enabling both stable and expressive learning dynamics. With appropriate tun- ing of the pre-activation statistics it allows to impose finite range of frequency at initialisation, allowing the network to capture high-frequency contents without introducing spurious components. To better understand the critical role of the initialisation in INRs, we complement
ReLU (PE)
our theoretical analysis with experiments based on the neural tangent kernel (NTK) framework (Jacot et al., 2018: Li et al. 2024]. We find that controlling gradient propagation at initialisation strongly influ- ences the NTK eigenvalues, which deter- mine the training speed of the frequencies associated with the corresponding eigen- vectors.
Tanh (Fourier-Xavier)
Finer
Wire
SIREN (Sitzmann)
SIREN (7u = 1)
SIREN (Proposed)
10
10-3 10-2 10
10
Beyond the NTK framework, our initial- isation prevents the degradation of deep neural network performance with increas- ing depth. We illustrate this property across several function fitting problem Figure [ where comparison of the per- formance of our initialisation against the original SIREN scheme and other base- lines is presented.
Figure 1: Generalization enror over different problems averaged over different architecture depths for 1d, 2d and 3d multi-scale function approximation. The results are displayed for different state-of-the-an art architectures including the one proposed in this work (SIREN Pro- posed). See Appendix [B.6.3 for details. In standard deviation of the error is colored in light gray.
1.2 RELATED WORK
Frequency representation. Our study will be based on the work of (Sitzmann et al. 2020). which introduced the S STREN architecture, neural network with sinuosidal activation functions designed to effectively learn high-frequency functions by using a tunable parameter Wo that controls the fre- quency range of the network. Architecturally, this approach is closely related to positionall-ncodinn and Random Fourier feature, which also address the challenge of learning high-frequency signals (ancik et al. 2020 Wang et al., (02TT. However, SIREN requires careful tuning of wip depending on both the network architecture and the dataset (de Avila Belbute- & Kolter 2023). Moreover, the effect of network depth on performance remains poorly understood and has so far been studied
mostly through empirical and observational analyses (Cai et al. 2024 Tancik et al. 2020). To the best of our knowledge, there is currently no work connecting theoretical gradient scaling with depth to the performance of such architectures.
Neural Tangent Kernel. The NTK framework provides theoretical foundation for understanding the training dynamics of neural networks, and how the initialisation properties affect the learning process (Jacot et al., 2018 Li et al., 2024 Yūce et al., 2022). Some works have already focused on the frequency learning aspect of the NTK, either for the Fourier Features (Wang al., (02T) or the SIREN architecture (de Avila Belbute Peres & Kolter 2023 These works have shown how the network architecture can be tailored to bypass the spectral bias. However they did not provide a full understanding of the impact of network depth on the networks properties and did not tackle the vanishing or exploding gradient impact on the learning dynamics.
Initialisation. Our focus on initialisation is closely related to the work Glorot & Bengio (2010) and He et al. (2015), which introduced the now widely used Xavier and Kaiming initialisation meth- ods, respectively. Both approaches aim to maintain stable activation and gradient distributions across layers. Xavier initialisation
developed for saturating nonlinearities such as hyperbolic tangent (Tanh), and is motivated by theoretical insights into variance preservation, though its derivation assumes approximate linearization. Kaiming initialisation was later introduced for rectified lin-
2

---

Published as a conference paper at ICLR 2026
units (ReLU), which allows for exact variance calculations. Although commonly applied to smoother activation functions such as GeLU or SiLU, its theoretical justification in those cases is only approximate. In the context of SIREN, tailored initialisations have been proposed (Sitzmann et al. 2020; de Avila Belbute-Peres & Kolter 2023) to control the distribution of pre-activations layer by layer. However, these initialisations tees for deep SIREN architectures, where gradient growth remains uncontrolled, as we shall see later in this work. We also note the recent work Novello et al. (2025), which identified the same issues and proposed an empirical method to control network's 's spectrum. However, their approach does not provide principled control of either the frequency spectrum or the network gradients, leading to significant adaptation effort for each problem.
only approximate and fail offer stability guaran-
Edge of Chaos (EOC). EOC is the critical initialisation regime where two key conditions are met: forward pre-activation variances remain stable, and backward gradients neither explode nor vanish. In the infinite-widie mean-field limit, these properties follow from coupled recursions for layer-wise variance and inter-sample correlations under the initialisation distribution, whose fixed points determine both activation and gradient stability (Poole et al.) 2016, Schoenholz et al. 2017). Yang & Schoenholz (2017) showed that placing conventional networks
the EOC improves fixed-point initialization was previously analyzed in the EOC framework, with asymptotic Jacobian and correlation calculations for wide range of
training performance. While
activations (Hayou et al. 2019 (Seleznova & Kutyniok] 2022. Hayou et al. 2022), including sine (Roberts et 2002), our work uniquely derives an explicit criticality expression for sine, result unattainable for other activations in the sigmoid class.
1.3 CONTRIBUTIONS
This works brings a deeper understanding over INR initialisation for signal representation and train- ing dynamic, with the following main contributions:
An explicit derivation of the initialisation for the SIREN architecture, which allows us to have an invariant distribution of the gradients across the layers and possibly depth- independent Fourier spectrum. This done by calculating the fixed point for the layer-wise gradient and the network output distribution, in the limit of infinite width and infinite depth.
The understanding of the key concepts for controlled frequency learning using wi, and how the initialisation properties, through the NTK, shape the training dynamics of the network, leading to controlled spectrum of the learned function.
A series of experiments presented in Appendix B demonstrates the effectiveness of the proposed initialisation scheme multi-dimensional and multi multi-frr function approx- imation, including audio signals, image denoising, and video reconstruction on the ERA-5 atmospheric reanalysis dataset. We further investigate the impact of this initialisation in the context of PDE solving using physics-informed neural networks.
Although our motivation comes from INRs, the proposed closed-fore initialisation for sine networks the edge of chaos is not specific this setting. Because stabilizes gradient propagation in deep architectures with periodic activations, it may also benefit broader applications where periodic features are desirable but have been limited by unsuitable initialisation schemes.
PRELIMINARIES
2.1 GENERALITIES ON IMPLICIT REPRESENTATION OF FUNCTIONS
Implicit neural representations have been introduced to find an approximation of function f:Ω + R from dataset D = {(x, y:) KeI y = f(x))) The goal is then to build a parametrized function V Ω + IRd. When this parametrized function is neural network, it is commonly referred as implicit neural representation (INR), Neural Fields (NerF), or Neural Implicit Functions.
In this work, we formally denote the involved neural network Ve, which can be written as the composition L layers:
V ho, ho,
3

---

Published as a conference paper at ICLR 2026
where each layer & E {l...., L is composed of na neurons, parameterized by set of parameters 0g = (We, be) where We Ruex"e are the weights and by E R the bias, and no denotes the imput dimension of the network. Each layer also relies activation function dy applied element-wise. The 8-th layer is thus defined by
hag = oz O (We + be).
(2)
For an input æ E R2 the preactivation refers to
Wehe- b where
he-1 ho--1 o... ho, (x))
The estimation of the parameters 0 ((e))E().....)) relies on the minimization of loss L over
dataset ID = { vi)ker):
min L(0) = [
Σ REI
Vo((); - vil12
(4)
The main challenges when considering INRs include selecting an appropriate architecture (i.e., parametrization and activation function), choosing a suitable initialization insure output stability, and determining and efficient optimization strategy. In this work, will focus on S SIREN archithec- tures (described in the next section). Regarding minimization strategy, we focus on gradient-base methods, leaving alternative minimization strategies outside the scope of our study.
2.2 CHOICE OF THE ARCHITECTURE
This work focuses on the so called STREN architecture, which stands for Sinusoidal Representation Network and introduced by Sitzmann et al. (2020). SIREN is partieular instance of equations with a final linear layer:
(x(x) = W, sin Wr-1 sin(. ( + b,)) + + b₁.
(5)
This architecture enables the estimation of natural frequency decompositions in broad range of problems while ensuring differentiabilitty The latter property particularly important for PDE- related applications, such as physics-informed neural networks, where accurate derivatives are often essential (Raissi et al. 2019).
WEIGHT INITIALIZATION
In the original STREN initialization (Sitzmann et al. 2020), the weights and biases were chosen
u
wa no
We ag
f=1,
b ~ u( NN ) E LJ.
We
(6)
u
LE (2, L},
where N n4 is the number of neurons per hidden layer, assumed to be the same across all layers, L - 1 is the number of hidden layers, and U denotes the uniform distribution. wo is an important tunable parameter, originally chosen to be must be adjusted according the network architecture and the Nyquist frequency of the signal to be reconstructed (de Avila Belbute-Peres & Kolter 2023).
Sitzmann et al. (020)) argued that the pre-activation of the l-th layer, defined in equation 3 fol- lows the distribution Ze ~ (0, 1), when the network is initialized following equation In this regime, most of the signal is sufficiently small to propagates through the quasi-linear range of the sine activation function, while still preserving a meaningful nonlinear contribution. This has been emphasized as key feature of the SIREN architecture. However, the initialization choice relied on approximate computations, did not provide constraints on gradients, and it has been observed that estimation quality decreases in the lerge-depth limit under such initialization (Cai al. 2024). To address this, we propose the refined initialization:
u
Wn
1
Wp
b ~2 of), LE (I....,
u
Cu
Cun
LE (2...., L).
VN
4

---

Published as a conference paper at ICLR 2026
with (\ c)) the normal distribution of zero mean and variance cg. This initialization introduces two parameters, Cu and cb, which we set by enforcing constraints on the variance of pre-activations and the rescaled layer-to-lyee Jacobian:
ahe+1 Th.
Oa = lim v Var[ze)
and
dg =
lim N.N
NVar
Using explicit computations to guarantee a normalized gradient flow across the network in the mean- field limit, namely dg 1, we will demonstrate in next sections that must lie on curve parame- terized by Cw:
log -1)
(
3
We now derive two particular initialization choices along this curve. The first is the Sitgmann- inspired choice, obtained by enforcing o. 1, which was only approximately realized in Sitzmann eet al. (2020) and which we will later show does not produce the desired spectral behaviour. The second, which we adopt our proposed initialization, much better spectral control (see Section B.3). The corresponding parameter pairs are
d, 0 and will be shown provide
Ja 1: (Cw, cb)
6 L+e
aa =0 0 (Proposed) (Cwo cb) (v3, 0), (9)
2
v3
We illustrate the effect of these two initialization schemes and on several additional reconstruction tasks (see Appendix [B). Across all depths L, the proposed initialization with σ, = 0 consistently yields more stable networks than the standard SIREN (Sitz- mann) architecture initialized with equation aadd other state-of-the-art approaches. In particular, as depth increases, most competing methods exhibit gradient explosion, which manifests spurious, noisy high-frequency artifacts in the reconstructed high-resolution images. We also find that the on = 1 initialization produces slightly noisier outputs for deep networks than the oa 0 scheme, behaviour explained in Section 3.3 and motivating our preference for the proposed initialization.
image fitting problem in Fig.
3.1 PRE-ACTIVATION DISTRIBUTION
In the following, we derive the exact form of the pre-activation distribution in the limit of infinitely wide and deep neural networks, explicitly accounting for the influence of the bias term, which turns out to be crucial. More precisely, we show that, for any initialization in the parameter space (Cw. Cb. ) the pre-activation distribution converges to a fixed point. The proof is provided in Appendix AA.I
Theorem 3.1 (Pre-activation distribution of SIREN). Considering SIREN network described in equation 5 Wweee,, for some Cwr Co E Rt, and for every layer i E ( L}, the weight matrix W, is initialized as random matrix sampled from u(-cw/ /.N//VVNN cw//VN,, W1 is sampled from ((-wo//no wo! no)n the bias by initialized as a random vector sampled from NO,, cg) Let (ze))e(1....)) the pre-activation sequence defined in equation Fand relying input E 1 Then, in the limits N. L + 00, the pre-activation sequence (Zt)een converges in distribution to NO0 of) with
u(-
=
c + Wo 6
(
-
20
(10)
2
where Wo is the principal real branch of the Lambert function. The sequence associated to the fixed point o., which is exponentially attractive for all values of cu + V3. For Cu = V3 the convergence will be of rate OO)).
variance of the pre-activation (Var(z:)) EEN converges
Remark 3.1. While the bias distribution is different in our initialization and in the original SIREN scheme, the choice Cas = /6 for the weight initialization can be recovered as a special case of equation TObb imposing a. assuming 4 O, and by neglecting the correction term introduced by the Lambert function. Using the expansion Wo (x) = x + + (x²), this correction term can be estimated as ~ e² which is small but not negligible Accounting for this correction term enables more precise control over the pre-activation variance da.
A precise estimate of this correction can he obtained using equation
29 to be derived later.
5

---

Published as a conference paper at ICLR 2026
Ground truth
STREN
SIREN
SIREN (Sitemana))
Wire
Pincr
Tunk (PA)
ReLA (PS)
Training
INT SEAD
ASTNSADEE
FOR MULAE
SPIR: SAJT
SNR: 12A1
SNIR: 4:7
SINRL -o0 oil
Evalcation
Gradient
image-fitting problem using an L = 10 hidden-layer neural network width N = 256. We train the model on a set (æ:; V/ )ee where æ; is location taken [I] 128 x 128 uniformly spaced grid on Ω [-1, 1]² and ys is the associated image value at this location. The top row shows the fitted 128 x 128 image. The middle shows the estimation
Figure 2: Comparison of several INR architectures and initializations
augmented resolution (512 512) assess the model's generalization and the last row provides zoom on part of the image. In all case, we use ADAM optimizer with learning rate 10 for 10000 epochs. The statc-of-the-es -hrttrrttt architecture considered in this experiment are: S SIREN (see (Sitzmann et al. (0200), FINER (see (Liu et al. (2024), WIRE (see (Saragadam et al. )022)). Tanh (FX) with Fourier features and Xavier initialization (see (Tancik et al. (0200), and the traditional ReLU with Positional Encoding (see (Nair & Hinton 2010)). We used for SIREN based architectures the previously discussed schemes. We observe that the proposed strategies (SIREN (o.. 0 and о, 1) lead to significant improvement in the model estimation with respect to other methods. For instance, it preserves sharp features compared to other SOTA method such as Wire, Finer, that yields extremely poor results for deep neural networks.
Remark 3.2. As stated in Theorem 3.1 the pre-activation variance converges exponentially fast to Te as the depth increases whenever Cw * V3. In that case, even relatively shallow networks already have pre-activations that effectively Gaussian with variance very close the fixed point When Cu v3, this convergence becomes much slower. For proposed choice oa 0, this means that the pre-activation variance decays toward zero only gradually with depth.
Deriving the fixed points of the pre-activation distribution is necessary first step toward character- izing the layer-wise gradient distribution and for establishing the optimal initialization value for ca and Cb. which we discuss in the next subsection.
3.2 GRADIENT DISTRIBUTION AND STABILITY
The distribution of Jacobian entries is another important property of neural networks that must be carefully controlled during initialization avoid gradient vanishing (He et al.) 2015 Yang & Schoenholz 2017). In this work, we show that a tractable derivation is possible for the sine activa- tion function. This result is described in Theorem 3.2 Combined with Theorem 3.1 it will enable us to propose a principled initialization strategy provided in Proposition 3.I
Theorem 3.2 (Jacobian distribution of SIREN). Let Jr Th. /0hh//h111 denote the Jacobian of the l-th layer. Considering SIREN network described in equation S we have
J = =diag (cos (zc)) W..
Under the same assumptions Theorem 3.1 and maintaining the limit of large N, each entry of J has zero mean and a variance oz, such that the sequence (No?)ter converges to
2
lim (No?) = = 6 (22)-
(11)
For a given network, with input x and output p (z), Theorem can be used to analyze the scaling behavior of gradients with respect to both the network parameters 0 and the input coordinates We

---

Published as a conference paper at ICLR 2026
denote by 0o, V the gradient of the network output with respect the parameters Oe of layer e, and by Oz v/ the gradient with respect to the input æ. By applying the chain rule, we have:
@V((a) 002
vv.
Tbe+r (a)(
@Va( @V)( Dh (a) dæ
(12)
0hg-1 The 002
Dhg-1 Th
dx
These relations can be used to obtained scaling of the gradients variances with the network depth and width (see Appendix |A.4 for a derivation):
Var(de, (o(x)) x N (02) (0?)--(- and Var(O=Wo(a))) x wā (o)) 1-2
(13)
From equation 13 we see that gradients in parameter space vanish or explode exponentially with network depth L. unless the scaling factor of is close 1. To conclude the analysis of the statistical
properties of SIREN networks and derive the initialization schemes provided in equations 7 7 we identify the values of Cur and ea allowing to control the scaling of gradients i.e. Bg
Proposition 3.1. Under the same assumptions in Theorem 3.1 setting Tg weight-biax variance curve Cs (g(w)) defined in equation Furihermore, choosing 0g = 0 (our
leads to the
proposed initialization) or oa determines a specific pair (Cw. (6) given in equation 9
Figure 3: Experimental standard deviation of the pre-activation distribution (left) and of the layer-wise Jacobian entries distribu- tion (right). as a function of the parame- ters (Cur, cb). The plain and dashed black lines indicate the theoretical predictions for oo and ag orems 3.I and 3.2. respectively. The black and red dots indicates the initialization provided in Proposition [3.]] the Pytorch dots corresponds to the default weight and bias initialization, and the green dots to the Sitzmann initialization.
pre activation
gradient
1.00
following The-
0.30
Pytoechs
Sitenans
Pstorch
Sizmann
V8
v$
Weight cocfficient
The proof is given in Appendix A.3 We verified the validity of this theoretical analysis, involving careful calculations of the Jacobian and pre-activation distributions, through numerical experiments displayed in 3 3 These experiments were done 20 times using SIREN neural network of width N = 256 of depth = 10, with input dimension 7g = 1, and output dimension n. = 1, wo = 1, and following the initialization scheme in equations 7
The neural network is then evaluated using
[T| = 500 input points æ; uniformly spaced between
1] to obtain the studied distributions.
In the next section, explain why choosing 0. 0 rather than da provides better control over the network's 's frequency spectrum.
3.3 FOURIER SPECTRUM AND ALIASING
The need constrain the Fourier spectrum of sinusoidal neural networks prevent ligh-frequency aliasing was noted in (Yüce et al. 2022), and closed-form expression for the spectrum of sine- based networks was later derived in Novello et al. 2025 Thm. 3), showing that each additional layer redistributes energy across Fourier modes. Since composing sine activations inherently broadens the spectrum with depth, controlling this growth requires either limiting the depth or enforcing d. = 0. In the latter case, deep layers are almost linear, because for 2, ~ (O, og) we have sin(2;) R 2, as oo - O. Empirically, our initialization with aa = 0 indeed suppresses the emergence of higher frequencies: as shown in Fig. 4 spectral broadening with depth is strongly reduced, and most of the energy remains confined below Wo. yielding a meaningful, appears compensate the nonlinearities
heptt-indeepedeent cutoff around wn-
The slow decay of os toward zero described in Theorem aapaperr just enough to avoid both explosion and collapse of the spectrum, even in very deep networks, a behaviour that remains unexplained and calls for further investigation.
In contrast, for o, = 1, and even more so under the Sitzmann initialization, the spectrum clearly broadens with depth, and substantial energy appears beyond wi. This excess energy is exactly what
7

---

Published as a conference paper at ICLR 2026
causes aliasing when the network input is discretized. For the PyTorch initialization, the opposite behavior occurs: the spectrum collapses rapidly with depth, reflecting a manishing-signa regime caused by unnormalized gradients. Overall, this analysis supports proposed initialization, which constrains aa = 0 and motivates choosing wi as the Nyquist frequency for sampled inputs. This ensures that the network can represent all frequencies present in the data while avoiding aliasing in the early stages of training.
E0m1
L-16
Ampituda
L-8
L-32
20
syen truch boodeenn
80m-
00
35
Leb
L-32
60m4
syen boodkerng
10n1
550
500
730
1000
1250
250
540
700
1800
1250
200
500
750
1903
1550
200
500
753
1000
1250
Brequency
Prequexcy
Prequeocy
Prequazay
Figure 4: One-dimensional Fourier spectra of Vo for multiple depths L E {4,8, 16, 32), driving frequencies wh E {100, 1000} (rows), and initialization schemes (columns). Each curve shows the magnitude of the discrete Fourier transform of V evaluated on equispaced grid; colors encode the depth L. The red vertical line marks 2 which corresponds to the input frequency encoded by the first layers and the black vertical line marks wo. The colored backgrounds group the different initializations (from left to right: proposed SIREN with a, = O. SIREN with 0, = 1, the initializa- tion of (Sitzmann et al. 2020), and the default PyTorch initialization).
SCALING OF THE NEURAL TANGENT KERNEL WITH DEPTH AND SIMPLIFIED LEARNING DYNAMICS
The Neural Tangent Kernel (NTK) framework a linearized description the training dynamics around initialization, allowing
study how the network evolves in the early phase of training Uacoo et al. 2018). When training neural networks, we typically use gradient descent to minimize the loss function, with updates 0.+1 O, do.,,,))) where de is the learning rate and 0, the parameter vector at iteration L.
To simplify restrict ourselves to scalar output neural network (i.e., 1). Then, have for the mean-squared error loss [()) = EiEt || ( - //TT3/[TI, and in the continuous-time limit dt + 0, the residuals u(x;, t) = Vo, (x:) Y/ satisfy
du(()) Kg ut), Ko.ss = VoMo, V Mo, (x,),
(14)
dt
where u(t) = (u(#), .)..., u( f)) and Kg, is the NTK matrix. Assuming the NTK remains constant during training (Ko, Ko,), the residuals evolve
(t)) = exp(-ttks )u(0)
(u(0), U.)u,,
(15)
where (Ag, v.) are the eigenpairs of the initialized NTK Ko,, ordered so that A ... Apj 0, and (..-) the Euclidean scalar product. Thus, the early training dynamics is fully determined by the spectral properties of the NTK at initialization.
Frequency bias in the NTK framework. Equation 15 shows that modes associated with large eigenvalues decay quickly, while those with small eigenvalues decay slowly, with characteristic timescale 1/2g- As illustrated in Fig. for the ID case, and observed in related settings (see e.g. (Wang et al. (20T)), the leading eigenmodes (small i) of the NTK can be identified with low- frequency Fourier modes, whereas higger-fr components (large i) correspond smaller
8

---

Published as a conference paper at ICLR 2026
eigenvalues A, Figure 5 provides an overview of this behavior. This illustrates the spectral bias of neural networks in the lazy training regime (i.e., nearly constant NTK) and emphasizes the im- portance of controlling the spectrum {Ar) to accurately capture all relevant target frequencies. more detailed study of the overlap between NTK and Fourier modes, for different initialisation schemes, is presented in Appendix B.2.2
First 6 cigenvectors of the NTK of a SIREN
1b
U6
37
Figure The first six eigenvectors Vo..... .... U6 of the NTK matrix Ko. ordered by decreasing eigenvalue A > X1 > ... > Às The NTK matrix was computed numerically on a uniform grid of [I] 500 points over the interval ? [-1, 1] using SIREN network of width N 512 and of depth L = 8 and using wi = 1. The eigenvectors exhibit increasingly oscillatory behavior as mode index grows, consistent with their interpretation as Fourier-like modes. This observation confirms the spectral structure predicted by our analysis and highlights the tendency of the NTK to prioritize low-frequency components associated with larger cigenvalues.
Empirical scaling of NTK eigenvalues and network gradients. To highlight the importance of initialization in the large depth limit, conducted an experiment comparing the original SIREN initialization (cf. equation 6). the new ones (cf. equations 7 1.. and the Pytorch one. We varied the depth L while fixing N 256, II| 200, and Wy 1. In figure 6 plot the normalized NTK trace (mean eigenvalue) expressed as Tr.,e,)/(ll () together with the gradient norm ||_ Vo, as functions of network depth. We the NTK trace computationally convenient proxy for the typical eigenvalue behavior as depth increases. With the original SIREN initialization, we observe exponential growth of both the NTK cigenvalues and the input gradients. this case, increasing depth accelerates training but also causes gradient explosion in imput space. This corresponds to spurious high-frequency components absent from the target signal, which degrade generalization, here understood as smooth interpolation between data points. With Py Torch initialization, the NTK eigenvalues decrease until reaching plateau, while the gradient in input coordinate space vanishes. By contrast, with our new initialisations, the NTK eigenvalues increases linearly with depth while the gradients remain constant. Consequently, the effective learning rate increases with depth L, while the input-space gradients stay normalized. These behaviors confirmed in practical set- tings, such as the mmage-fitting task shown figure 2 and in additional experiments presented in Appendix B
Interpretation of the scalings. The scaling of gradients with of is expected from section 3.2
with 0g v1.2 for SIREN, og for proposed initialization, and 0g v1/3 for PyTorch initialization. Similarly, is possible to explain the NTK eigenvalue scaling. We note first that diagonal element of the NTK matrix are Kg property of every gradient distribution, the variance of gradients in parameter space:
(2.)), From this and the zero mean relate the average eigenvalue of the NTK denoted Ā to
À + Tr(Ko,) = N2 Σ Var (w, Ma, (x:)] +NN Σ Var (V Ve, (x:)]
(16)
where We, by respectively weight and bias of the L-th layer. The sum involving weights parameters being dominant, we neglect the sum on bias terms in the following. When o² + 1, using equation I2 we obtain geometric sum, leading to
(02)
Tr Tr x
(17)
N
0²

---

Published as a conference paper at ICLR 2026
If og > (SIREN original), then À X o2 and the NTK explodes exponentially with depth L. This exponential scaling for the NTK eigenvalues without proper initialization was observed experimen- tally in (de Avila Belbute-Peres & Kolter 2023), yet without precise discussion on the causes and the effect of such behavior, since their focus was on the choice of wo rather than on weight and bias initialization.
If o, < depth limit, yielding slow convergence, together with vanishing gradients.
(SIREN PyTorch), NTK eigenvalues become independent from the depth L in the large
If og = (STREM o 0, 1). equation 17 does not apply. Each term of the sum on weight parame- ters in equation 16 gives the same contribution, leading to x I,, which is consistent with the results plotted figure for the Sa initialization, for a 0 it seems that the NTK eigenvalues verging to a fix distribution, and we attribute that to finite size effect of our initialization, indeed the convergence really slow towards da 0, which seems compensate the NTK eigenvalues growth with depth, for finite depth networks.
NTK trace vs depth
Gradient std vs depth
10"
10
10
10a
D
10
O(v1.24
1/ N)
10P
10
O
10P
10--2
SIREN (o. 0)
10-1
SIREN (r. 1)
O / (1/3)C
(Sitzmann)
10
10-1
SIREN (Pytorch)
10-
10"2
10
20
30
40
50
60
220
30
40
50
60
Number of hidden layers
Figure 6: The left plot stands for the scaling of the mean eigenvalue of the NTK matrix over the number of layer. The right plot stands for the scaling of the gradient of the network (in input coordinate space) with the number of layers. The experimental setup and hyper-parameters are the same as in figure except for the network depth which varies here.
DISCUSSION, CONCLUSION PERSPECTIVES
We proposed initialization scheme for sinusoidal neural networks that prevents gradient plosion and vanishing, and presented various applications, from noisy image fitting, video, and audio reconstruction (Appendix [B. The parametrization derived analytically by examining the variances of pre-activations and layer-to-layer Jacobians in the limit of infinitely wide and deep net- works. This approach removes the need for architectural tricks such as skip connections or empirical hyperparameter tuning to stabilize deep models. By analyzing both the neural tangent kernel and input-space gradients, we showed that this initialization enables deep networks to train with learning rates that scale linearly with depth, while suppressing spurious noise above the Nyquist frequency implicit neural representations. Whereas prior work motivated the use of sine activations by not- ing that derivatives of SIRENs remain well- well-behaved, theoretical analysis. We demonstrate that sinusoidal architectures not only preserve these desirable properties but also admit stronger theoretical justification. A key take-away is that fixing the Ja- cobian variance (σg = 1) is essential to control gradients, whereas setting the targeted fixed point pre-activation variance (a,, 0) gives direct control over the network spectrum initialization.
study goes further by providing deeper
Future work could extend the approach to more complex losses, including physics-informed set- tings, and potential applications atmospheric and oceanic field reconstruction based on INRs (Johnson et al. [2022] Mostajeran et al., 2022).. Furthermore, our study focuses solely on controlling the variance of the weights initialization. One could broaden this perspective by considering ad- ditional structural properties of the network such as the distribution of singular values of the layer Jacobians (presented in Appendix B.I
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
Code Implementation. All source code used in our experiments is provided in the supplementary material, including implementations of the architectures used for comparison.
Models and Architectures. Details on the choice of activation functions are given in the main text. Initialization methods and architectural specifications for each model are described within the corresponding experimental sections.
Experiments. Each experiment reported with its hyperparameters (e.g., learning rate, optimizer, number of epochs) in the relevant sections or figures. All experiments were run with fixed random seeds ensure exact reproducibility of the reported results.

---

Published as a conference paper at ICLR 2026
REFERENCES
Zhicheng Cai, Hao Zhu, Qiu Shen, Xinran Wang, and Xun Cao. Batch normalization alleviates the spectral bias in coordinate networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 25160-25171, June 2024.
Bartomeu Coll, Armengol Gasull, and Rafel Prohens. Asymptotic dynamics of a difference equa- tion with parabolic equilibrium. Qualitative Theory of Dynamical Systems, 19(2), 2020. ISSN 1575-5460. doi: 10. 10110077612346--00-000006666 URL https:/ //te:://doiio http://tt..oooo..///// 1007/
612346- 00406-0
Filipe de Avila Belbute-Pere Pertee and J Zico Kolter. soidal networks via their kernel bandwidth. In The Eleventh International Conference on Learning Representations, 2023. URL httpt:: /ttt:p::///peereiiiwwe wwwetteeeereeeeeee forum V1G=VVVC6gCC 4d
Simple initialization and parametrization of sinu-
Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Com- pression with implicit neural representations. In International Conference on Learning Represen- lations, 2021.
031
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth Interna- tional Conference Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https: http:///pocceeeinn...o mlr www.:/ssww.p.rrrss.emmmmeeeeeeee v9 wttpgooooorrooooooo html
Zhongkai Hao, Jiachen Yao, Chang Su, Hang Su, Ziao Wang, Fanzhi Lu, Zeyu Xia, Yichi Zhang, Songming Liu, Lu Lu, and Jun Zhu. Pinnacle: A comprehensive benchmark of physics-informed neural networks for solving pdes. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Pa- quet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37. pp. 76721-76774. Curran Associates, Inc., 2024. doi: 10.52200/079011-22422 files/ /ttpa//e/e.aaee/pp///////////11 Bn___accceeeeeeeereeeeeee__
URL httpss//prroeeddnggseuurrroopppp
Track frff
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International conference on machine learning, pp. 2672-2680. PMLR, 2019.
Soufiane Hayou, Amaud Doucet, and Judith Rousseau. The curse of depth in kernel regime, 13 Dec 2022. URL httptttt//ssocceeeeeee
mll. pttpp//ell////ayue/uuuu
html
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015. doi: 10. 109/ICCV.0015.233.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
J Emmanuel Johnson, Redouane Lguensat, Ronan Fablet, Emmanuel Cosme, and Julien Le Sommer. Neural fields for fast and scalable interpolation of geophysical ocean variables. arXiv preprint OrXiv:2211.104444 2022.
J JAS Kelso, A Mandell, F Shlesinger, and H Packard. Dynamic Patterns in Complex Systems, chapter 3. Addison-
Addison-Wesley, Wesley, 1986.
Christopher G Langton. Studying artificial life with cellular automata. Physica D: Non- linear Phenomena, 22(1):120-144 149, 1986. ISSN 0167-2789. doi: https///oi.orr/100 10. 101660166-278898699003337 URL https://wwwt sci peienneedirect. wwweect..oosccienneeeee atticile//1//0167778889999 ference.
Proceedings of the Fifth Annual International Con-
Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. On the eigenvalue decay rates of class of neural-netwood related kernel functions defined on general domains, 2024. URL https:
7/arxiv. htars//..orr//b///23555.. 02657
12

---

Published as a conference paper at ICLR 2026
Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng. Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodie activa- tion functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Mingze Ma,
Qingtian Zhu, Yifan Zhan, Zhengwei Yin, Hongjun Wang, and Yingiang Zheng. Ro- bustifying fourier features embeddings for implicit neural representations, 2025. URL https:
//arxiv.orq wwwxii..org/bb//2502222 25420 05482
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Farinaz Mostajeran, Nikhil M Pawar, Jonathan M Villarreal, and Salah A Faroughi. Context-aware implicit neural representations compress earth systems model data. Scientific Reports, 15(1): 25932, 2025.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learming. ICML'10, 10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.
Tiago Novello, Diana Aldana, Andre Araujo, and Luiz Velho. Tuning the frequencies: Robust train- ing for sinusoidal neural networks. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference (CVPR), pp. 3071-3080, June 2025.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In I. Guyon, U. Von Luxburg, S. Bengio, Н. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems. volume 30. Curran Associates, Inc., 2017. URL https: httptp:://ppooceeddsss...ss www.ips. htcpppps.ccppppae cc/paper files/paper/2017/ File/ (0F.0/CCcc.330000014444000000000
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 09-15 Jun 2019. URL
9a httl
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 000866777,2099 2019.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory. Cam- bridge University Press, 2022. https::/ rttpss//ddeeleaamiiagteoor com
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In arXiv preprint arXiv:2301.05187, 2022.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference Learning Representations, 2017.
13

---

Published as a conference paper at ICLR 2026
Mariia Seleznova and Gitta Kutyniok. Neural tangent kernel beyond the mnfit limit: Effects of depth and initialization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 Proceedings of Machine Learning Research, pp. 19522-19560. PMLR, 17-23 Jul 2022. URL htttps://procee
ngs. ml www......s.............. / v1 62/seleznova22..
html
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wet- 7stein. Implicit neural representations with periodic activation functions. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 7462-7473. Curran Associates, Inc., www.wne.uricc.o cctpssscccpppppppeepp filesspapee//220/
2020. URL
File/ TLle/53c0411ddEIl
Yannick Strümpler, Janis Postels, Ren Yang. Luc Van Gool, and Federico Tombari. Implicit neural representations for image compression. Shai Avidan, Gabriel Brostow, Moustapha Cissé, Gio- vanni Maria Farinella, and Tal Hassner (eds.), Computer Vision - ECCV 2022, pp. 74-91, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-198094
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kei,, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. NeurIPS, 2020.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature net- works: From regression to solving multi-scale pdes with physics-informed neural networks. Com- puter Methods in Applied Mechanics and Engineering, 384:113933 2021. ISSN 0045-7825. doi: 101 occiinec/Arrttaeee//////55533311
URL https: /tt////wwww/// sciencedirecttt. com/
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30, 2017.
Gizem Yůce, Guillermo JimineJimenez, Beril Besbinar, and Pascal Frossard. A structured dictionary perspective on implicit neural representations. In Proc. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR 2022). June 2022.
14

---

Published as a conference paper at ICLR 2026
A MATHEMATICAL APPENDIX
A.1 INPUT DISTRIBUTION
Theorem (Restatement of Theorem 3.1]. Considering SIREN network described equation where, for some Cwn Cb E RR, and for every layer & 12,...,LI the weight matrix We is ini- tialized a random matrix sampled from U( u(-Cw/ NN,//VNN, and the bias by is initialized as a random vector sampled from (N0, <t. Let (Ze)sE(1.....) the preactivation sequence defined in equation 3 and relying an input E TR Then, in the limit of large N, the preactivation sequence (EN converges in distribution to NO0 o.) where
1 Wo
=
20g
(18)
with Wo is the principal real branch of the Lambert function. functinn, Additionally, the sequence associated to the variance of the preactivation (Var(z))) EEN converges to a fixed point da. which is exponen- tially attractive for all values of Cw # v3.
Proof. The proof can be split in three steps: (i) prove that the sequence preactivations follows Gaussian distribution (cf. Lemma Le.maa..T). (ii) give expression of the variance of the output of a sin activation when the input follows zero-mean Gaussian distribution of s.tdd aa (cf. Lemma (..)....2). (ii) provides the expression of the variance of each element of the preactivation sequence using the result (ii) and proves its convergence to a fixed point da (cf. Lemma A.3
Lemma A.1. Considering SIREN network described in equation 5 where, for some Cws Ch E R.+ for every layer E (2....; ., L.), the weight matrix W is initialized as a random matrix sampled from U( (=c////VV /ooo//N,c///VV N),, W, is sampled from (odowo//no no, wo/ (o), and the bias by is initialized as random vector sampled from NO, cc)) Let (Ze)rE(1.....) the preactivation sequence defined in equation 3 and relying on an input x E Re Then, in the limil of large N, each element of the preactivation sequence (Ze yeer is distributed according a zero-mean Gaussian distribution.
Proof. We recall that for the first layer, hg and, for every l E LI....,LI.
h sin (Whhe-1 + by).
Since the sine activation an odd function, it preserves the zero-mean property of any distribution: if Wche-1 by has mean, then h, will also have zero mean. This property propagates layer by layer.
As W1 and bj are assumed to have zero mean (by definition, cf. equation 7 and is deterministic vector, it ensures that the first-layer pre-activation has zero-mean. Moreover, as W and b, are assumed to have mean the zero-mean property holds for all subsequent pre-activations z, and h
Second, prove that the preactivation sequence is distributed according to Gaussian. We first rewrite each element of the preactivation sequence as
W Weighe-1d b
(19)
As sum of two Gaussian stays Gaussian and because by assumed be Gaussian with a standard deviation σbe the main purpose here is then to prove that E/ We..hhe-.. follow a Gaussian distribution.
Thanks to the Central Limit Theorem, whatever is the distribution of he the term E We,,yhh--,, converges distribution a Gaussian distribution the limit of large Since the bias also normally sampled, each component z,, follows a gaussian distribution in the same large N limit, with zero and variance denoted oz
15

---

Published as a conference paper at ICLR 2026
To compute this variance, let first compute the variance of each summand denoted of given by the product of two independent random variables with zero mean, namely We, and by- 1.-1.j.
oe., Var [We, Var [he-15l..
(20)
Since W uniformly distributed [- / /N, Cw/ VNJ. have:
C
Var [We,a5]
(21)
3N
While the variance of b is still unknown, we can express it from the knowledge of z leading
c Var (sin (z+-1,))
(22)
3N
whose expression of Var (Zi
1, 1.)] will be provided later.
As the bias variance follows a Gaussian distribution as described in equation 7 the variance of all the elements of the preactivation 24 is
of T Var [sin(ze-1)] cz.
(23)
Lemma A.2. Let be normally distributed random variable and mean N(00 Then we have
Var (sin (z)] (1-e20²
1 e20 )
(24)
The proof combined the properties of the Gaussian distribution with the fact that the sine function is an odd function. We have:
Proof of Lemma A.2
Var [sin (2) = E [sin (x)) - E [=]]
Since sin is odd and since the expectation of is zero, we have E [sin(2)] = 0. In addition, using
sin' 2 (z) = (1 - cos(22))/2 we obtain
sin [0)] 2 2 [cos(2:)]
The characteristic function of the Gaussian distribution with zero mean and variance da is given by:
g=(t) (t) E(e = -
Now we notice that
(2=)] = E R [2 R [g, (2)] e 20 The first equality uses the linearity of the mean. This leads to the final result:
1 (1 (1- - 2n
Var [sin (z)] 2
Lemma A.3. Considering SIREN network described in equation 5 where, for some Cw Cb E R+ and for every layer f {1....,L)) L), the weight matrix We is initialized as a random matrix sam- pled from U(-cw/ /N /,,co/ VNN,VN, and the bias be initialized a random vector sampled from (00, ) Let x E Rud Then, in the limit of large N, the preactivation sequence (Zt)ee(1.....)) defined in equation
distributed according a Gaussian distribution with zero-mean and. for
every E, a variance
c 2-2--0 +
6
Moreover, the sequence (o?) converges to
- 6
Wo.-1
cw - 20
with Wo,-1 the two real branches of the Lambert W function. And for cno # 33, this convergence is exponentially fast.
16

---

Published as a conference paper at ICLR 2026
Proof of Lemma A.3
Fixed Point Value: Combining equation 23 and equation A.3 layer is
the variance of the pre-activation
of
we 6
[ - 20021) t
To characterize the fixed point of the sequence ) we define a function f as
f(x)
Ca
2x 6 (1-e 21- + cG.
(25)
The fixed point of this function is given by the solution of the equation f(x) 2. Rearranging the different term gives:
2
2 x
+ 6
2x
(26)
6
6
22 Using y + cg yields
2y
C 2
+
ye
Then, using the definition of the real valued Lambert W function, we get
1 Wi y 2
(-
c (-rmee (- -
where k 6
The Wo branch is called the principal branch and is defined on (-c-1, +oo). The W-1 branch is defined for (-e-1,0) ,0). To obtain positive variance, the branch to consider is Wo. as illustrated
numerically in figure 7
Branch Wo
Branch W-1
3
2
3
2
Cw
CN
10
-2
o²
o²
Figure 7: The solution emerging from the Wo branch on the left and W_, branch the right
Convergence Speed To quantify the convergence towards the fixed point o3 consider the deriva- tive of at the fixed point:
20 ((
The fixed point is exponentially attractive whenever (' < 1, which is immediately satisfied for Cur < V3. For Cw. > V3, Lemma A.3 gives
((0 = 2(-5(2())))) +
+ + -Wo
2 --
6
Since
2 2 /3-20 <0,
17

---

Published as a conference paper at ICLR 2026
the properties of the principal branch Wo imply ( < 1. Hence, the fixed point is exponentially attractive for all values of Cw * v3, and convergence occurs rapidly. For cw v3, the map f can be written
f(x) - -
(1-
x z20.
Taylor expansion at yields
2 3 xx² + O(x²),
[(x) =
so that f is tangent to the identity at the origin, i.c. f(0) 0 and (0) - 1. Moreover, since f(x) < x for all æ 0, the map f admits as its unique fixed point on [0, oo), and any sequence (ge)e20 defined by ct+1 f(ae) with do > 0 strictly decreasing and converges to O. Furthermore thanks to the previous extension fits into the general class of one-dimensional parabolic maps studied in (Coll et al. 2020) Theorem 1). That theorem provides a complete asymptotic expansion of the orbit (az); in particular,
1 e
as l + co.
This concludes the proof of the Lemma A.3 and of the Theorem 3.1
A.2 GRADIENT DISTRIBUTION
Theorem (Restatement of Theorem 3.2 Let J = 0hz/0be-1 denote the Jacobian of the f-th
layer. Under the same assumptions a Theorem 3.1 we have
J. diag(cos (z+)) We
In the limit of large N, each entry of J has zero mean and a sequence of variance o7 such that the sequence N(o2)eeN converges
22 og 6 (1 2033)
Proof. An element of the Jacobian the f-th layer are writen as:
Obg W & cos Th.- 1.k
W Weighe-is + be,i We,i, cos (ze,,)
with ze, the ith component of pre-activation vector defined in equation In the limit of large width Noo We and b5 are independent (leave-onc-out justification), resulung in the independence of variable We..k and cos (zz,s). The variance of their product denoted o? can then be expressed as the product of their variance:
07 Var [We, Var [cos (ze,a)]-
Considering the same arguments for Theorem 3.1 and replacing sin by cos, the sequence (Noz) converges to
0f 6
(1 20
with o2 the limit variance of the pre-activation, given by Theorem 3.1
18

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF EQUATION 8 AND INITIALIZATION 9
We propose to initialize the weights and biases of SIREN networks as follows:
u
w0 no
1=
no
We ~
u
E (2,...,L) L}:
VN
vN
and
by NO, k)), 8 (1......1) L.). To control the distribution scaling of gradients, following equation
we impose og = i..
1 = =1. 1.
(27)
Let's recall that the fix point da verifies
2 (1 1 -
20%
+ t0
6
From equation 27 and, we easily get
6 -1 Ca
Cb
log
(28)
Combining this result with equation 27 leads to an implicit equation for cg.
We discuss in the text two particular points, corresponding to o. and σ₁ 1, respectively:
The case aa (proposed initialization) leads to (Cwr ()) (v3,0) 0).
The case 0a = 1 leads to a 6/(1 + - To obtain an explicit expression for Cb, it is convenient to use the fixed-point equation 26 with r = 1, leading to:
(1-c + 1,
(29)
which, using equation equation
27 simplifies to
2 2-
c
(30)
A.4 DERIVATION OF THE PROPOSED SCALING
Let (o) defined in equation 5 scalar output function, initialized in the previous theorems, and considering given value of og resulting from the initialization.
Derivation of the parameter-wise Gradient scaling: Considering a
@v) which can be rewritten as We, i.j
with é of the ε-th layer, we study the scalar 8W,,.5"
( 2v Th,-1 Oh (h)(x)
OW,i Oh Shr.-2 Th, aWe.j
Then from theorem under the choice of our initialization we know that the Jacobian matrices J, /h, have variance / in the limit of large and large N. Moreover, we have from
the definition of a the expression of the vector ah consider first the sensitivity vector ge:
aæ, W. L with Var(W,)) 1/N. Let us
N Ah
Oh
g: Oh Oh
(31)
Th.
19

---

Published as a conference paper at ICLR 2026
Owing to the impact of matrix multiplication every components, have Var(g:) (No3) Let us now consider now the term (h,(m) This is zero vector except for the i-th
aWrss"
( 1. component, verifying @h),( = h cos( We be + b,), with variance Var( oWe,s. 8Wais
Hence, the parameter-wise gradient can be rewritten as:
(V)())
OW f be cos( We- h b;).
Assuming independence between ge, and @V)(=) WWens we finally obtain the desired variance scaling,
namely Var (We.s
(W)(())
(0))2-111/ /N.
Derivation of the input-wise Gradient scaling: Following the same notations above, have:
@v)(2) 0Vv(x) 0hr-1 The (z)
Ōæ Ohi-1 ShL-2 Th Ōx
Recalling that g1, has variance Var(g1) (o (o In that case the 1/N factor will cancel out due to the term Sh. (a)
Indeed, we have:
dx
Th, (x) diag(cos (Wi + b)) W1,
dæ
which is non-trivial matrix of variance Var( (h) w for both the original and proposed SIREN initialization. Focusing on one input coordinate I,, we get:
o g1 diag(cos (Wix b)) W1.:1= ΣBI, Bi, (diag(cos (Wx + W1,.a Oz,
The variance of each term scales as (02) Supposing independence between each sum- mand leads to Var( (V)(()) ~ (02)
dx
B EXPERIMENTAL APPENDIX
B. END END JACOBIAN, SINGULAR VALUE SPECTRUM
As discussed in (Pennington et al. 2017) an important notion of stability in neural networks is captured by the singular value distribution of the end-to-end Jacobian: when these singular values concentrate around 1, the network preserves the norm of signals during backpropagation. This property, known as dynamical isometry, is closely linked to stable and efficient training and will be the subject of further investigation for SIREN architectures in future work.
As a preliminary step toward this analysis, we plot figure g the full singular value distribution of the end-to-end Jacobian obtained with our proposed initialization. Since focus INR settings, we define the end-to-end Jacobian as the matrix of size x N, where N denotes the width of the network:
J Oh.-1
Oh
Once again, our initialization with a. = 0 exhibits a stable and nearly unitary normalized maxi- mum singular value, independently of network depth. This behaviour is not observed for the other initialization schemes, where the largest singular value either grows steadily with depth collapses rapidly, as the case of the PyTorch initialization. However, our initialization does achieve full dynamical isometry, indicating that there remains room for improvement while still satisfying the key constraints established earlier. Exploring additional constraints on the weight distribution may therefore lead to enhanced stability with respect to dynamical isometry.
B.2
NTK SPECTRUM AND FOURIER OVERLAP
B.2.1 NTK SPECTRUM
In the main text, we restricted our analysis of the Neural Tangent Kernel (NTK) speetrum to its trace, which captures only its mean behaviour. However, the trace alone does not reflect the full structure
20

---

Published as a conference paper at ICLR 2026
SIREN
STREN (7.
SIREN (Sitemann)
SIREN (Pytorch)
00m
20m
1.
16
107
102
107
Singular value Index
Figure 8: Full singular value spectrum evolution with depth for the proposed initializations oa
and a = 1. for the original Sitzmann initialization, and for the PyTorch default weight initialization. Each spectrum was averaged over five independently initialized networks. The Jacobian distribution was computed twice and averaged, using 10 sample points on the domain T).
of the spectrum. In this section, we therefore examine the complete NTK eigenvalue distribution in order to highlight its finer characteristics.
6rg
(Situmann)
(Pytorch)
L:4
L::16
10-4
10
80-18
10
Eigenvaluc index
Figure 9: Full NTK eigenspectrum evolution with depth for the proposed initializaaions oa = 0 and Oa 1, for the original Sitzmann initialization, and for the Py default weight initialization. Each spectrum was averaged over five independently initialized networks. The NTK was computed on the domain [- ,, x] using 256 sample points.
The full spectrum analysis shown figure 9 reinforces our previous observations based on the NTK trace, namely that the Sitzmann and Porch initializations become extremely ill-conditioned as depth increases.
contrast, the aa One can observe noticeable lifting of the cigenvalues lifting is much smaller and more uniform under the o, = 0 initialization. This behaviour could be directly related to aliasing phenomena in such networks, where high frequencies can be used earlier to fit a signal.
and оa = initializations remain comparatively stable.
high indices for σ, = 1, whereas this
This interpretation is further supported by the next analysis, where show that under ill- conditioned initializations the low-index NTK eigenvectors begin to encode increasingly high fre- quencies depth grows.
B.2.2 FOURIER OVERLAP
To support our NTK analysis and our explanation of spectral bias, we previously assumed (see F5gure form of alignment between the eigenvectors of the SIREN NTK and the Fourier modes. To verify this assumption for our different initialization schemes, we examined the power spectrum of the NTK eigenvectors, which corresponds to their overlap with the Fourier modes:
((vn, 22) 1/ vn(xx) -twe
1
kww dx
(32)
21

---

Published as a conference paper at ICLR 2026
SIREN (o.
SIREN (o.
SIREN (Sitzmann)
SIREN (Pytorch)
159
0.00
189
0.00
189
0.00
1.9
Depth
159
22.00
Eigenvector Index
Eigenvector Index
Elgenvector Index
Eigenrector index
Figure 10: Overlap evolution with depth of the NTK cigenbasis over the Fourier modes, for the proposed initializations O, and 0, 1, the original Sitzmann initialization and the initialization with Pytorch default initialization weight. The power spectrum has been calculated using over the interval [-64, 64] using 512 points. wy has been chosen be two times smaller than the Nyquist frequency of the input points for the sake of vizualization. The horizontal red dashed lines correspond to the frequencies +wo.
1,
The previous analysis reveals that the only initialization preserving the expected ordering, low fre- quencies corresponding low NTK eigemalues, our proposed initialization with aa 0. This observation is consistent with our Fourier-spectrum study (see Section B.3). Indeed, observe in Figure I0 an almost perfect alignment between the Fourier modes and the NTK eigenspectrum for frequencies below wo-
For the other initialization schemes, this alignment deteriorates substantially as depth increases, calling into question the relevance of NTK- explanations of spectral bias. Indeed, in the NTK regime, the first modes learned are no longer the www-freuy components; instead. higher- frequency modes increasingly dominate for aa and the Sitzmann initialization. For the PyTorch
22

---

Published as a conference paper at ICLR 2026
initialization, the situation is reversed: the entire spectrum collapses, preventing any meaningful frequency ordering.
B.3 AUDIO FITTING EXPERIMENTS
To investigate the effect of the proposed initialization on the network's ability fit figqufruyy signals, we consider a 7-second audio clip sampled at the standard rate of 44,200 Hz. To expose potential generalization effects, we subsample the signal by factor of three and set wo 7000, which is approximately the Nyquist frequency corresponding to this reduced sampling rate. The results are shown figure
SIREN
SIREN
SIREN (Sitamann)
Wire
Finer
Tanh (Fourier Rell (PE)
SNR ZUP
S00R 25
SNa: o04
SM0 24.2 05
SOUB: 0.0
SOOR:
SIVEL:
MSE: 1.006443
2.43--03
8.330-0
MSE 20
MEST: 3-3
1200
diae
Figure 11: Comparison of several state-of-the-art methods (described in Figure 2 with SIREN using our proposed initialization. All networks, with depth L = 15 and width N 256, were trained for 10,000 epochs using the ADAM optimizer with a learning rate of 3 x 10-5.
Both the SNR and MSE metrics show consistent improvement when using our proposed initial- ization on generalization tasks, while also providing strong training performance. The initialization with a, = also achieves competitive results, though its generalization accuracy remains notice- ably lower. For the other initialization schemes, even when training performance satisfactory, the generalization error remains far too large to reliably encode continuous signal.
B.4 VIDEO FITTING EXPERIMENTS
Video fitting on ERA-5 wind fields. To evaluate the impact of the initialization on complex video-fitting task, we consider the hourly ERA-5 atmospheric reanalysis on the spherical Earth, focusing on the 10 m meridional (South-North) wind component v(t, X, sp).
Where the data is defined on a regular longitude- latitude grid with
À [0, 360),
4 E [-90, 90),
discretized into
N₂ 1440 and spatial points, respectively. We restrict ourselves to the first Tmax = 30 hourly time steps. For training, we form set of onput- pairs
N₂p 720
a V1) iET'
where each index i corresponds to a triplet (1, A, sp) on this spatio-temporal grid. The target yi is obtained from vtt, A, sp) by standard affine normalization (subtracting a global mean and dividing by a global standard deviation computed over the first Tmax frames).
Each input vector is defined
x; = (T((),,, Ao, :Pi):
where the time coordinate rt)) is obtained via linear rescaling of the discrete time index such that the effective Nyquist frequency along the time axis matches that of the two spatial axes (longitude and latitude). This ensures a comparable frequency bandwidth in all three input directions and allows us pick wp 0.7 for every direction.
23

---

Published as a conference paper at ICLR 2026
For training, we randomly subsample fixed fraction of the full spatial gridded points {l..... N x {1....,N; N₂p} (10% of all points, justifying the choice of wy), while for evaluation we use the com- plete spatio-temporal grid.
Regarding the batching, to avoid I/O bottlenecks when accessing the dataset, we organize the data into time-slice batches. Concretely, consider spatio-temporal grid
E .O..... Tmax 1}, A (AI....., ANah, I (PI....,PN
and for each fixed time index t we form a batch containing many spatial points on the sphere. For a given time I, we define a (possibly subsampled) index set Ie C (l....,AA) Na} fI,..., Noj. and construct the corresponding mini-batch
B: { (1,k) E [z),
where each input is æt.j.k (T(1), Ajs pt) and the target yr.j.k is the normalized wind value at time t and location (x).
We benchmark previous state-of-the-art INR methods and our SIREN models with different ini- tialization schemes on this ERA-5 re-analysis to assess their ability fit and generalize complex
dynamics on the sphere.
Jusse sala
Book
Figure 12: Comparison over three different time frames of several sfatc-of-the-art methods on the ERA-5 reanalysis dataset (first 30 hours), using networks with width N 256 and depth 15. All models were trained for 6,000 epochs with the ADAM optimizer and a Reduce-on-Platen learning-rate scheduler, starting from an initial learning rate of 10-3, For batching, we used the time-slice structure described above with 5 gradient accumulation steps. To reduce computation time, we employed gradient scaling together with automatic mixed seeed-precision (AMP) training.
Once again, our initialization with = 0 yields better generalization performance, even on com- plex tasks and geometries such as video fitting on the sphere. contrast, the Sitzmann and a, initializations tend to produce noticeable noisy artifacts. Moreover, the F FINER and WIRE methods appear clearly unstable for high-depth networks. We also highlight the comparatively good perfor- mance of the positional encoding ReLU (PE) network in this setting.
B.5 DENOISING EXPERIMENTS
We consider grayscale image y" Ω C R² + [0 (the ast ronaut image), defindd on continuous domain Ω. For training, we sample a regular grid of locations
(ax)iet,
I [1,.... 128) (1,..., 128).
which we identify with points in [-1, 1j2. The clean training targets are
y, y (a;)) E [0,
i E T.
24

---

Published as a conference paper at ICLR 2026
To study denoising and the implicit spectral regularization of different initializations, corrupt only the training targets with synthetic high- figh-freueecc noise. Let N = 128 be the spatial resolution of the training grid and let
N
fNN
denote the associated Nyquist frequency (in cycles per unit length on [-1,1], We construct a high- frequency noise field as superposition of K random waves whose spatial frequencies lie strictly above fNyq:
K
n(x)
= Σ sin 2t (f(h) x1 + x2) + +0(1)).
where for each l we draw (8) We then normalize this field on the training grid have zero mean and unit variance.
1b ~ (2fNvgg 4fNya), @(h) (1, 2T), and a = x2)
(xx)) 1 Ejer n(a,))
T
i Eell
V ( - +/| Eeer n(x:)))
and scale it by prescribed noise level noise > 0. The noisy training targets are finally defined as y, y: noise 17/+
We train all INR models on the noisy dataset ( ÿs) het and evaluate on higher- resolution grid covering the full image domain, using the clean image y" reference. This setup isolates the ability of each initialization to act as an implicit frequency-space regularizer for denoising, independently of network depth.
Greedirch
Trstiee
Goofent
Figure 13: Results of the denoising experiments for the different state-of-the-art methods.using networks with width N 256 and depth = 10. All models are trained on the noisy dataset {(x,, 9:) hee described above using noise 0.05 and evaluated the original high-resolution image of size 512 x 512 to assess denoising performance. The networks were trained for 10 000 epochs using the ADAM optimizer with learning rate of 10-1
Figure 13 illustrates our claim that the proposed initialization acts regularizer on the frequency content that the network can represent. Indeed, observe higher SNR and lower MSE for our ini- tialization σ, = O, together with a significantly larger training loss. This indicates that the network does not fit all of the high- tigh-frequency background noise, but instead focuses reconstructing the underlying clean signal.
B.6 PHYSICS INFORMED EXPERIMENTS
Physics-Informee Neural Networks (PINNs) approximate the solution u of differential equation with v by embedding the underlying physical laws into the loss function. Given a PDE of the form
N [uu (xx)
E Ω
25

---

Published as a conference paper at ICLR 2026
with boundary/initial conditions B(u]] g(x)) on 02, the neural network Vy is trained by minimiz- ing the composite loss
( > Σ ByED,
(WVol(<a) f(z))) d Σ [ - g( mcED
where D, and Do denote collocation points in the domain and on the boundary. Automatic differ- entiation used compute p] allowing the network to satisfy the governing equations as part of the training process.
In order compare the several model at stake and the impact of the initialization, we used the PINNacle benchmark (Hao et al. (024), which allowed us to have a pre-builtin solver for each differential equation we studied.
B.6.1
BURGER 1D
We consider the mneedimensional viscous Burgers equation, written in the generic PDE form
0.01
Nu1 (ax t) = + u Uz Viltx O,
(2, t) E
The apatio-tempord domain is defined as Ω = [-1,]] 00, 1].. The initial and boundary conditions are given (z 0) (TT) ((-1,t)=u(1/)) =0
SEREN
Ther
Tonbs Frweree
3:4
co
Figure 14: Results of the Burgers ID solutions for the different state of the art methods, using network with width N 256 and 15. The networks were trained for 10 000 epochs using the ADAM optimizer with a learning rate of 10-4, For the SIREN based architectures, we chose wo = 2.
We observe figure 14 that the different initialization schemes yield very similar results, with the ception of the FINER and ReLU networks. Interestingly, for this specific task, the original Sitzmann initialization appears provide the most favorable performance. We conjecture that this behavior is related to the nature of the Burgers equation, whose sharp propagating front can be effectively represented even under a highly ill-conditioned gradient distribution.
B.6.2 STATIONARY NAVIER-STOKES
STOKES 2D
We consider the stationary incompressible 2D Navier-Stokes equations Nº. [u. p] (u V)u Vp V 0, for the velocity field (u, v) and pressure p, with = 1. The spatial domain Q is defined
Nolu V
Ω == ( \ UR. R
where each R; denotes circular obstacle. For further details about the boundary conditions please see the original PINNacle benchmark (Hao et al. 2024).
The impact of initialization observed figure 15 far more pronounced in that case than for Burger. We observe that having proper control over the spectral properties of the initialization can lead to significant improvement in performance. The Sitzmann initialization exhibits, expected, problematic high-frequency components, while other models such as FINER, Tanh, and ReLU fail completely reconstruct the physical solution.
26

---

Published as a conference paper at ICLR 2026
10
Figure 15: Results of the Navier-Stokes 2D solutions for the different state of the methods, using
network with width N 256 and L = 15. The networks were trained for 10 000 epochs using the ADAM optimizer with a learning rate of 10-4, For the SIREN based architectures, we chose wo = 2.
B.6.3 HEAT EQUATION IN COMPLEX GEOMETRY
We consider the transient 2D heat equation
(x 1) u, Au= O, (x, EQ x [0.3].
The spatial domain Ω is defined
Ω = [[-8,8) x 12, 12))
UR,
where each RG denotes a circular obstacle. For further detail about the boundary conditions please see the original PINNacle benchmark (Hao et al.) (2224).
Gooend brith
ATTEN (Wimmardd
Tean Sunter)
9:40
MP Libra
Figure 16: Results of the 2D heat equation experiments for the different state of the art methods, using a network with width N 256 and L = 15. The networks were trained for 10 000 epochs using the ADAM optimizer with learning rate of 10-4 For the SIREN based architectures, chose wo =
The results for different initializations are shown figure 16
The distinction between c, 1 and a, striking. The former produces noticeably noisy and unstable solutions, whereas setting 0, = 0 successfully reproduces the behavior of the ground-truth solution. For the other initialization methods, the observations are consistent with those made in the Navier-Stokes experiment.
B.7 SYNTHETIC EXPERIMENTS
B.7.1 1D FITTING EXPERIMENTS
For the ID fitting experiments, we generated synthetic data by sampling from multi-scale function:
fia(x) sin(3x) + 0.7 0.7.cos(8.r
+ 0.3 s40r 1) exp(-x²)
27

---

Published as a conference paper at ICLR 2026
To explore the impact of initialization the performance of various neural network architectures, we studied two tasks: function fitting and PDE solving. Since image and video fitting reduce to function fitting, focus it. This choice lets us control the target function's frequency content. As result, we can probe the different scales present in the data.
1d Averaged Generalization error
1d Averaged Training error
109
10-2-
10-1
10-2
10-6
10
10-10
10
10 12
10-5
10
8
10
Number of hidden layers
SIREN (a_ 0) SIREN (r,,
SIREN (Sitzmann)
Finer
ReLU (PE)
Wire
Tanh (Pourier-Xavier)
Figure 17: 1d Averaged generalization and training error for the 1D fitting problem. The results averaged over 10 runs for each architecture of width N = 128. The error bars represent the standard deviation of the results.
The results plotted figure 17 show that proposed initialization matches or exceeds the accuracy of the traditional SIREN architecture for fitting a function. Moreover, it delivers significantly lower generalization error compared the original SIREN. Notably, the Tanh-based positional positionaleencoding network also shows strong generalization performance, despite its slightly higher training error.
B.7.2 2D FITTING EXPERIMENTS
We applied the same methodology to a two-dimensional, multi-scale test function: f2a(0, y) (3x) cos(3y) + sin(15r cos(15y)
exp(- +2)))
for (x,y) E [3,,1]2 The exponential term ensures no architecture can represent the function trivially. We sampled 3600 random training points, giving Nyquist frequency above 15. Each network was trained for 5000 epochs using Adam (learning rate 10-4) under various initialization schemes. We then evaluated generalization error 10 000 test points. The comparative results appear in Fig. 18
The results mirror the ID fitting experiments. Our proposed initialization clearly outperforms all other architectures on the generalization task. At the same time, it maintains a very low training error, comparable the SIREN architecture.
B.7.3 3D FITTING EXPERIMENTS
For the 3D fitting experiments, dimensional function with multi-scale features:
the same framework in ID and 2D. We test three-
Jaa(æ, y, 2) sin()x)) cos coso(12)) (12y) sin(3z)
+ expp (2² + y2 2²)).
for (z. v. E 133 The exponential term prevents trivial representation by any architecture. We
sample 8000 random training points, ensuring a Nyquist frequency above 12. Each network trains for 5000 epochs using Adam with learning rate 10 -4 under various initialization schemes. We then evaluate generalization error 70 000 test points. The results appear in Fig. 19
28

---

Published as a conference paper at ICLR 2026
Averaged Generalization error
Averaged Training error
10P
10a
10-2
10-3
10-2
10-3
10-3
10-4
Number of hidden layers
SIREN (d. 0)
SIREN (Sitzmann) Wire
Finer
ReLU (PE)
SIREN (0.
Tanh (Fourier-Xavier)
Figure 18: 2d Averaged generalization and training error for the 2D fitting problem. The results
are averaged over 10 runs for each architecture of width N 1238. The error bars represent the standard deviation of the results.
Averaged Generalization error
3d Averaged Training error
10"
10-1
10
10
10-3
10
10
10
10-6
2
Number hidden layers
SIREN (r,, 0) SIREN (r,
SIREN (Sitemann)
Finer
ReLU (PE)
Wire
Tanh (Fourier-Xavier)
Figure 19: 3d Averaged generalization and training error for the 2D fitting problem. The results averaged over 10 runs for each architecture width N = 128. The error bars represent the standard deviation of the results.
Once again, proposed initialization delivers strong results. It clearly outperforms all other archi- tectures on generalization. Its fitting error remains very low, only slightly above the classic SIREN. Interestingly, the number of layers increases, SIREN's training error decreases alongside rising high-frequency content. This suggests that fitting high frequencies may harm generalization-a drawback our method avoids.
C ABLATION STUDIES
Since our theoretical analysis is derived in the infinite-width and infinite-depth regime, we also evaluate our model in the opposite setting: using small widths and very large depths. This allows us to examine, on one hand, how finite-size effects modify the experimental behaviour, and on the other hand, whether theoretical predictions remain valid when the depth becomes extremely large. This analysis further reveals how these factors influence the overall performance of such neural networks.
29

---

Published as a conference paper at ICLR 2026
C.I FINITE WIDTH EFFECT
The fintt experiment (with N - 32) leads to the same conclusions as the theoretical study: deep networks initialized with the Sitzmann scheme with oa exhibit high noise level. In contrast, our proposed initialization maintains a lower noise level (see the gradient section of Figure 20
even for very small widths, although it severely harms performance.
Greand
SIREN 2 N w33
SEREN Co Xiw
SIREN Cr. V.V 28
SEAEN Us - 190
Starere 201
Skoma CX
Training
TOR SUOVEE
Evaluation
SEMA CLSA
DAIIC CAAD
Gradient
Figure 20: Comparison of the discussed initialization method, and how finite width (N 32 and N = 128) affects their performance and behavior. The setting the experiments are the same as one described in Figure 2
C.2 LARGE DEPTH EFFECT
As shown Figure 21 the large-depth experiments with L = 10 and L = 40 confirm our previ- ous thcoretical discussion in the infinite-depth limit. In the case d, O, increasing the depth to L = 40 even improves performance and further reduces the effective noise level. For performance at large depth is surprisingly good, despite the clear growth of high-frequency com- ponents with depth observed in the Fourier spectrum (see Figure 4) this observation still holds at = 10. We attribute this behaviour to the long training time. For the Sitzmann original initializa- tion. expected, the increasing of depth severely impacts the generalization performances, due to overwhelming presence of high frequency components.
1, the
tran
SIREN =17.-
SEREN -.
SIRENIs In. 1.1 1e
SEVEN b.. 1
Stmerrat !1
SZomann CC.- 80
Training
2IEA
WLLLER
Evaluution
ANG SAARER
ARE DUAVE
VIG SLILER
Cradlent
Figure 21: Comparison of the discussed initialization method, and how large depth affect their performance and behavior. The setting of the experiments are the same as one described in Figure
30


Reviews:
[{"review_id": "bVtzSkswPL", "reviewer": "Reviewer_K6uc", "summary": "This work introduces a theoretically grounded initialization scheme for SIRENs that enforces:\n (1) the preservation of the pre-activation distribution across layers, and\n (2) stable gradients throughout the network (avoiding vanishing or exploding gradients).\n\nThe authors evaluate their method on image-fitting tasks and demonstrate stable training in deep SIRENs. Furthermore, they leverage Neural Tangent Kernel (NTK) theory to show that the proposed initialization avoids the introduction of spurious high-frequency components at the start of training.\n\nTraining deep SIRENs is a challenging problem with several applications in implicit neural representations (INRs). I tested the proposed initialization on image-fitting experiments and found it promising. However, a straightforward extension to SDF fitting did not yield good results, suggesting that this adaptation may not be trivial.", "questions": "[L25] When stating that the new initialization “consistently outperforms state-of-the-art methods,” which methods are considered SOTA? If only Xavier, Kaiming, and SIREN, this comparison is limited, these are not typically used for signal representation. Please also consider TUNER [1], FINER [3], and [4].\n\n[L47] Does the proposed initialization accelerate training convergence?\n\nHow does the method perform on high-frequency audio fitting, where deeper architectures might be more beneficial?\n\n[L188] Does the initialization assume identical hidden-layer widths (same N) across the network?\n\n\n[1] Novello et al., Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks, CVPR 2025.\n\n[2] Yüce et al., A Structured Dictionary Perspective on Implicit Neural Representations, CVPR 2022.\n\n[3] Liu et al., FINER: Flexible Spectral-Bias Tuning in Implicit Neural Representation by Variable-Periodic Activations, CVPR 2024.\n\n[4] Yeom et al., Fast Training of Sinusoidal Neural Fields via Scaling Initialization, ICLR 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and well organized.\n\nThe proposed initialization is supported by NTK theory, which provides a solid theoretical foundation.\n\nThe empirical results indicate robustness in fitting tasks.\n\nImproving the trainability of deep SIRENs has significant implications for a wide range of INR applications.", "weaknesses": "[L53–54] It is unclear whether the authors refer to (1) the gradient of the loss function with respect to network parameters or (2) the derivative of the network output with respect to input coordinates. In L212, it seems to refer to (2). This point is addressed in Eqs. 10–11.\nCould the authors include a specific example where gradients explode under the standard SIREN initialization but remain stable with the proposed one? In Fig. 1, both methods seem to yield noisy reconstructions. Including analytical gradient visualizations (as in TUNER) would provide a clearer comparison.\n\n[L107] TUNER [1] is a closely related work proposing frequency-aware initialization for SIRENs. It also offers a formula that could explain the emergence of high-frequency artifacts when increasing network depth [L43]. I recommend including it in the comparisons. Additionally, FINER [3] is another recent method addressing similar initialization and spectral control challenges.\n\n[L132] Evaluation is restricted to RGB image fitting with an MSE loss (Eq. 4).\n\n[L144] INRs are generally trained to represent low-dimensional signals (e.g., 2D images, SDFs, occupancy, or radiance fields), often including a regularization term (e.g., the Eikonal constraint for SDFs).\nIt would strengthen the paper to show results for SDF fitting or at least discuss how the proposed initialization behaves under such regularization. I attempted to implement the initialization for SDF reconstruction using the repository from “Exploring Differential Geometry in Neural Implicits” and observed poor convergence.\n\n[L90] Yüce et al. [2] should be cited, as they also analyze SIRENs through the NTK framework.\n\n[L159] Mention explicitly that \\theta denotes the union of all \\theta_i​’s.\n\n[Fig. 2] A 10-layer MLP with 256 neurons per layer is quite large for image fitting. Testing smaller architectures (e.g., 16 or 32 neurons) would help assess scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62Eeumlgmh", "reviewer": "Reviewer_s653", "summary": "\\paragraph{Summary}\nThe paper proposes a closed-form initialization for SIREN-style sinusoidal networks. Hidden-layer weights and biases are set via two scalars \\((c_w, c_b)\\) chosen so that pre-activations converge to a fixed variance and the layerwise Jacobian entry variance satisfies\n$\n\\sigma_g^2 = \\frac{1}{N}.\n$\nSolving these constraints yields\n$\nc_w = \\sqrt{\\frac{6}{1+e^{-2}}}, \\qquad\nc_b = \\sqrt{\\frac{1}{3}}\\, c_w e^{-1}\n\\quad \\text{(Eqs.~(7)--(8)).}\n$\nExperiments on 1D/2D/3D synthetic functions and a single image-fitting task suggest improved “generalization” (via upsampled reconstructions) over baselines. NTK measurements indicate the mean NTK eigenvalue grows roughly linearly with depth under the proposed initialization, versus exponential growth.", "questions": "- Can you justify or bound $\\mathrm{Cov}(W_{\\ell,ik}, z_{\\ell,i})$ as width $N$ grows? A tensor-program or leave-one-out derivation—or empirical covariance vs.\\ $N$—would strengthen Theorem~3.2.\n - Please report singular-value distributions (or spectral norms/condition numbers) of end-to-end Jacobians across depths and widths for multiple inputs, to demonstrate stability beyond entrywise variance.\n- Beyond the trace, provide the NTK eigenvalue spectrum vs.\\ depth (including tails and condition numbers) and analyze eigenvector frequency alignment to support claims about depth-wise learning dynamics.\n - Add kernel-bandwidth SIREN inits and residual/skip/normalized SIREN variants. Include realistic INR settings (e.g., NeRF or PINNs) with standard metrics (PSNR/SSIM/relative $L_2$).\n- Evaluate under noisy/irregular sampling, distribution shift, and different input scalings to test whether the fixed-point targeting remains effective.\n- Will you release code, seeds, and exact training logs? A concise “How to initialize” box in the main text would aid adoption.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- A simple, analytical initialization for sine activations; easy to implement (Eqs. (7)–(8)). \n\n- Clear fixed-point analysis for pre-activation variance using Lambert-W; explicit gradient-variance target. \n\n- Useful NTK-based depth scaling narrative; the paper measures trace/eigenvectors and links depth to speed. \n\n- Repeated small-scale experiments show consistent wins on their metrics.", "weaknesses": "- The Jacobian-variance factorization assumes $W_\\ell \\perp z_\\ell$ even though $z_\\ell = W_\\ell h_{\\ell-1} + b_\\ell$. Without a large-width justification (e.g., leave-one-out / tensor-program) the claim that targeting $\\sigma_g^2 = 1/N$ stabilizes gradients is fragile.\n- Matching entrywise Jacobian variance does not ensure favorable singular-value spectra of end-to-end Jacobians (dynamical isometry). No SVD/condition-number results are reported.\n- No correlation recursion or $\\chi$-coefficient/fixed-point correlation study to substantiate an EoC claim beyond gradient-variance targeting.\n- Mostly small synthetic INR tasks and a single image; no NeRF/PINN or noisy/irregular sampling settings. Image evaluation uses upsampled reconstructions instead of standard PSNR/SSIM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYaZY2eXNh", "reviewer": "Reviewer_1Ux1", "summary": "The paper proposes a new initialization for INRs that rely on sinusoidal activation functions. Choosing the parameters of this new initialization is theoretically motivated by avoiding vanishing/exploding gradient variances of the network with respect to the network depth and width. This is done by a series of theoretical results relying on NTK and assuming it is constant at the beginning of training. The result is the avoidance of spurious frequencies and other quality degradation that occurred otherwise when increasing the depth of the network using previous types of initialization. Empirical evaluations show better generalization when using the proposed initialization.", "questions": "Please, see the Weaknesses section for most questions. One additional question:\n- What happens if in equation (7) one decides to keep the bias uniformly sampled (as in equation (6)), instead of Gaussian sampled, while keeping the rest of equation (7) the same. I wonder what kind of performance will result from it—it could be a good ablation to try.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper’s strengths are:\n- The paper studies a relevant problem which is theoretically motivated and has practical applications.\n- The paper’s motivation and contributions are overall clear.\n- The paper makes good connections with existing literature in both its applied and theoretical derivations.", "weaknesses": "I have a series of things to point out about the paper, which are the reason for the current score assigned to the paper.\n\n**>>About the results:**\n- It is my understanding that the SIREN initialization does not work well for large depths, and it is argued that the initialization by the authors (equation (7)) does not suffer from it. Well, this is really hard to know *for sure*: I understand that the theoretical results were derived in the large depth limit and using the NTK at initialization, but in practice, we are not in the NTK regime across training, so things could be very different. It would be important to try experiments where one keeps increasing the network’s depth using the authors’ initialization and see if there is a **tipping point** after which larger networks lead to deterioration on generalization tasks (due to spurious high frequencies or another cause). This is largely unexplored in the paper. If such tipping point is found, it will establish an important limitation on the practicality of the authors’ method. One could compare such a tipping point to the value of depth over which SIREN leads to deterioration—would such tipping point be x2, x10, etc. magnitude larger?\n- There is something which is not clear at all to me: the meaning of the expression “limit of large $N$” in Theorem 3.2. When talking about limits, either you (i) have $N\\to C$ where $C$ is a constant that is much larger than some other parameters, or you (ii) have $N\\to\\infty$. I believe Theorem 3 refers to item (i), since item (ii) does not make much sense because the equation of $\\sigma_g^2$ depends on $N$. Then, what are the other parameters that are less than the constant $C$ that I defined above? I hope my question makes sense. The expression “limit of large $N$” **makes no sense** unless **you specify what makes $N$ “large”**. This is important to know for the paper. \n- As a follow-up to the previous point: is equation (12) also under the “limit of large $N$”? If so, it would be good to make it explicit in the text and respond to my previous point — otherwise, a reader **can’t know what makes $N$ large enough for these expression to hold**.\n- Another problem is that Theorem 3.2’s  results are under the same assumptions as Theorem 3.1.—i.e., assuming the **random initialization** proposed by the authors. Thus, the results of Theorem 3.2 such as the expression of $\\sigma_g^2$ **must hold under some probability**. Can the authors clarify the probability under which these results hold? This is also very important for the formality of the results—until now, the reader can’t know if the results hold with probability $1$ or some probability dependent on problem parameters (such as the network width $n_0$, for example).\n\n**>>Motivation:**\n- In the last paragraph of the contributions in Section 1.3, it is mentioned that various results from the paper “will be valuable beyond the specific case of INR with sinusoidal activations”. Can the authors list examples of other cases beyond “INR with sinusoidal activations” for which their results are valuable and informative? All the results derived in the paper are highly specific to the setting of INR with sinusoidal activations, how is it then that the paper’s results can be transferred to other application domains?\n- Lines 175-178: why is it important to ensure differentiability in the applications mentioned therein?\n- The paper focuses on INR and SIREN, nonetheless, there is not much mention about areas (with citations) where it has been successfully applied. I see that PDE-related applications are mentioned (without providing a citation), but I am sure there are more applications of interest. This will strengthen the motivation of the paper.\n- In Section 4, a considerable assumption is made: that the NTK remains constant during training. Nonetheless, this, to my best understanding, is not something that will commonly hold in practice. The paper seems to claim this is the case, since it mentions “the early training dynamics is fully determined by the spectral properties of the NTK at initialization”. Can the authors provide references/citations or a more formal justification for such a claim? If I were to make a guess, networks with very large width are the ones that more likely will preserve the spectral properties of the NTK given at initialization after a few iterations of gradient descent with a small learning rate (but this is a guess, the authors should look at the literature or provide a good formal justification).\n- In the last paragraph of Section 5 it is mentioned that the “pre-activation variance ($\\sigma_a^2=1$) may be adapted and optimized in future work”. However, how could anyone decide to change the value of $\\sigma_a^2$ when this could possibly lead to vanishing/exploding gradients (at least close to initialization, if I understood correctly)?\n\n**>>Clarification:**\n- Line 049-051: it seems to state that it is known that there is a frontier between vanishing-gradient and exploding-gradient that one needs to fit the training in. Do you have a reference for it? How is it known (e.g., from previous literature) that “spurious high-frequency artifacts” can lead to both vanishing and exploding gradients during training?\n- Line 051: it is said that we seek a regime where “gradients remain stable”. Does “stable” mean that the norm of the gradient will be uniformly bounded below and above by a non-zero constant?\n- Figure 1: for each vertical bar, what does the difference between the two colors (dark vs light gray) represent?\n- In the example described in the paragraph starting at line 345: how many input points are used for training?\n\n**>>Other clarifications:**\n- About Section 1.2, first paragraph: it is said that SIREN has the tunable parameter $w_0$ to control the frequency range of the network—however, this is only guaranteed at initialization, nothing is known during training, correct? (Unless you are in the NTK domain). Please, clarify this in the text.\n- About Section 1.2, first paragraph: it is said that SIREN is related to other alternatives such as positional encoding and Random Fourier features. Why would somebody use SIREN instead of these other alternatives? Is it all about the calibration of $w_0$? If so, why is this so important compared to other approaches that allow for frequency control?\n- In Theorem 3.1, clarify in the last sentence that the convergence is as $L \\to \\infty$.\n\n**>>Other things:**\n- Line 039: please, provide examples to better understand “signals of high-frequency content”. I imagine learning functions that correspond to images with considerable texture could be one example, is that correct?\n- Please, specify that you consider the $l_2$-norm, e.g., in equation (4).\n- In line 254: indicate after $W_l$ that $2\\leq l \\leq L$.\n- For consistency, use “decay” instead of “converge” when referring to the small eigenvalues below line 377.\n- Please, proofread the paper for typos and grammatical errors which **should have not happened** at this point. Examples:\n  - Delete “have” and the “-d” at the end of “converged” in the last line of page 4 (below line 215).\n  - Add parenthesis after “Hinton (2010)” in line 242. Also, “Nair & Hinton (2010)” should be “(Nair & Hinton, 2010)” since it is referring to the paper itself, not the authors as subjects. For this, we can use “\\citep” in Latex. More instances of this mistake are found throughout the paper.\n  - Title of section 3.1: it says “PRE-CTIVATION”, it should be “PRE-ACTIVATION”.\n  - end of line 254: it should say “$W_1$ is sampled” instead of “let $W_1$ be sampled”.\n  - line 255: add “and” before “$b_l$”.\n  - The first word in Remark 3.2 should be “As”.\n  - Line 290: the beginning should be “Under the same assumptions as Theorem 3.1,\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zUHV3XoDu", "reviewer": "Reviewer_Le3d", "summary": "This paper studies the problem of vanishing/exploding gradient of sinusoidal neural networks at initialization. Based on theoretical analysis, it argues that the prior standard initialization scheme should be adjusted by an additional factor to prevent the vanishing/exploding gradient issue. Its empirical results on an image-fitting problem and NTK analysis support this new initialization strategy.", "questions": "See comments in weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1: The new initialization method is theoretically justified to be better in stabilizing the gradient when network depth is large. \n\n2: The efficacy of the new method is empirically verified in a real-world problem, and is also supported by the numerical analysis of the NTK matrix.\n\n3: The new initialization method is simple to implement and requires small extra computation.\n\n4: The paper is well-presented and is easy to follow.", "weaknesses": "1: The paper claims a causal relation between the gradient control and (spurious) high-frequency modes. For example, in abstract (Line 21) and introduction (Line 50, 55). However, I don’t quite see a proof or analysis of this causal relation in the paper. It seems the most related is Eq. 12, but this equation only shows a correlation, not causality. In addition, this correlation is loose: a stable gradient not necessarily tied with no spurious high frequency, as it also depends on $\\omega_0$.\n\n2: The scope of the paper is confined to the setting of $sin$ activation function. Does it extend to commonly used activation functions, such as ReLU? I know there are some empirical results on the image-fitting problem in Figure 2 for other activations. I’d like to know whether it extends beyond this problem, such as image classification, image generation where ReLU (or similar) are much more often used than $sin$. In addition, I’d like to see whether the theory extends.\n\n3: It would be better if there is more empirical evidence, in addition to the image-fitting problem.\n\n4: The paper exceeds the page limit by approximately one-third of a page and includes an acknowledgment section that may partially reveal the authors’ identities.\n\nIf the above are addressed, I am happy to increase the rating", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)