Simple initialization and parametrization of sinusoidal networks via their kernel bandwidth
Authors: Filipe de Avila Belbute-Peres
J Zico Kolter
Venue: ICLR 2023 poster (2023)
Keywords: sinusoidal
periodic
neural tangent kernel
implicit models
physics informed

Abstract: Neural networks with sinusoidal activations have been proposed as an alternative to networks with traditional activation functions. Despite their promise, particularly for learning implicit models, their training behavior is not yet fully understood, leading to a number of empirical design choices that are not well justified. In this work, we first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth. Finally, we utilize these insights to inform the sinusoidal network initialization, optimizing their performance for each of a series of tasks, including learning implicit models and solving differential equations.

Published as a conference paper at ICLR 2023
SIMPLE INITIALIZATION AND PARAMETRIZATION OF SI- NUSOIDAL NETWORKS VIA THEIR KERNEL BANDWIDTH
Filipe de Avila Belbute-Peres Carnegie Mellon University Pitsburgh, PA
J. Zico Kolter
Carnegie Mellon University & Bosch Center for Al Pitthurgh, PA z zko welleeercs cmm.cmc.e...
filipedecs. cmu edu
ABSTRACT
Neural networks with sinusoidal activations have been proposed as an alternative to networks with traditional activation functions. Despite their promise, particularly for learning implicit models, their training behavior not yet fully understood, leading number of empirical design choices that not well justified. In this work, we first propose simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis. We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kemel approximates a low-pass filter with an adjustable bandwidth. Finally, we utilize these insights to inform the sinusoidal network initialization, optimizing their performance for each of a series of tasks, including learning implicit models and solving differential equations.
INTRODUCTION
Sinusoidal networks neural networks with sine nonlincarities, instead of the traditional ReLU or hyperbolic tangent. They have been recently popularized, particularly for applications in implicit rep- resentation models, in the form of SIRENs (Sitzmann et al.) 2020). However, despite their popularity, many aspects of their behavior and comparative advantages are not yet fully understood. Particularly, some initialization and parametrization choices for sinusoidal networks are often defined arbitrarily, without a clear understanding of how to optimize these settings in order to maximize performance.
In this paper, we first propose simplified version of such sinusoidal networks, that allows for easier implementation and theoretical analysis. We show that these simple sinusoidal networks can match and outperform SIRENs in implicit representation learning tasks, such as fitting videos, images and audio signals. We then analyze sinusoidal networks from a neural tangent kernel (NTK) perspective (Jacot et al. 2018), demonstrating that their NTK approximates a low-pass filter with adjustable bandwidth. We confirm, through an empirical analysis this theoretically predicted behavior also holds approximately in practice. We then use the insights from this analysis to inform the choices of ini- tialization and parameters for sinusoidal networks. We demonstrate of a sinusoidal network by tuning the bandwidth of its kernel to the maximum frequency present in the input signal being learned. Finally, we apply these insights in practice, demonstrating that "well tuned" sinusoidal networks outperform other networks in learning implicit representation models with good interpolation outside the training points, and in learning the solution to differential equations.
can optimize the performance
BACKGROUND AND RELATED WORK
Sinusoidal networks. Sinusoidal networks have been recently popularized for implicit modelling tasks by sinusoidal representation networks (SIRENs) (Sitzmann et al. 2020). They have also been evaluated for physics-ifformed learning, demonstrating promising results in series of domains (Raissi et al. 2019b] Song et al. 2021 Huang et al. 2021b a Wong et al. 2022). Among the benefits of such networks is the fact that the mapping of inputs through (initially) random linear layer followed by a sine function is mathematically equivalent to a transformation to a random Fourier basis,
rendering them close networks with Fourier feature transforms (Tancik al. 2020 Rahimi & Recht

---

Published as a conference paper at ICLR 2023
2007), and possibly able to address spectral bias (Basri et al. 2019 Rahaman al. 2019 Wang et al. 2021). Sinusoidal networks also have the property that the derivative of their outputs is given simply by another sinusoidal network, due to the fact that the derivative of sine function is phase-shifted sine.
Neural tangent kernel. An important prior result to the neural tangent kernel (NTK) is the neural network Gaussian process (NNGP). At random initialization of its parameters A, the output function of a neural network of depth L with nonlinearity Ø, converges to a Gaussian process, called the NNGP, the width of its layers :1,..... , + 30. (Neal 1994, [ee et al. 2018). This result, though interesting, does not say much on its own about the behavior of trained neural networks. This role is left to the NTK, which is defined as the kernel given by (z,, shown that this kernel can be written out as recursive expression involving the NNGP. Importantly, Jacot et al. (2018) demonstrated that, again as the network layer widths ; nl -> xo, the NTK is (1) deterministic at initialization and (2) constant throughout training. Finally, it has also been demonstrated that under some assumptions on its parametrization, the output function of the trained neural network fe converges to the kernel regression solution using the NTK (Lee et al. 2020 Arora ct al. 2019). In other words, under certain assumptions the behavior of a trained deep neural network can be modeled as kernel regression using the NTK.
(z)), fefo (z)). It can be
Physics-informed neural networks. Phyyics-informme neural networks (Raissi et al. 2019a) are a method for approximating the solution to differential equations using neural networks (NNs). In this method, a neural network ū(,, xz; f), with learned parameters 6, is trained to approximate the actual solution function u(t, to a given partial differential equation (PDE). Importantly, PINNs employ not only a standard "supervised" data loss, but also a physics-ins physics-informed loss, which consists of the differential equation residual N Thus, the training loss consists of a linear combination of two loss terms, one directly supervised from data and one informed by the underlying differential equations.
3 SIMPLE SINUSOIDAL NETWORKS
There are many details that complicate the practical implementation of current sinusoidal networks. We aim to propose simplified version of such networks in order to facilitate theoretical analysis and practical implementation, by removing such complications.
As an example we can look SIRENs, which have their layer activations defined as fxx) sin (w< + br)). Then, in order to cancel the w factor, layers after the first one have their weight
initialization follow uniform distribution with range
/0/n ya/n where is the size of the layer. Unlike the other layers, the first layer is sampled from a uniform distribution with range [-1/n, 1/m].
|- of
We instead propose simple sinusoidal network, with the goal of formulating an architecture that mainly amounts to substituting its activation functions by the sine function. We will, however, keep the w parameter, since (as we will see in future analyses) is in fact a useful tool for allowing the network to fit inputs of diverse frequencies. The layer activation equations of our simple sinusoidal network, with parameter w, are defined as
fi (x) = sin(w (Wix + bi)), f((x)) (W, + by), 1> 1.
(I)
Finally, instead of utilizing a uniform initialization as in SIRENs (with different bounds for the first and subsequent layers), we propose initializing all parameters in our simple sinusoidal network using a default Kaiming (He) normal initialization scheme. This choice not only greatly simplifies the
initialization scheme of the network, but it also facilitates theoretical analysis of the behavior of the network under the NTK framework, as we will see in Section
Analysis of the initialization scheme. The initialization scheme proposed above differs from the one implemented in SIRENs. We will now show that this particular choice of initialization distribution preserves the variance of the original proposed SIREN initialization distribution. As a consequence, the original theoretical justifications for its initialization scheme still hold under this activation, namely that the distribution of activations across layers are stable, well-behaved and shift-invariant. Due to space constraints, proofs are presented in Appendix A Moreover, we also demonstrate empirically that these properties are maintained in practice.
Lemma 1. Given any e, for X N (0. )c2) and Y ~ 21 (-c,c), have Var Var[X]= Var[Y] 3.2.
2

---

Published as a conference paper at ICLR 2023
This simple Lemma and relates to Lemma 1.7 in Sitzmann et al. (2020), showing that the initialization we propose here has the same variance as the one proposed for SIRENs. Using this result we can translate the result from the main Theorem 1.8 from [Sitzmann et al. (020). which claims that the SIREN initialization indeed has the desired properties, to our proposed initialization |
For uniform input in 1,,,1}, the activations throughout a sinusoidal network are approximately standard normal distributed before each sine mineiiiyrrty and presine-distribeee after each sine non-linearity, irrespective of the depth of the network, if the weights are distributed normally, with mean 0 and variance E. where Tu is a layer's 's fan-in.
Empirical evaluation of initialization scheme. To empirically demonstrate the proposed simple ini- tialization scheme preserves the properties from the SIREN initialization scheme, we perform the same analysi sis performed by /Sizzmann et al. (2020). We observe that the distribution of activations matches the predicted normal (before the non- iinariiy)) and arcsine (after the non- non-linearity) distributions, and that this behavior is stable across many layers. These results are reported in detail in the Appendix B
3.1 COMPARISON TO SIREN
In order to demonstrate our simplified sinusoidal network has comparable performance to a standard SIREN, in this section we reproduce the main results from Siizmann et al. (2020). Table T compiles the results for all experiments. In order to be fair, compare the simplified sinusoidal network proposed in this chapter with both the results directly reported in Sitzmann et al. (2020). and our own reproduction of the SIREN results (using the same parameters and settings as the original). We can see from the numbers reported in the table that the performance of the simple sinusoidal network proposed in this chapter matches the performance of the SIREN in all cases, in fact surpassing it in most of the experiments. Qualitative results are presented in Appendix C
It is important note that this not a favorable setting for simple sinusoidal networks, given that the training durations were very short. The SIREN favors quickly converging to solution, though it does not have as strong asymptotic behavior. This effect is likely due to the multiplicative factor applied to later layers described in Section 3 We observe that indeed in almost all cases we can compensate for this effect by simply increasing the learning rate in the Adam optimizer (Kingma & Ba. 2014).
Finally, we observe that besides being able to surpass the performance of SIREN in most cases in a short training regimen, the simple sinusoidal network performs even more strongly with longer training. To demonstrate this, we repeated some experiments from above, but with longer training durations. These results are shown in Table in Appendix C
4 NEURAL TANGENT KERNEL ANALYSIS
In the following we derive the NTK for sinusoidal networks. This analysis will show us that the sinu- soidal networks NTK is approximately a low-pass filter, with its bandwidth directly defined by w. We support these findings with an empirical analysis as well in the following section. Finally, we demon- strate how the insights from the NTK can be leveraged to properly "tune" sinusoidal networks to the spectrum of the desired signal. Full derivations and extensive, detailed analysis are left to Appendix [
The NTK for simple sinusoidal network with single hidden layer is presented in the theorem below. The NTK for siren with 1 and 6 hidden layers are shown in Figure
Theorem 2. Shallow SSN NTK. For a simple sinusoidal network with one hidden layer f()) RT +
RT= following Definition] 7 its neural langent kernel (NTK), as defined in Theorem 6 is given by
(w²
T +
1)
+ 1)
1
+ 1) 1) 10t5
- 2nd + +=
We note that despite being named Theorem 1.8 Sitzmann et al. (020)), this result is not fully formal, due to the Gaussian distribution being approximated without a formal analysis of this approximation. Additionally, a CLT result is employed which assumes infinite width, which is not applicable in this context. We thus refrain from calling our equivalent result theorem. Nevertheless, the extent that the argument is applicable, it would still hold for our proposed initialization, due to its dependence solely on the variance demonstrated in I. a
3

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of the simple sinusoidal network and SIREN results, both directly from Sitzmann et al. (2020) and from our own reproduced experiments. Values above the horizontal center line are peak signal noise ratio (PSNR), values below are mean squared error (MSE), except for SDF which uses composite loss. Audio experiments utilized separate learning rate for the first layer.
Experiment
Simple Sinusoidal Network
SIREN [paper]
SIREN [ours]
Image
50.04
49 (approx.)
49.0
Poisson (Gradient)
39.66
32.91
38.92
Poisson (Laplacian)
20.97
14.95
20.85
Video (cat)
34.03
29.90 32.88
32.09
Video (bikes)
37,4
33.75
Audio (Bach))
1.57 10-5
1.10 10 5
3.28 10
Audio (counting)"
3.17 10-4
3.82 10-4
4.38 10
Helmholtz equation
5.94 100
5.97 10-2
SDF (room)
12.99
14.32
-
SDF (statue)
6.22
5.98
We can see that for values of w > 2, the second term quickly vanishes due to the e 2nd factor. This leaves us with only the first term, which has Gaussian form. Due to the linear scaling term x s, this is only approximately Gaussian, but the approximation improves as increases. We can thus observe that this kernel approximates a Gaussian kernel, which a low-pass filter, with its bandwidth presents visualizations for NTKs for the simple sinusoidal network, compared
defined by w. Figure 1 to (scaled) pure Gaussian with variance - showing there a close match between the two.
If we write out the NTK for networks with more than hidden layer, it quickly becomes un-interpretable due to the recursive nature of the NTK definition (see Appendix D). However, as shown empirically in Figure [ these kernels are still approximated by Gaussians with variance =2
We also observe that the NTK for SIREN with single hidden layer is analogous, but with sinc form, which is also low-pass filter.
Theorem 3. Shallow SIREN NTK. For a single hidden layer SIREN (()) Ro + R following
Definition] 7 its neural langent kernel (NTK), as defined in Theorem is given by
²² (w2
9 (1) (z,
(I
+
+1)
II
sinc(cw (xy - āz))
6
1=1
(+T
1) e 2 Il sinclew (xz + ū2)) + w² 2x²(x=++ +1 + 1.
For deeper SIREN networks, the kernels defined by the later layers are in fact Gaussian too, as discussed in Appendix D This leads to an NTK that is approximated by a product of sine function and a Gaussian. These SIREN kernels are also presented Figure]
5 EMPIRICAL ANALYSIS
As shown above, neural tangent kernel theory suggests that sinusoidal networks work as low-pass filters, with their bandwidth controlled by the parameter us In this section, we demonstrate empirically that we can observe this predicted behavior even in real sinusoidal networks. For this experiment, generate a 512 x 512 monochromatic image by super-imposing two orthogonal sinusoidal signals, each consisting of single frequency, f(x, y) cos (2)8mr) cos(32ry) This function is sampled in the domain |-,, 1]2 to generate the image on the left of Figure 2
To demonstrate what signal, we perform a discrete Fourier transform (DFT), cut off frequencies above a certain value, and perform inverse transform recover the (filtered) image. The MSE of the reconstruction,
expect from applying low-pass filters of different bandwidths this
4

---

Published as a conference paper at ICLR 2023
5SW [xx
300
SSN O (x.,))
SUREN e R), A 10
SIREN 2003
1500
2003
493
50% off..e 2. 10
Offic EL 2,000030
SAEN 34N.pPny...--Gaamm
L
m M
500 Ft P. Min
10
W 30
10
w 30
SSN
SIREN
Figure 1: The NTK for SIREN and SSN at different Top: Kernel values for pairs i) E []1 Bottom: Slice at fixed ğ = 0. SSN plots show a superimposed Gaussian kernel with variance w scaled match the max and min values of the NTK. Similarly, SIREN plots show sinc function.
Ground truth
64
Figure 2: Left: The test signal used to analyze the behavior of sinusoidal networks. It is created from two orthogonal single frequencies, f(x,y) cos( c0s(1287T)) + cos(32 Right: Examples of the reconstructed signal from networks with different w, demonstrating each of the loss levels in Figure
function of the cutoff frequency, is shown in Figure 3 We can see that due the simple nature of the signal, containing only two frequencies, there are only three loss levels. indeed the NTK analysis is correct and sinusoidal networks act as low-pass filters, with bandwidth controlled by w, should be able to observe similar behavior with sinusoidal networks with different w values. We plot the final training loss and training curves for sinusoidal networks with different in Figure 3] We can observe, again, that there are three consistent loss levels following the magnitude of the w parameter, line with the intuition that the sinusoidal network is working low-pass filter. This also observable in Figure 2 where we see example reconstructions for networks of various w values after training.
However, unlike with the DFT low-pass filter (which does involve any learning), see in Figure 3 that during training some sinusoidal networks shift from one loss level to lower one. This demon- strates that sinusoidal networks differ from true low-pass filters in that their weights can change, which implies that the windwidth defined by w also changes with learning. We know the weights W, in the first layer of sinusoidal network, given by fi (x) sin (w W x: + b,). will change with training. Empirically, we observed that the spectral norm of W, increases throughout training for small w values. We can interpret that as the overall magnitude of the term w W x increasing, which is functionally equivalent to increase in itself. In Figure we observe that sinusoidal networks with smaller values take a longer time to achieve a lower loss (if at all). Intuitively, this happens because, due to the effect described above, lower values require larger increase in magnitude by the weights W, Given that all networks were trained with the same learning rate, the ones with a smaller w require their weights move longer distance, and thus take more training steps to achieve lower loss.
5

---

Published as a conference paper at ICLR 2023
Reconstruction
Ricco Rccorstraction
044
0.1
04-
150
203
2208
15,00
Figure 3: Left: Reconstruction loss for different cutoff frequencies for a low-pass filter applied to Figure 2 The three loss levels reflect the 2 frequencies present in the simple signal: 128 and 32. Superimposed is the final loss, after fitting the same input image with sinusoidal networks, for different values of (scaled to align with the frequencies). Right: Training curves for the sinusoidal networks. The three loss levels reflect the frequencies present the simple signal, and demonstrate that the sinusoidal network is indeed acting low-pass filter with bandwidth defined by
TUNING w
As shown in the previous section, though the bandwidth of a network can change throughout training, the choice of w still influences how casily and quickly (if at all) it can learn a given signal. The value of the w parameter is thus crucial for the learning of the network. Despite this fact, in SIRENs, for example, this value is not adjusted for each task (except for the audio fitting experiments), and is simply set empirically to an arbitrary value. In this section, we seek to justify proper initialization for this parameter, such that it can be chosen appropriately for each given task.
Moreover, it is often not the case that we simply want fit only the exact training samples but instead want to find a good interpolation (i.e., generalize well). Setting too high, and thus allowing the network to model frequencies that are much larger than the ones present in the actual signal is likely to cause overfitting. This is demonstrated empirically in Figure
Consequently, we want instead tune the network to the highest frequency present in the signal. However, we do not always have the knowledge of what is the value of the highest frequency in the true underlying signal of interest. Moreover, we have also observed that, since the network learns and its weights change in magnitude, that value in fact changes with training. Therefore, the most we can hope for to have a good heuristic to guide the choice of w.. Nevertheless, having a reasonable guess for w is also likely sufficient for good performance, precisely due to the ability of the network to adapt during training and compensate for a possibly slightly suboptimal choice.
Choosing from the Nyquist frequency. One source of empirical information the relationship between w and the sinusoidal network's 's "learnable frequencies" is the previous section's 's empirical analysis. Taking into account the scaling, can see from Fig. thaat around w 16 the network starts to be able to learn the full signal (freq. 128). We can similarly note that at about w = 4 the sinusoidal network starts to be able to efficiently learn signal with frequency 32, but not the one with frequency 128. This scaling suggests a heuristic of setting w to about 1/8 of the signal's 's maximum frequency.
For natural signals, such as pictures, it is common for frequencies up the Nyquist frequency of the discrete sampling to be present. We provide an example for the "camera" image we have utilized so far in Figure in Appendix [B where we can see that the reconstruction loss through low-pass filter continues to decrease significantly up to the Nyquist frequency for the image resolution.
In light of this information, analyzing the choices of for the experiments in Section 3.1 again suggests that w should be set around 1/8 of the Nyquist frequency of the signal. These values of w are summarized in Table in the "Fitting w" column. For example, the image fitting experiment shows that, for an image of shape 512 x 512 (and thus Nyquist frequency of 256 for each dimension), this heuristic suggests an w value of 256/8 = 32, which is the value found to work best empirically through search. We find similar results for the audio fitting experiments. The audio signals used in the audio fitting experiment contained approximately 300, 000 and 500, 000 points, and thus maximum frequencies of approximately 150, 00 and 250, 000. This suggests reasonable values for of 18, 750 and 31, 250, which are close to the ones found empirically to work well. In examples such as the video fitting experiments, in which each dimension has different frequency, it is not completely

---

Published as a conference paper at ICLR 2023
clear how to pick single w to fit all dimensions. This suggests that having independent values of w for each dimension might be useful for such cases, as discussed in the next section.
Finally, when performing the generalization experiments in Section 7 we show the best performing w ended up being half the value of the best w used in the fitting tasks from Section [3.]] This follows intuitively, since for the generalization task set apart half the points for training and the other half for testing, thus dividing the maximum possible frequency in the training sample in half, providing further evidence of the relationship between w and the maximum frequency in the input signal.
Multi-dimensional w. In many problems, such the video fitting and PDE problems, not only is the input space multi-dimensional, it also contains time and space dimensions (which are additionally possibly of different shape). This suggests that employing frequencies for each dimension might be beneficial. In practice, if we employ scaling factor Ad)" we have the first layer of the sinusoidal network given by X = (A1 A
specifying different
f1()) = sin(w (W1 (A + b,)) = W11 (9 @ x) + wb1),
(2)
where Ω [iy
Aaw] works as a multi-dimennional w. In the following experiments, we employ this approach to three-dimensional problems. in which we have time and differently shaped space domains, namely the video fitting and physics-informed neural network PDE experiments. For these experiments, we report the w in the form of the (already scaled) Ω vector for simplicity.
Apw
Choosing w from available information Finally, in many problems we do have some knowledge of the underlying signal we can leverage, such as in the case of inverse problems. For example, let's say we have velocity fields for a fluid and we are trying to solve for the coupled pressure field and the Reynolds number using a physics-informee have access to two components of the solution field. Performing Fourier transform on the training data we have can reveal the relevant spectrum and inform our choice of w. If the maximum frequency in the signal is lower than the Nyquist frequency implied by the sampling, this can lead to more appropriate choice than suggested purely from the sampling.
neural network (as done in Section 7 this case, we
7 EXPERIMENTS
In this section, we first perform experiments to demonstrate how the optimal value of w influences the generalization error of a sinusoidal network, following the discussion in Section After that, we demonstrate that sinusoidal networks with properly tuned w values outperform traditional physics- informed neural networks in classic PDE tasks.
7.1 EVALUATING GENERALIZATION
We now evaluate the simple sinusoidal network generalization capabilities. To do this, all exper- iments in this section we segment the input signal into training and test sets using a checkerboard pattern along all axis-aligned directions, points alternate between belonging to train and test set. We perform audio, image and video fitting experiments. When performing these experiments, we search for the best performing w value for generalization (defined as performance on the held-out points). We report the best values on Table 2 best performing w values follow the heuristic discussed above, and are fact half the best-performing expectation. This is
We observe that, as expected from the discussion in Section 6 the
value found in the previous fitting experiments from Section 3.1 confirming also demonstrated in the plot in Figure 4 Using a higher w leads to overfitting and poor generalization outside the training points. This is demonstrated in Figure 4 in which we see that choosing ap- propriate w value from the heuristics described previously leads to a good fit and interpolation. Setting too high leads to interpolation artifacts, due to overfitting of spurious high-frequency components.
For the video signals, which have different size along each axis, we employ a multi-dimensional w. We scale each dimension of w proportional to the size of the input signal along the corresponding axis.
7

---

Published as a conference paper at ICLR 2023
Table 2: Generalization results and the respective tuned w value. Generalization values are mean squared error (MSE). We can observe the best performing for generalization is half the ui used previ- ously for fitting the full signal due to the fact that this task used half the sample points from previously.
Experiment
SIREN
SSN
W
Fitting w
Image
2.76 10-4
1.25 10 10-4
16
32
Audio (Bach)
4.55 10-6
8,000
15,000
Audio (counting)
1.37 10-4
16,000
32,000
Video (cat)
3.40 10-3
10-3
4
8
8
8
Video (bikes)
2.74 10-3
8.79-10-
4
4
8]
8
Goneraliaation MSE for various or vallees
5
Ground truth
3
w 128
Figure 4: Left: Final training loss for different values of in the image fitting generalization experiment. Right: Examples of generalization from half the points using sinusoidal networks with different values of w. Even though both networks achieve equivalent training loss, the rightmost one, with w higher than what would be suggested from the Nyquist frequency the input signal, overfits the data, causing high-frequency noise artifacts in the reconstruction (e.g., notice the sky).
7.2 SOLVING DIFFERENTIAL EQUATIONS
Finally, we apply our analysis physics-informed learning. We compare the performance of simple sinusoidal networks to the tanh networks that are commonly used for these tasks. Results are summa- rized in Table Details for the Schrödinger and Helmholtz experiments are presented in Appendix
7.2.1 BURGERS EQUATION (IDENTIFICATION)
This experiment reproduces the Burgers equation identification experiment from Raissi al. (2019a). Here we are identifying the parameters A₃ and X2 of a ID Burgers equation, u, + Anut, A2 given known solution field. The ground truth value of the parameters are A = 1.0 and A2 0.01
In order to find good value for w, we perform low-pass pass reconstruction of the solution as before. We observe in Figure 5 that the solution does not have high bandwidth, with most of the loss being minimized with only the lower half of the spectrum. Note that the sampling performed for the training data (N 2, 000) sufficient to support such frequencies. This suggests 10. Indeed, we observe that w = 10 gives the best identification of the desired parameters, with errors of 0.0071% and 0.0507% for A and λ2 respectively, against errors of 0. 0.0521% and 0.4522% of the baseline. This value of w also achieves the lowest reconstruction loss against the known solution, with an MSE of 8.034 10-4, Figure shows the reconstructed solution using the identified parameters.
value in the range
7.2.2 NAVIER-STOKES (IDENTIFICATION)
This experiment reproduces the Navier-Stokes identification experiment from Raissi al. (2019a). In this experiment, we are trying to identify, the parameters λ₁, Ag and the pressure field p of the 2D Navier-Stoker equations given by du X1u Vu Vp X2 V²u, given known velocity fields and v. The ground truth value of the parameters are \ = 1.0 and X = 0.01.
Unlike the ID Burgers case, in this case the amount of points sampled for the training set (N 5, 000) not high, compared to the size of the full solution volume, and is thus the limiting factor for the bandwidth of the input signal. Given the random sampling of points from the full solution, the
8

---

Published as a conference paper at ICLR 2023
Table Comparison of the sinusoidal network and MLP with tanh non-linearity on PINN experiments from (Raissi et al. 2019a; Sitzmann et al. (020). Values are percent error relative to ground truth value for each parameter for identification problems and mean squared error (MSE) for inference problems. The Helmholtz experiment is the same from Section 3.1
Experiment
Baseline
SSN
w
Burgers (Identification)
[0.0521%, 0.4522%
0.0071 1, 0.0507%)
10
Navier-Stokes (Identification)
[0.0046%, 2.093%
00 10.0038%) 182%%
[0.6
0.3
1.2]
Schrödinger (Inference)
1.04 10
4.30 10 10-4
4
Helmholtz (Inference)
5.97 10
5.94 10
16
frequencies
0.20
-05
-14
Dva COCE JORMA
0.34
Co 9:25
9.50
To 6.75
0.30
Cutorf
Figure 5: Left: Reconstruuttin loss for different cutoff frequencies for low-pass filter applicd the solution the Burgers equation. Right: Reconstructed solution of the Burgers equation using the iden- tified parameters with the sinusoidal network, together with the position of the sampled training points.
Vortictt
Predicted pressure
Exact pressure
0
0
0.2
0.0
-2
Figure 6: Left: One timestep of the ground truth Navier-Stokes Stokes solution. The black rectangle indicates the domain region used for the task. Right: Identified pressure field for the Navier-Store Stokkr equations using the sinusoidal network. Notice that the identification is only possible up to a constant.
generalized sampling theorem applies. The original solution has dimensions of 100 x 200. With the 5. 000 randomly sampled points, the average sampling rate per dimension is approximately 17, on average, corresponding to a Nyquist frequency of approximately 8.5. Furthermore, given the multi-dimensional nature of this problem, with both spatial and temporal axes, employ independent scaling to w for each dimension. The analysis above suggests an average w ss 1, with the dimensions of the problem suggesting scaling factors of [0.5
2) Indeed, we observe that = 10.3 0.6 1.2)" gives the best results. With with errors of 0.0038% and 1.782% for X₃ and A2 respectively, against errors of 0.0046% and 2. 2.093% of the baseline. Figure the identified pres- sure field. Note that given the nature of the problem, this field can only be identified up to a constant.
8 CONCLUSION
In this work, we have present a simplified formulation for sinusoidal networks. Analysis of this architecture from the neural tangent kernel perspective, combined with empirical results, reveals that the kernel for sinusoidal networks corresponds to a low-pass filter with adjustable bandwidth. We leverage this information in order to initialize these networks appropriately, choosing their bandwidth such that is tuned to the signal being learned. Employing this strategy, we demonstrated improved results in both implicit modelling and physics-informed learning tasks.

---

Published as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- Grained Analy- sis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks. arXiv: PeXXv:/901.08580 [cs, stat], May 2019. URL http:////aww...rr////////0000..0 arXiv: 1901.08584.
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies. June 2019. URL https:
orq/abs/ 1906./9966..00 60425v3
Xiang Huang, Hongsheng Liu, Beiji Shi. Zidong Wang, Kang Yang, Yang Li, Bingya Weng, Min Wang, Haotian Chu, Jing Zhou, Fan Yu, Bei Hua, Lei Chen, and Bin Dong. Solv- ing Partial Differential Equations with Point Source Based on Physics-Informed Neural Net- works. arXiv:2111.01388 [physies], November 2021a. URL http::/arxiv.org/ arXiv: 2111.01394.
2111.01394
Xinquan Huang, Tariq Alkhalifah, and Chao Song. A modified physics-informed neural network with positional encoding. pp. 2484, September 2021b. doi: 10. 10.1 1011990eamm22221355551222
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kemel:: Convergence and Generalization in Neural Networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. /tLL/tttps///prceeeeeiie
wwwriss................... cc/
.tmml
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. 2020.
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. 2014. URL
http://arxi:> org/
wwwgg/bb///ee abs 1412.69888
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and
Jascha Sohl-Dickstein. Deep Neural Networks as Gaussian Processes, March 2018. utth
00165 arXiv:1711.00165 [cs, stat].
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. Journal of Statistical Mechanics: Theory and Experiment. 12440,, December 2020. ISSN 1742-5468 doi: 10. 10.1088/1742-55668abaaat 902 06720 arXiv: 1902.06720.
URL
Radford M. Neal. Priors infinite networks. Technical Report CRG-TR-94-1, University of Toronto, cs www.:sstttoooooooooooooooooooo
1994. URL http:///www....o..oo...oo/////ar
Tim Pearce, Russell Tsuchida, Mohamed Zaki, Alexandra Brintrup, and Andy Neely. Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions, June 2019. URL http:: wwwp:xivvorggaaoooooooo abs 06076
190.066700067 [cs, stat].
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 2301-53104 PMLR, May 2019. URL ISSN: 2640-3498.
https: http:://poccceeeneeoe mlr press /97 wtt////aaaaaa.. rahaman 9a html
Ali Rahimi and Benjamin Recht. Random Features for Large-Scale Kernel Ma- chines. In Advances in Neural Information Processing Systems, volume 20. Cur- ran Associates, Inc., 2007. URL https::/papers.nii http://sw....aaccccpppp///////// cc//saper/2007/aaahh html
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics-informed neural net- works: A deep learning framework for solving forward and inverse problems involving non- linear partial differential equations. Journal of Computational Physics, 378, 2019a. doi:
10.1016/j.cp.2208.0000 URL https: /tEEs:://iiiigghub... elsevier. com/ wttp://ee..comeereeeeeeeeee retrieve/ 011/50021999118888
10

---

Published as a conference paper at ICLR 2023
Maziar Raissi, Zhicheng Wang, Michael S. Triantafyllou, and George Em Karniadakis. Deep Learning of Vortex Induced Vibrations. Journal of Fluid Mechanics. 861:119-137, February 2019b. ISSN 0022-1120 1469-7645. doi: 10.1017/ffm.2018.872 08952 arXiv: 1808.08952.
1808
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit Neural Representations with Periodic Activation Functions. 2020. URL
attp::/aaxiae owg. h 20060000.006661
Chao Song, Tariq Alkhalifah, and Umair Bin Waheed. A versatile framework to solve the Helmholtz equation using physics-informed neural networks. Geophysical Journal International, 228(3): 7750-1762, November 2021. ISSN 0956-540X, https://acadeni. oup. com/ /ttp://////aaaaaa/////////////// 640122
doi: 10. g0.g093/gj/ggaab444. URL
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. 2020.
Russell Tsuchida. Results on infinitely wide multi-layer perceptrons. PhD Thesis, The University of Queensland, December 2020. URL httthttss///sppaeee library. uq. edu. adw.s//veww/00
co33959
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks: neural networks. Computer
From regression to solving multi-scale pdes with Methods in Applied Mechanics and Engineering, 384: 1 13938, 2021.
Jian Cheng Wong, Chinchun Ooi, Abhishek Gupta, and Yew-Soon Ong. Learning in Sinusoidal Spaces with Physics- Neural Networks. 100 00938 [physics], March 2022. URL arXiv: 2109.09338
/ttp:///rxiv..o wwtp:/wwwwww//b////.ssooooo 09338
version: 2.

---

Published as a conference paper at ICLR 2023
A SIMPLE SINUSOIDAL NETWORK INITIALIZATION
We present here the proofs for the initialization scheme of the simple sinusoidal network from Section
Lemma 4. Given any c. for X ~ N (0, joo) and ~ u (-c, c), we have Var [X] = Var [Y] = J.0
Proof. By definition, Var[X] o² Jec For Y, we know that the variance of uniformly distributed random variable with bound ba is given by [b a)?. Thus. Var[Y] = 12 (2c)2 = J.c.
Theorem 5. For a uniform input in [-1,1], the activations throughout a sinusoidal networks are approximately standard normal distributed before each sine non-linearity and rresine-iittrbuueed after each sine non-linearity, irrespective of the depth of the network, if the weights are distributed normally, with mean and variance 2 with n is the layer's fan-in. n
Proof. The proof follows exactly the proof for Theorem 1.8 in Sitzmann et (2020), only using Lemma 4 when necessary to show that the initialization proposed here has the same variance necessary for the proof to follow.
B EMPIRICAL EVALUATION OF SSN INITIALIZATION
Here we report empirical analysis the initialization scheme of simple sinusoidal networks, ref- erenced in Section 3 For this analysis we use a sinusoidal MLP with 6 hidden layers of 2048 units, and single-dimensional input and output. This MLP is initialized using the simplified scheme described above. For testing, 2 equally spaced inputs from the range [-1, 1] are passed through the network. We then plot the histogram of activations after each linear operation (before the sine non-linearity) and after each sine non-linearity. To match the original plot, we also plot the ID Fast Fourier Transform of all activations in layer, and the gradient of this output with respect each activation. These results are presented in Figure g The main conclusion from this figure is that the distribution of activations matches the predicted normal (before the non-linearity) and aresine (after the non-linearity) distributions, and that this behavior is stable across many layers. We also reproduced the same result up to 50 layers.
We then perform an additional experiment in which the exact same setup above is employed, yet the ID imputs are shifted by large value (i.e., x + x 1000). We the show the same plot as before in Figure 9 We can see that there is essentially no change from the previous plot, which demonstrates the sinusoidal networks shift-invariance the input space, one of its important desirable properties, as discussed previously.
C EXPERIMENTAL DETAILS FOR COMPARISON TO SIREN
Below, we present qualitative results and describe experimental details for each experiment. As these a reproduction of the experiments in Sitzmann al. 2020), refer their details as well for further information.
C.1 IMAGE
In the image fitting experiment, we treat image function from the spatial domain to color values (x, 3) + (r, b). In the case monochromatic image, used here, this function maps instead one-dimensional intensity values. We try to learn function f IR2 IR parametrized as a sinusoidal network, in order to fit such an image.
Figure 7 shows the image used in this experiment, and the reconstruction from the fitted sinusoidal network. The gradient and Laplacian for the learned function are also presented, demonstrating that higher order derivatives also learned appropriately.
12

---

Published as a conference paper at ICLR 2023
Table 4: Comparison of the simple sinusoidal network and SIREN on some experiments, with longer training duration. The specific durations are described below in the details for each experiment. We can see that the simple sinusoidal network has stronger asymptotic performance. Values above the horizontal center line are peak signal to noise ratio (PSNR), values below are mean squared error (MSE). Audio experiments utilized a different learning rate for the first layer, see the full description below for details.
Experiment
Simple Sinusoidal Network
SIREN [ours]
Image
54.70
52.43
Poisson (Gradient)
39.51
38.70
Poisson (Laplacian)
22.09
20.82
Video (cat)
34.64
32.26
Video (bikes)
37.71
34.07
Audio (Bach)*
5.66
10
3.02 10-6
Audio (counting)
4.02 10 5
6.33
10
Image
Gradient
Laplacian
Figure 7: Top Ground truth image. Bottom: Reconstructed with sinusoidal network.
13

---

Published as a conference paper at ICLR 2023
Activation Distribution Activation Spectrum
Gradient Distribution
Gradient Spectrum
5e-1
Imput
660
oc+
1e+0
0e+0
Ie+0
Oc+0
2c+1
4c+1
3c+
5e
1
0e+0
Sc
0e-0
2c+1
4e+1
100 lnsaarity
1t layer
bedtre
2e
2045
20-1
0e+0
0e+0
Ue+0
-le+2
0610
16r2
2011
40+1
-5e4
20+1
Se+0
20+3
5e-1
ron after
20+3
0e+0
1e+0
0e+0
le+0
0c+0
2e+1
4e+1
det
1e:3
0c+0
le-3 0e-0
10e+0
2e+1
4e+1
Se+3
30+3
2nd layer
beffere
te+0
Ue+0
-2e+0
de+0 2e+0
0c+0
2e+1
4c+1
-5844
De
Sei4
0e-0
2e+1
4e+1
Don Lnvarityy
20+3
l0+0
ofter
20+0
2e+3
0c+0
1ee0
1e+0
be+0
46+1
10c+0
2e+1
Tefoce Iun.uuttt
le+0
e+
3rd layer
1e+3
Oc+0
-2e+0
0e+0 2e+0
Oc+0
2e+1
4e+1
3c+0
0c+0
Se-4
0e-0
2et1
4e+2
20+3
1e+3
le+0
atfir
2c+0
0e+
0c+0
be+0
-5e-4
5e-4
De+0
20+1
son lnmeaatt
before
2e+3
c++
4th layer
0c+0
-e+0
0e+0
2e+0
Oe+0
2e+1
de+1
0e+0
Se-4
0e+0
5e-d
0e-0
2e+1
4e+1
2e+3
ofher lizuarriy
2c+0
le+0
0a+0
2o+00
De ++00
0c+0
1c+0
0c+0
20+1
4c+1
5e 4
06+0
50 4
0e-0
2e+1
4e+1
non lnvauity
5th layer
before
2e-1
0e+0
Be+0
Www
0e+0
Dc+0
2e+3
ofterr
2e+0
Se-1
san
Oc+0
1e+0
de+0
1e+0
de+0
2e+1
4e+1
8e+0
-5e-4
0e+0
5e-4
0e-0
2e+1
4et2
100 lunarrtty
6th layer
bechure
2e1
2e+3
le+0
Re+0
0e+0
Ge+0
40+1
-5e-4
20+1
let0
ofter
0e+0
1e40
0e+0
1e+0
0e+0
2e+1
4e+1
0e+0
-Se-4
0e+0
5e-4
0e-0
2e+1
4e+1
Figure 8: Activation statistics for layers of a simplified sinusoidal network, demonstrating the observed distributions matched the theoretical expectation and preserves the properties from the SIREN initialization.
14

---

Published as a conference paper at ICLR 2023
Activation Distribution Activation Spectrum
Gradient Distribution
Gradient Spectrum
26+0
Imput
5e44
C+0
detl
Oc+0
3-
1e+3
le+3
Oc+0
Se+0
le+1
2e+
0+0
2e+0
06+0
2c+0
0e-0
Se+0
le+1
2e+1
1000 lnsarrity
1t layer
bedtre
1c.5
2e+8
0+3
2e-1
mm
0e+0
0e
16r5
Se+0
Se+2
e++
5e-1
M
ron after
0e+0
1e+0
0e+0
le-0
0e+0 Set0 let1 2et
-2-3 0e+0 2e:3
0e+0 Se+0 le+1
2e+1
mmmm
Sel
2nd layer
beffere
c+3
2e-1
De+0
-2e+0
de+0
2e+0
Oc+0 5e+0 le+1 20+1
-20-3
06+0 20-3
0e+0 Se+0
10+1
20+1
D00 Lnvarity
0+
m
ofter
2e+0
5e-1
0c+0
1ee0
be+g
20c
Deto
2e+1
Tefoce lun.uutt
5e-1
3rd layer
Oc+0
-2e0
0e+0
2e+0
OE+D
Set0
le+1
2et
c+0
-2e3
De+0
2e:3
0e-0
Se+0
le+1
2e+1
non hanwaray
c+2
le0
stfor
2c+0
0e+9
0c+0
-2033
0910
26-3
0e10
l0+1
2011
son lnmoaatt
let0
before
4th layer
0c+0
-e+0
0e+0
2e+0
Oe+0
Se+0
le+1
2et1
e+0
2ee3
0e+0
2e-3
0040
0e-0
Se+0
let1
2et1
ofeer lizuarriy
2++2
2c+0
c+2
le+0
0a+0
0c+0
1c+0
0c+0
5e+0
lc+1
20+1
2003
0c+0
20-3
Be-0
se+0
let1
20+1
non lnvurity
20+3
5th layer
before
2e-1
0e+0
200l
0et0
2e+1
B++2
ofter
2e+0
5042
e+2
Se-1
2e-1
san
0c+0 1e+0
de+0
1e-0
oeto
Set0
le+1
26+1
e+0
22-3
0e+0
2e:3
0e+0
Se+0
le+1
2e+1
1s0 lunarityy
Se+2
Mmmm
c+33
6th layer
bechure
2e-1
le+0
Los+0
Ge+0
2041
2et0
mmm
ofter
0e+0
1e+0
0e+0
1e+0
oc+0
Se+0
le+1
2e+1
-2e3
0e+0
2e:3
0e-0
Se+0
le41
2e+1
Figure Activations for 6 layers of simplified sinusoidal network which the input has been shifted by large value, i.e., x + 1000. The distribution characteristics preserved, demonstrating the sinusoidal network's 's shift-invariance.
15

---

Published as a conference paper at ICLR 2023
Training parameters. The input image used is 512x mapped an input domain [-1, The sinusoidal network used is 5-layer MLP with hidden size 256, following the proposed initialization scheme above. The parameter w is set to 32. The Adam optimizer is used with learning rate of 3 10-3, trained for 10, 000 steps in the short duration training results and for 20, 000 steps in the long duration training results.
C.2 POISSON
Image
Gradient
Laplacian
Figure 10: Top row: Ground truth image. Mid: Image reconstructed from sinusoidal network supervised on gradient. Bottom: Image reconstructed from sinusoidal network supervised on Laplacian.
These tasks are similar to the image fitting experiment, but instead of supervising directly on ground truth image, the learned fitted sinusoidal network is supervised on its derivatives, constituting a Poisson problem. We perform the experiment by supervising both on the input image's 's gradient and Laplacian, and report the reconstruction of the image and it's gradients in each case.
Figure shows the image used in this experiment, and the reconstruction from the fitted sinusoidal networks. Since reconstruction from derivatives only be correct up to scaling factor, scale the reconstructions for visualization. As in the original SIREN results, we can observe that the reconstruction from the gradient is of higher quality than the from the Laplacian.
Training parameters. The input image used is of size 256 x 256, mapped from an input domain 1]2. The sinusoidal network used a 5-layer MLP with hidden size 256, following the proposed initialization scheme above. For both experiments, the parameter w is set 32 and the Adam optimizer is used. For the gradient experiments, in short and long training results, a learning rate of 1 10 is used, trained for 10, 000 and 20, 000 steps respectively. For the Laplace experiments, in short and long training results, a learning rate of 10 used, trained for 10, 000 and 20, 000 steps respectively.
16

---

Published as a conference paper at ICLR 2023
C.3
VIDEO
Figure 11: Top row: Frames from ground truth "cat" video. Bottom: Video reconstructed from sinusoidal network.
Figure 12: Top row: Frames from ground truth "bikes" video. Bottom: Video reconstructed from sinusoidal network.
These tasks are similar to the image fitting experiment, but we instead fit video, which also has temporal input dimension, (f, y) + (r, g. b). We learn function f R R33 parametrized as sinusoidal network, in order to fit such video.
Figures II and 12 show sampled frames from the videos used in this experiment, and their respective reconstructions from the fitted sinusoidal networks.
Training parameters. The cat video contains 300 frames of size 512 x 512. The bikes video contains 250 frames of size 272 x 640. These signals are fitted from the input domain [-1, 1|3, The sinusoidal network used is a 5 5-layer MLP with hidden size 1024, following the proposed initialization scheme above. The parameter w 3 10-4 trained for 100, 000 steps in the short duration training results and for 200, 000 steps in the long duration training results.
8. The Adam optimizer is used, with learning rate of
C.4
AUDIO
Bach
Counting
Uniund trth
-0.5
85
0.0
Resonstaction
Figure 13: Ground truth and reconstructed waveforms for "Bach" and "counting" audios.
17

---

Published as a conference paper at ICLR 2023
In the audio experiments, we fit an audio signal in the temporal domain waveform t We to learn a function f R + R. parametrized as a sinusoidal network, in order to fit the audio.
Figure 13 shows the waveforms for the input audios and the reconstructed audios from the fitted sinusoidal network.
In this experiment, we utilized lower learning rate for the first layer compared to the rest of the network. This was used to compensate the very large w used (in the 15, -00-30, 000 range, compared to the 10 10-30 30 range for all other experiments). One might argue that this is re-introducing complexity, counteracting the purpose the proposed simplification. However, we would claim (1) that this is only limited to cases with extremely high w, which was present in any case except for fitting audio waves, and (2) that adjusting the learning rate for an individual layer is still an approach that is simpler and more in line with standard machine learning practice compared to multiplying all layers by scaling factor and then adjusting their initialization variance by the same amount.
Training parameters. Both audios sampling rate of 44100Hz. The Bach audio is 7s long and the counting audio is approximately 12s long. These signals are fitted from the input domain [-,, The sinusoidal network used a 5-layer MLP with hidden size 256, following the proposed initialization scheme above. For short and long training results, training is performed for 5, 000 and 50, 000 steps respectively. For the Bach experiment, the parameter w is set to 15, 000. The Adam A
optimizer is used, with general learning rate of 3 10-3. A separate learning rate of 1 10 is used for the first layer to stabilize training due to the large w value. For the counting experiment, the parameter is set to 32, 000. The Adam optimizer is used, with general learning rate of 10-3 and a first layer learning rate 10-6
C.5 HELMIOLTZZ EQUATION
Real
Imaginary
Figure 14: Top row: Ground truth real and imaginary fields. Bottom: Reconstructed with sinusoidal network.
In this experiment we solve for the unknown wavefield + R² + R2a in the Helmholtz equation
A 1-2 ()))(()))
(xx,
with known wavenumber & and source function (a Gaussian with μ 0 and o² 10-4). We solve this differential equation using a sinusoidal network supervised with the physics-informed loss fg [ + 1.) (z)) + f())]l dz, evaluated at random points sampled uniformly in the domain
Ω 11,13
18

---

Published as a conference paper at ICLR 2023
Figure 14 shows the real and imaginary components of the ground truth solution to the differential equation and the solution recovered by the fitted sinusoidal network.
Training parameters. The sinusoidal network used a 5-layer MLP with hidden size 256, follow- ing the proposed initialization scheme above. The parameter w is set to 16. The Adam optimizer is used, with a learning rate of 3 10 trained for 50, 000 steps.
C.6 SIGNED DISTANCE FUNCTION (SDF)
In these tasks we learn 3D signed distance function. We learn function f :R IR parametrized as sinusoidal network, to model a signed distance function representing a 3D scene. This function supervised indirectly from point cloud data of the scene. Figures 16 and 15 show 3D renderings of the volumes inferred from the learned SDFs.
Training parameters. The statue point cloud contains 999, 996 points. The room point cloud contains 10, 250, 68S points. These signals are fitted from the input domain [-1, 1].. The sinusoidal network used parameter w is set to 4. The Adam optimizer is used, with a learning rate of 8 10 and a batch size of 1400. All models are trained for 190, 000 steps for the statue experiment and for 410, 000 steps for the room experiment.
5-layer MLP with hidden size 256 for the statue and 1024 for the room. The
Figure 15: Rendering of the "room" 3D scene SDF learned by the sinusoidal network from a point cloud.
Figure 16: Rendering of the "statue" 3D scene SDF learned by the sinusoidal network from a point cloud.
D NEURAL TANGENT KERNEL ANALYSIS AND PROOFS
D. PRELIMINARIES
In order to perform the subsequent NTK analysis, first need to formalize definitions for simple sinusoidal networks and SIRENs. The definitions used here adhere to the common NTK analysis practices, and thus differ slightly from practical implementation.
19

---

Published as a conference paper at ICLR 2023
Definition 1. For the purposes of the following proofs, (simusoidal) fullyiiidalflllooooooooo fully-connected neural network with L hidden layers that takes as input z RR",, is defined as the function f())) R - R1, recursively given by
(x) ((no) x
(z)= W(L) 1 -sin (sl(-1) + Bl2)
VAL
where w E R. The parameters {WGn} have shape nj+1 ny and all have each element sampled independently either from (0, 1) (for simple sinusoidal networks) from U(-c, c) with some bound E R (for SIRENs). The {b6)), are 11+11 nists-dimensional vectors sampled independently from (0, Ing 1)-
With this definition, we now state the general formulation of the NTK, which applies in general to fully-connected networks with Lipschitz non non-lincarities, and consequently in particular to the sinusoidal networks studied here as well. Let us first define the NNGP, which has covariance recursively defined by
(x, 3) = 10 D~ ([()))) + +
with base case ((1) (r, 3) = LTT + B2, and where 3 gives the variance of the bias terms in the neural network layers Neal 1994 Lee et al. 2018). Now the NTK is given by the following theorem.
Theorem 6. For neural network with L hidden layers f(l)) R
B following Definition as the size of the hidden layers ai...., nn + oc sequentially, the neural tangent kernel (NTK) of f(l) converges in probability to the deterministic kernel (LL) defined recursively
( (x, y (2, (6 (x²
(z,- .) + y (x,
where { (E(0) / are the neural network Gaussian processes (NNGPs) corresponding to each f(l) and
~0 (x, .) = (u.,)) [cos( (cos(()cos(u))
Proof. This is a standard general NTK theorem, showing that the limiting kernel recursively in terms of the network's NNGPs and the previous layer's NTK. For brevity we omit the proof here and refer the reader to, for example, Jacott et al. (2020).
Jacco
The only difference for the base case (0), due to the fact that we have an additional w parameter in the first layer. It is simple
that the neural network with 0 hidden layers, i.e. the linear model (W) + ((0)) will lead to the same Gaussian process covariance kernel the original proof, IT z + 1, only adjusted by the additional variance factor 2
Li
Theorem 6 demonstrates that the NTK can be constructed as a recursive function of the NTK of previous layers and the network's NNGPs. In the following sections will derive the NNGPs for the SIREN and the simple sinusoidal network directly. We will then use these NNGPs with Theorem 6 to derive their NTKs as well.
To finalize this preliminary section, we also provide two propositions that will be useful in following proofs in this section.
Proposition 7. For any w E R. x gd
Ew-A(001a) (cw(u=s) = -
20

---

Published as a conference paper at ICLR 2023
Proof. Omitting (O (0, Ia) from the expectation for brevity, we have
jin w = E Σj= "yys
E
By independence of the components of w and the definition of expectation,
E
e m Ej=
suyay
e wu u 7 dwy.
2m
Completing the square, we get
00
e swwmsse
tm dwy
1
e (( 22i 2in, mi dwy
V 2m
V
-00
==1
j=1
000
e -- 2im mymyy xynn dwy
V
==1
2
(w dwy-
2m
000
Since the integral and its preceding factor constitute Gaussian pdf, they integrate to 1, leaving the final result
d II
05
- Ej of ce- 10,
c
==1
Proposition 8. For any c, G E IR, Rdd
d
Ew-i((-coo) (e(u(u=2) = II sinc(cwaz).
Proof. Omitting ~ Ual-all c) from the expectation for brevity, we have
E [el(("" x) = E t Σj= wyæ;
By independence of the components of w and the definition of expectation,
d
d
E
é dw Djan Wyas
II
dw mya
II
jw my dwy
é
ju s dwy
2c
2c
1=1
J=1
1=1
Now, focusing on the integral above, we have
e jwwnay = [ wy + [
sin(w sin( wyj,
sin(w Wyxz)) way
cos(so wyzry)
wKy
2sin(ewr;))
wx,
Finally, plugging this back into the product above, we get
I dw
2sin 2sin(cway)
II
II
II
sinc sino(cwz;).
2c
2c
Wxy
j=1
21

---

Published as a conference paper at ICLR 2023
D.2 SHALLOW SINUSOIDAL NETWORKS
For the next few proofs, we will be focusing on neural networks with single hidden layer, i.e. L = 1. Expanding the definition above, such network is given by
(z) w())
1 sin w (w + x+0)))++)))))) b(D)
(4)
V
The advantage of analysing such shallow networks is that their NNGPs and NTKs have formulations that are intuitively interpretable, providing insight into their characteristics. We later extend these derivations to networks of arbitrary depth.
D.2.1 SIREN
First, let derive the NNGP for a SIREN with a single hidden layer.
Theorem 9. Shallow SIREN NNGP. For single hidden layer SIREN f()) : R + R following Definition as the size of the hidden layer no - oc, f()) tends (by law of large numbers) to the neural network Gaussian Process (NNGP) with covariance
c² (x,
no
no
II
sinc(ew (zg -5,) - 2w
II j=1
sinelew (z, + x,)) +1.
= 6
Proof. We first show that despite the usage of a uniform distribution for the weights, this initialization scheme still leads to an NNGP. In this initial part, we follow an approach similar to Lee et al. (2018). with the modifications necessary for this conclusion to hold.
From our neural network definition, each element (x)) in the output vector is a weighted combination of elements in W'1) and ((1), Conditioning on the outputs from the first layer (L = 0), since the sine function is bounded and each of the parameters is uniformly distributed with finite variance and zero mean, the (x), become normally distributed with mean zero as nı + oo by the (Lyapunov) central limit theorem (CLT). Since any subset of elements in f f()) (x:) is jointly Gaussian, we have that this outer layer is described by Gaussian process.
Now that we have concluded that this initialization scheme still entails an NNGP, we have that its covariance is determined by of ( + of = +1 + 1, where
lim +00
sin (1) (x) sin (0)) (3)
11
1 n;
sin (x) sin sin (110
lim
#1-100
n1
1 sin
w (w + b,0)) sin w (w:30 22 66666 2 8/9)
lim
T2
Now by the law of large number (LLN) the limit above converges to [sin(w (w x + b) sin (w (w ā + +b)]]
where w E 100 and b E R. Omitting the distributions from the expectation for brevity and expanding the exponential definition of sine, we have
e Su (u(uFa+b)) - in( (uu())t)) 1 (w (cu(("=+t) (wb))+ - (u(u")+)))]
E
2i
2i
E ((u(u22+t) 2+b) (()
ciu(w" (+)) -i(( (u(w=2+)) - - -iw("> 2 2+))+ (w + e cu(wTo+) o+b) -iv( x-)))
- I
[B (w==))=(++)
((+2))) E [eub E (x=x))) E { ((=(w=(E_x)) + E (-==3) E [e -iub]
22

---

Published as a conference paper at ICLR 2023
Applying Propositions 7 to each expectation above and noting that the sinc function is even, we are left with
1
Yla
sinc(ew (xy + +2)) 2e 2
nlg
sinclew (zy 3;)))
4
j=1
mp
sinelew (x, -bj)) 6- 2n²
II
sino(ew (1z + +B))
For simplicity, if we take the case of one-dimensional output (e.g., an audio signal or monochro- matic image) with the standard SIREN setting of c = /6, the NNGP reduces to
(2) sinc (VBw (x x) - sinc (V66 (I +1.
2n²
We can already notice that this kernel is composed of sine functions. The sinc function is the ideal low-pass filter. For any value of > completely dominate the expression, due to the exponential -2 factor. In practice, w is commonly set to values at least one order of magnitude above 1, if not multiple orders of magnitude above that in certain cases (e.g., high frequency audio signals). This leaves us with simply
see the the first term in the expression above will
(x, sinc (Vow -))
Notice that not only does our kernel reduce to the sinc function, but also reduces to a function solely of A x Z. This agrees with the shift-invariant property we observe in SIRENs, since the NNGP is dependent only on Az, but not on the particular values of x and т. Notice also that w defines the bandwidth of the sinc function, thus determining the maximum frequencies it allows to pass.
The general sine form and the shift- infttiinvariance of this kernel can be visualized in Figure 17 along
with the effect of varying w on the bandwidth of the NNGP kernel.
SIREN (X) X), w
SIREN [X, X), w 10
SIREN [ x), w 30
1.8
1.6
1.4
1.2
1.2
1.2
1.0
1.0
100
-1
0.8
0.8
0.8
2), 2-0
SIREN golte,
2100.01
1.0---7------00 006 620 1,00
-280-677-115--00
025 052 025
7500--7---00--220 110 025 850 07
Figure 17: The NNGP for SIREN at different values. The top row shows the kernel values for pairs (x, 7) E [-1, 1ja. Bottom row shows slice at fixed ã
We can see that the NTK of the shallow SIREN, derived below, maintains the same relevant charac- teristies the NNGP. We first derive E in the Lemma below.
23

---

Published as a conference paper at ICLR 2023
Lemma 10. For R, (x, uo
x R + R is given by
c²
nh
0 (x, =
sinc(cw (zy- - R2]) + e 2n
sinelew (z, + k,))
+1.
6
Proof. The proof follows the same pattern Theorem 9 with the only difference being few sign changes after the exponential expansion of the trigonometric functions, due to the different identities for sine and cosine.
Now we can derive the NTK for the shallow SIREN.
Corollary 11. Shallow SIREN NTK. For single hidden layer SIREN f() R - R2 following
Definition 7 its neural tangent kernel (NTK), as defined in Theorem 6 is given by
(x, = (w² (x" 1))
sinc(ew (xz - -;,) 2
II
sinclew (xz + x,)) +1
==1
c²
+
sinc(cw (xz -;,)) + e 2
sinc(ew (z) + 2z))
+1
6
2
no
w
+ 1)
+1)
II
siw (I, - 2,))
6
j=1
(w² (x"
T 1 1) 1) e- 2
II sinc(ew (Ig + +z))) + w² (x"x+ +11+1 +1.
6
Proof. Follows trivially by applying Theorem 9 and Lemma 10 to Theorem
Though the expressions become more complex due to the formulation of the NTK, we can see that many of the same properties from the NNGP still apply. Again, for reasonable values of w, the term
with the exponential factor ~2w will be of negligible relative magnitude. With v6, this leaves us with
(1-0+11) (x= +1)
II sinc (vow (1, (xx - -;)) + w (xfa+1) +1
which of the same form as the NNGP, with some additional linear terms x T J. Though these linear terms break the pure ihift-invariance, we still have strong diagonal and the sinc form with bandwidth determined by w., as can be seen in Figure 18
Similarly to the NNGP, the SIREN NTK suggests that training shallow SIREN is approximately equivalent to performing kernel regression with a sinc kernel, a low-pass filter, with its bandwidth defined by w. This agrees intuitively with the experimental observations from the paper that in order to fit higher frequencies signals, a larger w is required.
D.2.2 SIMPLE SINUSOIDAL NETWORK
Just as we did in the last section, we will now first derive the NNGP for a simple sinusoidal network, and then use that in order to obtain its NTK well. As we will see, the Gaussian initialization employed in the SSN has the benefit of rendering the derivations cleaner, while retaining the relevant properties from the SIREN initialization. We observe that similar derivation of this NNGP (using cosine functions instead sine) can be found in Pearce et al. (2019), with a focus on a Bayesian perspective for the result.
24

---

Published as a conference paper at ICLR 2023
SIREN (x)) x),
SIREN (x 8), 10
SIREN (x) x),
30
1500
1000
500
500
0
SIRFN dy- 9), 9 4-
S1DP% ex R), Fm ma
SIREN 8 R.. 2 0. w 50
6:30
-000
-203
111-0-75555 25 825
85
Figure 18: The NTK for SIREN different values. The top row shows the kernel values for pairs (z. B) E []]. Bottom row shows slice at fixed ã = 0.
Theorem 12. Shallow SSN NNGP. For a single hidden layer simple sinusoidal network f
Ih Th following Desinition ! as the size the hidden layer n1 xc, f(] tends (by law of large numbers) to the neural network Gaussian Process (NNGP) with covariance
(x,
2 -0
(c-110
1
c f +
Proof. We again initially follow an approach similar to the one described in Lee al. (2018).
From sinusoidal network definition, each element (()) in the output vector is weighted combination of elements in (1)) and ((1), Conditioning the outputs from the first layer (I. 0), since the sine function is bounded and each the parameters is Gaussian with finite variance and zero mean, the (x)) are also normally distributed with mean zero by the CLT. Since any subset of elements in (x) is jointly Gaussian, we have that this outer layer is described by a Gaussian process.
Therefore, its covariance is determined by D() + of =11) + where
(x, lim mi 700
sin ((0) (xx) , (11)) (2))
n1
lim
sin ((0) (x)) sin ((0
a: -000
n1
n1
w ( + b/o)) sin (w +6,000) (100))
lim
sin
aL-100
n1
Now by the LLN the limit above converges to
Eww- [sin(w (w- x + sin (w b))]
25

---

Published as a conference paper at ICLR 2023
where w E R and b E R. Omitting the distributions from the expectation for brevity and expanding the exponential definition sine, we have
e quu("" - ((((att)))) 2i (cu((u" - -iu(w 840))
2i
xtb) cwwlw" 2+b) tw ( (+b)-iw(u (ww"=== (======= 2+6) wwm( + =bbb) + e -ww - 2-b)
E
E (o02)))) E [e2imb] E [c - BB [c +E [ B [e (e-2úu6]
Applying Proposition
7 to each expectation above, it becomes
2
|
2
1| e
1lataB++ fllttt1B + 20²)
c
1
5 | =|1
-
- 11 -
We an once again observe that, for practical values of w, the NNGP simplifies to
e |a |
This takes the form of a Gaussian kernel, which also a low-pass filter, with its bandwidth determined by w. We note that, similar to the /6 setting from SIRENs, in practice a scaling factor of v2 is applied to the normal activations, as described in Section 3 which cancels out the 1/2 factors from the kernels, preserving the variance magnitude.
Moreover, we can also observe again that the kernel is function solely of Az. in agreement with the shift invariance that is also observed in simple sinusoidal networks. Visualizations this NNGP are provided in Figure 19
SSN (x
(x, X), w
2.0
SSN (x, X), 10
SSN (X)
(x, X), = 30
1.8
0
0 R
2),
SSN
60 -2--22-- ore 52s ose 87 170
1131-01-22 201-00 05 152 07
-1.0-21--0 00--000--3301100
am
Figure 19: The NNGP for SSN at different values. The top row shows the kernel values for pairs (x, .) E [-1, 1j2 Bottom row shows slice at fixed ă 0.
We will now proceed to derive the NTK, which requires first obtaining Σ.
Lemma 13. For w E IR, (1,) .) R x Ro - R is given by
50) 8) 2
1 (e
+ = 1 2 +
26

---

Published as a conference paper at ICLR 2023
Proof. The proof follows the same pattern Theorem 12 with the only difference being a few sign changes after the exponential expansion of the trigonometric functions, due to the different identities for sine and cosine.
Corollary 14. Shallow SSN NTK. For a simple sinusoidal network with a single hidden layer
f0); ; 1R + = following Definition is given by
[
its neural tangent kernel (NTK), defined in Theorem 6
(x, = (w² (x" + +1)
[
(
+ - 2002 e-200) +
2
1
e +
x + -20)" 2w" +1
2
(w²
(x
1-=21
(w² (o a + - 1) - l 2 + + (x (CT++1 + +1.
Proof. Follows trivially by applying Theorem and Lemma 13 to Theorem 6
We again note the vanishing factor 2w which leaves us with
2 (w² (x E + +
+ f
+ w (1)++1)+1.
(5)
As with the SIREN before, this NTK is still of the same form as its corresponding NNGP. While again we have additional linear terms x² 5 in the NTK compared to the NNGP, in this case well the kernel preserves its strong diagonal. It is still close to a Gaussian kernel, with its bandwidth determined directly by w. We demonstrate this in Figure 20 where the NTK for different values of is shown. Additionally, we also plot pure Gaussian kernel with variance w² scaled to match the maximum and minimum values of the NTK. We can observe the NTK kernel closely matches the Gaussian. Moreover, we also observe that, at F 0 the maximum value is predicted by Rd w²/ /2, as expected from the scaling factors in the kernel in Equation 5
This NTK suggests that training simple sinusoidal network is approximately equivalent to perform- ing kernel regression with Gaussian kernel, a low-pass filter, with its bandwidth defined by w.
We note that even though this sinusoidal network kernel approximates a Gaussian kernel, an actual Gaussian kernel can be recovered if combination of sine and cosine activations are employed, as demonstrated in Tsuchida (2020) (Proposition 18).
D.3 DEEP SINUSOIDAL NETWORKS
We will now look at the full NNGP and NTK for sinusoidal networks of arbitrary depth. As will see, due to the recursive nature of these kernels, for networks deeper than the ones analyzed in the previous section, their full unrolled expressions quickly become intractable intuitively, especially for the NTK. Nevertheless, these kernels can still provide some insight, into the behavior of their corresponding networks. Moreover, despite their symbolic complexity, we will also demonstrate empirically that the resulting kernels can be approximated by simple Gaussian kernels, even for deep networks.
D.3.1 SIMPLE SINUSOIDAL NETWORK
As demonstrated in the previous section, simple sinusoidal networks produce simpler NNGP and NTK kernels due to their Gaussian initialization. We thus begin this section by now analyzing SSNs first, starting with their general NNGP.
Theorem 15. SSN NNGP. For a simple sinusoidal network with L hidden layers f(.) : R - R following Definition 1 as the size of the hidden layers mi Bt. co sequentially, (LL
27

---

Published as a conference paper at ICLR 2023
SSN (x, X),
SSN (xx, x), w
100
SSN e (x X), 30
12
10
400
(00 80,
SSN (X X), 8= 0, 6
SSN 6 RI, 2 = w
300
100
1.00 0.75 232 5.25
9.05
5.25 0.50 5.75 2.50
1.00-0.2
Figure 20: The NTK for SSN different values. The top row shows the kernel values for pairs (z, Z) E [-11, Bottom row shows a slice at fixed T 0, together with a Gaussian kernel scaled to match the maximum and minimum values of the NTK.
tends (by law of large numbers) to the neural network Gaussian Process (NNGP) with covariance ())) (z, Z). recursively defined as
(r, w² (x" +1)
(GIL4-)( - (w..0) +1.
(x, 2
Proof. We will proceed by induction on the depth L, demonstrating the NNGP for successive layers as M1 n1...... ,.....,, Rg - Do sequentially. To demonstrate the base case 1, let us rearrange (1) from
Theorem 12
in order to express it in terms of inner products,
12--
4 + 2w²
2
o x+22
2+3 x=22222+
2 +1
2
H *+1 =+1)] ( (x - -1: + 1.
e
Given the definition of Σ this is equivalent to
(x: (1. (100)) - -x100)
which concludes this case.
Now given the inductive hypothesis. nı....., 7L-1 we have that the first L 1 layers define network f(L-1) with NNGP given by (x,-1)(xx 8). Now it is left to show that as nz - we the NNGP given by (44) Following the same argument in Theorem 12 the network
(z) WLL) sin (f(L-1) + ((L)
Vnz
constitutes a Gaussian process given the outputs of the previous layer, due to the distributions of (4 and ((L), Its covariance is given by x(4) + of = (L) + 1, where
lim
sin ((--1) (4)) sin ((4-))
nt. +00
n17
nr
sin (f(t- sin (f(t-11)
lim
at. -000
n2 j=1
28

---

Published as a conference paper at ICLR 2023
By inductive hypothesis, f( f(1-1) is Gaussian process Σ (x, Ø). Thus by the LLN the limit above equals
(x,
E (U.))) (sin(u)sin(t))
Omitting the distribution from the expectation for brevity and expanding the exponential definition of sine, we have
(etH
2i (e (e---)t)) =- --4 (((utt) E [e - E l i(( + ((attt
]]
2i
Since and are jointly Gaussian, p = u + and m a are also Gaussian, with mean 0 and variance
p = a + o + Cov(u, o] (((- (z, x)) + L - - (x, 3),
o o + a Covlu, - (z, x) Σ
2 (z.
We rewriting the expectations in terms of normalized variables
Bsn.((0.1) [e [e Ez~A((0.1) [e Ee-.V(0,1) [e (e=im=]] + EsmN((.1)) [e-ing]]]
Applying Proposition to each expectation, we get
1
o
2
1
((,,)) - -10) (1516-1(10.))40 ((22)) (e (12le-1(ee))) (e,x) --
2
2
Unrolling the definition beyond L = leads to expressions that are difficult to parse. However, without unrolling, we can rearrange the terms in the NNGP above as
1 - (1..)))544-122.2 (x,))) (x,E) e x01- +
(x,
2 e ( (2,2) (8.0) - - (2,0) (x,,)) (2.))+ +
Since the covariance matrix E).-1) is positive semi-definite, we can observe that the exponent expressions can be reformulated into quadratic forms analogous to the ones in Theorem 12 We can thus observe that the same structure is essentially preserved through the composition of layers, except for the w factor present in the first layer. Moreover, given this recursive definition, since the NNGP at any given depth L is a function only of the preceding kernels, the resulting kernel will also be shift-invariant.
Let us now derive the E kernel, required for the NTK.
Lemma 16. For E I,, (x, Ro x Ro + R. is given by
1 (x,)) = -
(2000 ((.0)) + 1.
Proof. The proof follows the same pattern as Theorem 15 with the only difference being a few sign changes after the exponential expansion of the trigonometric functions, due the different identities for sine and cosine.
As done in the previous section, it would be simple to now derive the full NTK for a simple sinusoidal network of arbitrary depth by applying Theorem 6 with the NNGP kernels from above. However,
29

---

Published as a conference paper at ICLR 2023
there is not much to be gained by writing the convoluted NTK expression explicitly, beyond what have already gleaned from the NNGP above.
Nevertheless, some insight can be gained from the recursive expression of the NTK itself, as defined in Theorem 6 First, note that, as before, for practical values of w, ② ~ Σ, both converging to simply a single Gaussian kernel. Thus, our NTK recursion becomes
(1)) ()) (0(1-) (z, E (xr,
Now, note that when expanded, the form of this NTK recursion is essentially as a product of the Gaussian 2 kernels,
@(4) (I, k) ~
(2, (2, =1) ( + ...) 114-1) ) (z, + 1) (0 (2,
=
) (x, E (x, T).
(6)
We know that the product of two Gaussian kernels is Gaussian and thus the general form of the kernel should approximately sum Gaussian kernels. As long as the magnitude of one of the terms dominates the sum, the overall resulting kernel will be approximately Gaussian. Empirically, we observe this to be the case, with the inner term containing w² dominating the sum, for reasonable values (e.g., w 1 and L < 10). In Figure 21 we show the NTK for networks of varying depth and w, together with a pure Gaussian kernel of variance w², scaled to match the maximum and minimum values of the NTK. We can observe that the NTKs are still approximately Gaussian, with
their maximum value approximated by x wtw, as expected from the product of w2 and L kernels above. We also observe that the width of the kernels mainly defined by w.
SSN gi 2), A wmd
SSN 8XR|X, R), 8= 0. w- 10
SSN (x R1, 8= 0, w 30
M
50-
||
60
1.2
1.
14
00-1.35-4 25-
nJs 8 1.00
-1 -0:00- 50-335 0.00 n25 aio 575 1:00
-1 00-6-15-6.0--222
0.00
035 0.00 0.5 1.50
(x, 8) u
(x, 8),
30
20.0
4.5
3.2
3.5
7.5
5.0
-1.00-0 35-2 25-2-22--7 6at a2s 656 6.75 1.00
10000:75555 00-05 3.00 025 0.90 0.s 1.00
0330
025 030
S5N (' R), 2- 0. w 4
SSN ( R1, 8 D. w 10
S5N
ex 8). 0,
-.w.333 30
3150
3175
3.5
"1
3125
3.4
8
2.5
1.00 0.15 0.50 0.5 0.00 0.25 030 0.15 3.00
000---1----------- 0:00
1.00
-1 00-00 10-000--00---11 0.00
025 050 632 1.03
4
10
W = 30
Figure 21: The NTK for SSN different w and network depth (L) values. Kernel values at slice for fixed ã = 0 are shown, together with a Gaussian kernel scaled to match the maximum and minimum values of the NTK.
D.3.2 SIREN
For completeness, in this section we will derive the full SIREN NNGP and NTK. As discussed previously, both the SIREN and the simple sinusoidal network have kernels that approximate low-
30

---

Published as a conference paper at ICLR 2023
pass filters. Due the SIREN initialization, its NNGP and NTK previously shown to have more complex expressions. However, we will show in this section that the sinc kernel that arises from the shallow SIREN is gradually "dampened" as the depth of the network increases, gradually approximating a Gaussian kernel.
Theorem 17. SIREN NNGP. For a SIREN with L hidden layers f(L) : + R following Definition as the size of the hidden layers 11....,. RL - oc sequentially, f(L) tends (by law of large numbers) to the neural network Gaussian Process (NNGP) with covariance E( ;), recursively defined as
2²
ng
Bo
(x,
sinc(ew (xy -5,5))) 2
Il
sinclew (zy + xz))
+
(14 1 ((2 (x.))+1(00000 (e e -1(4)))(
+
2
Proof. Intuitively, after the first hidden layer, the inputs to every subsequent hidden layer are of infinite width, due to the NNGP assumptions. Therefore, due to the CLT, the pre-activation values at every layer are Gaussian, and the NNGP is unaffected by the uniform weight initialization (compared to the Gaussian weight initialization case). The only layer for which this is not the case the first layer, since the input size is fixed and finite. This gives rise to the different (1).
Formally, this proof proceed by induction on the depth L, demonstrating the NNGP for successive layers as ....... nr + do sequentially. The base case comes straight from Theorem After the base case, the proof follows exactly the same in Theorem 15
For the same reasons as in the proof above, the Y kernels after the first layer are also equal to the ones for the simple sinusoidal network, given in Lemma 16
Given the similarity of the kernels beyond the first layer, the interpretation of this NNGP the same discussed in the previous section for the simple sinusoidal network.
Analogously to the SSN case before, the SIREN NTK expansion can also be approximated as product of Σ kernels, as in Equation 6 The product of sinc function with L - 1 subsequent Gaussians "dampens" the sinc, such that as the network depth increases the NTK approaches a Gaussian, can be seen in Figure
E EXPERIMENTAL DETAILS
E
GENERALIZATION
Where not explicitly commented, details for the generalization experiments are the same for the comparisons to SIREN, described in Appendix C
E.2 BURGERS EQUATION (IDENTIFICATION)
We follow the same training procedures in Raissi al. (2019a). The training set is created by randomly sampling 2, 000 points from the available exact solution grid (shown in Figure 5). The neural networks used are 9-layer MLPs with 20 neurons per hidden layer. The network structure is the same for both the tanh and sinusoidal networks. As in the original work, the network is trained by using L-BFGS to minimize mean square error loss composed of the sum of an MSE loss over the data points and physics-ifformed MSE loss derived from the equation
+ Aiule AzUzz = 0.
E.3 NAVIER-STOKES (IDENTIFICATION)
We follow the same training procedures as in Raissi et al. (2019)). The training set is created by randomly sampling 5, 000 points from the available exact solution grid (one timestep is shown in
31

---

Published as a conference paper at ICLR 2023
SIKEN (x. 2- 0 44
SIREN | A), 5-0, 10
SIREN "/ 50.50000
000
+00
400
201
SIREN 0f
SIREN (x X1, Km
SIREN (/ xm a
1099
1900
200
201
-1.00-1 15---000-----1- a30 025 aão 635 130
-3.06-2.75- 1.26--25-60--12
c2s 030 631
1.00
12:00-077 535-022 aão 1000
12.00
600
600
500
209
200
-1.00 2535 050
s5n 6m
-176- 275 27 min
nin
1.00
1:00
3
10
30
Figure 22: The NTK for SIREN different and network depth (L) values. Kernel values a slice for fixed i 0 are shown.
MSE for various cutoff frequencies
0.175
0.150
0.125
0.100
0.075
0.050
0.000
150
200
250
Cutoff frequency
Figure 23: Reconstruction loss through a low-pass filter for the "camera" image. Notice the loss continues to go down up to the Nyquist frequency.
6). The neural networks used are 9-layer MLPs with neurons per hidden layer. The network structure the same for both the tanh and sinusoidal networks. As in the original work, the network trained by using the Adam optimizer to minimize a mean square error loss composed of the sum of an MSE loss over the data points and physics-informed MSE loss derived from the equations
Ut + >o (wurz + ruy) = PP + >g + Uyyy)
(8)
U/ A (wvz + viy) -P + >a + Ugg).
E.4 SCHRÖDINGER (INFERENCE)
This experiment reproduces the Schrödinger equation experiment from Raissi al. (2019a). In this experiment, we are trying to find the solution to the Schrödinger equation, given by
ih, 0.5ht 2h
(10)
32

---

Published as a conference paper at ICLR 2023
MSE for various cutoff frequencies
0.05
MSE
0.03
0.02
0.01
0.00
20
40
60
80
100
Cutoff frequency
Figure 24:
Reconstruction loss for different cutoff frequencies for a low-pass filter applied to the solution of the Navier-Stokes equations.
Since in this case we have a forward problem, we do not have any prior information to base our choice of w on, besides maximum limit given by the Nyquist frequency given the sampling for our training data. We thus follow usual machine learning procedures and experiment with number of small w values, based the previous experiments.
We find that w = 4 gives the best results, with a solution MSE of 4.30 10 10-4, against an MSE of 1.04 10-8 for the baseline. Figure 25 shows the solution from the sinusoidal network, together with the position of the sampled data points used for training.
hh, x)
5
NONOX XEX MINNOODE
Not Mox XMx
X
Data (150 points)
3
2
1
-5
>00004
XX
MW -
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
t
0.59
t= 0.79
0.98
[h(t, x)
[h(t, x)
[h(tx)]
X
X
X
Exact
Prediction
Figure 25:
Solution to the Schrodinger equation with the sinusoidal network, together with the position of the sampled data points used for training.
Training details. We follow the same training procedures in Raissi et al. (2019)). The training set is created by randomly sampling 20, 000 points from the domain (x E [-5, 5J. t E [0, n 22) for evaluation for the physics-informed loss. Additionally, 50 points are sampled from each of the boundary and initial conditions for direct data supervision. The neural networks used are 5-layer MLPs with 100 neurons per hidden layer. The network structure is the same for both the tanh and sinusoidal networks. As in the original work, the network is trained first using the Adam optimizer by 50, 000 steps and then by using L-BFGS until convergence. The loss is composed the sum of MSE loss over the data points and physics-iffrrmed MSE loss derived from Equation T0
33

---

Published as a conference paper at ICLR 2023
E.5
HELMHOLTZ EQUATION
Details for the Helmholtz equation experiment are the same as in the previous Helmholtz experiment in Appendix C
34


Reviews:
[{"review_id": "VAO_PimWWu", "reviewer": "Reviewer_pDxg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. This paper proposed a simplified version of the original form of sinusoidal networks in SIREN, that allows for easier implementation and theoretical analysis.\n2. The proposed sinusoidal networks outperform SIRENs in implicit representation learning tasks.\n3. Analysis of the proposed sinusoidal networks from an NTK perspective. The NTK approximates a low-pass filter with adjustable bandwidth.\n4. Demonstrate that the performance of a sinusoidal network can be optimized by tuning the bandwidth implied in the NTK approximation.", "summary_of_the_review": "The message that this paper delivers is simple but helpful to understanding the behavior of sinusoidal networks. It also provides some guidelines of choosing better initialization parameters to achieve better models empirically. In the meantime, from the technical perspective, it is a successful attempt to use NTK as an approximation technology to analyze the proposed model in the theoretical way.", "strength_and_weaknesses": "Strength:\n- The modification over the original SIREN networks is simple, effective and easier to analyze.\n- The theoretical framework is solid and clear.\n- It is a significant discovery that the sinusoidal networks work approximately as low-pass filters with bandwidth controlled by w, which can be utilized to design the initialization parameters of sinusoidal networks.\n\nWeakness:\n\n- This paper compared the Simple Sinusoidal Network with the original implementation of SIREN using different initialization strategies. I wonder how much contribution the Kaiming normal initialization and different network models make respectively.\n- What happens if applying the proposed initialization method to the original sinusoidal networks? Will it behave whether a comparable or a worse performance?\n- One question: You claimed that the modified sinusoidal networks are easier to analyze in theoretical way, but in Corollary 12 it seems that it is possible to figure out the NTK expression for original SIREN networks as well. In what aspect the modified model could embody better potential for analysis?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M2v_LMOkWR7", "reviewer": "Reviewer_FbCm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors analyse the NNGP kernel and NTK of neural networks with sinusoidal activations. The NTK has an adjustable band-width parameter, which allows users to understand such kernel models in terms of a low-pass filter. A discussion on how the cutoff frequency is chosen is provided. The authors show that such kernels can be applied to implicit models and differential equations.", "summary_of_the_review": "This paper studies neural nets with sinusoidal activations in an NTK framework. A key bandwidth parameter and its importance on some representative problems is given. While the mathematical analysis does not require a huge amount of mathematical creativity, the paper is well organised and presents a clear story. There are some minor issues surrounding mathematical clarity and discussion of previous work.", "strength_and_weaknesses": "**Strengths:**\n- To the best of my knowledge, this is the first paper that studies in-depth the way in which infinite-width neural network theory can help inform neural networks with sinusoidal activations that are trained using gradient-based optimisers.\n\n**Mathematical correctness:**\n- Theorem 2. In particular, the phrase \"approximately standard normal distributed\" is not precisely mathematically defined anywhere in the paper before the theorem is presented. Either change remove the theorem status of this statement, or define \"approximately standard normal distributed\". More importantly: asymptotic normality will hold (due to the previously cited results of Lee et al. 2018). If all you need is Lee et al.'s result, then just state this.\n- \"We can thus observe that this kernel approximates a Gaussian kernel, which is a low-pass filter, with its bandwidth define by ω\". If you really want a squared exponential, RBF kernel, you can obtain this using half sine and half cosine activations. You can observe that summing the two kernels cancels out some terms. For example, see Proposition 18 of \"Results on Infinitely Wide Multi-layer Perceptrons\".\n\n**Minor:**\n- Theorem 13 has typo in \"shalow\". Note this kernel is very similar to the kernel in \"Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions\", which uses cosine instead of sine (but the analysis is pretty much identical). Maybe it is identical to Lemma 17. It is perhaps also worth mentioning that it is straight-forward to handle non-zero mean weights for these activations in the NNGP setting, since the integrals also have a closed-form expression. In general, this work might be worth citing as a Bayesian counterpart to the gradient-descent approach considered here (with the other difference that cosine is replaced with sine, which is not too important).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F41jlM7Oqw", "reviewer": "Reviewer_YnQa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes sinusoidal neural networks from the Neural Tanget Kernal (NTK) perspective.  This analysis leads to a number of important observations.  The most important finding, perhaps, is that their NTK approximates a tuneable low-pass filter.  This insight is subsequently used to develop guidelines for optimizing the performance of a sinusoidal network by tuning the bandwidths of its kernels according the maximum frequency present in the input signal.  The paper also suggests an initialization scheme for sinusoidal networks that leads to improved results.  The ideas developed in this work are evaluated using two tasks---1) learning implicit models and 2) solving differential equations---and the results suggest that the ideas developed in this work have merit.", "summary_of_the_review": "This paper advances our understanding of sinusoidal networks, consequently this paper will be of interest to the larger machine learning community, and in particular the ICLR community.", "strength_and_weaknesses": "This is a well-written paper.  I quite liked the narrative structure of this manuscript.  \n\nThe paper begins by constructing a simplified sinusoidal network model that mimics the key characterstics of SIREN but is much more amenable to theoretical analysis.  In addition to providing mathematical reasoning that confirms that the simplified sinusoidal model used in this work is similar to SIREN, the paper also provides empirical results that show that the simplified network achieves performance similar to that attained by SIREN.\n\nSection 4 shows that the kernals of the simplified network approximates a Gaussian kernel whose width can be tuned.  \n\nSection 5 uses a toy example to show how network behaves as this \"width\" parameter value shifts.  It is not immediately obvious to me how to parse the results presented in Figure 3.  The following sentence confuses me, \"We can see that due to the simple nature of the signal, containing only two frequencies, there are only three loss levels.\"  Why is this?  It would be useful for a reader like me to include a sentence for why there are only three loss levels for the current two-frequency problem setup.\n\nSection 6 discusses how to tune the aforementioned \"width\" parameter.  The motivation being that this value is \"crucial for the learning of the network.\"  This discussion presents a heuristic for setting this \"width\" parameter to one-eighth of the maximum frequency in the signal.  Results in Figures 3 seems to imply this heuristic.  This section of the paper is somewhat underwhelming.  Why one-eighth?  Why not, say, one-tenth?  Given that it is okay to select a slightly suboptimal value for this \"width\" parameter since the network is able to adjust it during training.  It is however clear that using too large a value may results in overfitting.\n\nSection 7 and 8 presents results and conclusions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "CLpy3BniQxa", "reviewer": "Reviewer_sbxp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a restriction of the SIREN structure (neural network with sine activation function), named SSN. SSN has a tunable frequency in the first layer only (instead of all the layers), and its weights are initialized with a Gaussian distribution, instead of a uniform distribution.\n\nThey compute and compare the Neural Tangent Kernel (NTK) of SIREN and SSN, and conclude that both perform low-pass filtering.", "summary_of_the_review": "I have some doubts about two theoretical results, and I am not able to evaluate the impact of the results.", "strength_and_weaknesses": "## Strengths\n\nThe NTK computation for SIREN and SSN is new. It can definitely be useful to understand the dynamics of SIREN and SSN neural networks.\n\n## Doubts\n\n### Dependence on Sitzmann's proofs\n\nTheorem 2 depends entirely on Sitzmann's proof (*Implicit Neural Representations with Periodic Activation Functions*, 2020). However, it appears that, by using their Theorem 1.5 (Central Limit Theorem), they assume implicitly that they are at the infinite-width limit, that is, the number of neurons per layer tends to infinity. So, this theorem is not valid with a finite number of neurons per layer. (Or I may have missed an argument.)\n\nI recall that the tails of the distributions of the pre-activations in a ReLU NN tends to become heavier and heavier after each layer (in a finite-width setup). So, the term ``approximately standard normal'' should be more explicit: what is the considered distance?\n\n### Scaling $\\omega$\n\nAt the beginning of Section 3, the authors recall that, when initializing SIRENs, the weights are sampled from $\\mathcal{U}([-c/\\omega, c/\\omega])$, excluding the weights of the first layer (why?). This choice is understandable in common NNs, and leads to different learning trajectories (while it does not change the function represented by the NN at initialization).\n\nMy questions are the following:\n1. when initializing SSNs, the weights are Gaussian. But what is their variance?\n2. it seems that, for SSNs and SIRENs, the weights are initialized without being scaled by $1/\\omega$. Why?\n3. if we scale the variance of the initialization distribution of the weights by $1/\\omega$, do we recover similar results when computing the NTKs?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster