Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a restriction of the SIREN structure (neural network with sine activation function), named SSN. SSN has a tunable frequency in the first layer only (instead of all the layers), and its weights are initialized with a Gaussian distribution, instead of a uniform distribution.

They compute and compare the Neural Tangent Kernel (NTK) of SIREN and SSN, and conclude that both perform low-pass filtering.
strength_and_weaknesses: ## Strengths

The NTK computation for SIREN and SSN is new. It can definitely be useful to understand the dynamics of SIREN and SSN neural networks.

## Doubts

### Dependence on Sitzmann's proofs

Theorem 2 depends entirely on Sitzmann's proof (*Implicit Neural Representations with Periodic Activation Functions*, 2020). However, it appears that, by using their Theorem 1.5 (Central Limit Theorem), they assume implicitly that they are at the infinite-width limit, that is, the number of neurons per layer tends to infinity. So, this theorem is not valid with a finite number of neurons per layer. (Or I may have missed an argument.)

I recall that the tails of the distributions of the pre-activations in a ReLU NN tends to become heavier and heavier after each layer (in a finite-width setup). So, the term ``approximately standard normal'' should be more explicit: what is the considered distance?

### Scaling $\omega$

At the beginning of Section 3, the authors recall that, when initializing SIRENs, the weights are sampled from $\mathcal{U}([-c/\omega, c/\omega])$, excluding the weights of the first layer (why?). This choice is understandable in common NNs, and leads to different learning trajectories (while it does not change the function represented by the NN at initialization).

My questions are the following:
1. when initializing SSNs, the weights are Gaussian. But what is their variance?
2. it seems that, for SSNs and SIRENs, the weights are initialized without being scaled by $1/\omega$. Why?
3. if we scale the variance of the initialization distribution of the weights by $1/\omega$, do we recover similar results when computing the NTKs?
clarity,_quality,_novelty_and_reproducibility: ## Clarity

The paper is very well written.

## Quality

As mentioned above, I have some doubts about Theorem 2 and the role of the scaling $\omega$.

Besides, I am not a specialist of periodic activation functions, so I cannot evaluate the impact of such theoretical results.

## Novelty

The results are definitely new.
summary_of_the_review: I have some doubts about two theoretical results, and I am not able to evaluate the impact of the results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 1. This paper proposed a simplified version of the original form of sinusoidal networks in SIREN, that allows for easier implementation and theoretical analysis.
2. The proposed sinusoidal networks outperform SIRENs in implicit representation learning tasks.
3. Analysis of the proposed sinusoidal networks from an NTK perspective. The NTK approximates a low-pass filter with adjustable bandwidth.
4. Demonstrate that the performance of a sinusoidal network can be optimized by tuning the bandwidth implied in the NTK approximation.
strength_and_weaknesses: Strength:
- The modification over the original SIREN networks is simple, effective and easier to analyze.
- The theoretical framework is solid and clear.
- It is a significant discovery that the sinusoidal networks work approximately as low-pass filters with bandwidth controlled by w, which can be utilized to design the initialization parameters of sinusoidal networks.

Weakness:

- This paper compared the Simple Sinusoidal Network with the original implementation of SIREN using different initialization strategies. I wonder how much contribution the Kaiming normal initialization and different network models make respectively.
- What happens if applying the proposed initialization method to the original sinusoidal networks? Will it behave whether a comparable or a worse performance?
- One question: You claimed that the modified sinusoidal networks are easier to analyze in theoretical way, but in Corollary 12 it seems that it is possible to figure out the NTK expression for original SIREN networks as well. In what aspect the modified model could embody better potential for analysis?
clarity,_quality,_novelty_and_reproducibility: Quality and Clarity: Great. The writing and organization of this paper is clear and well structured. The basic idea and methodology is easy to understand by readers, and the theoretical analysis and its related framework were solid.

Originality: Good. This paper is mainly based on two existing foundations, sinusoidal network and NTK technology. The author modified the original sinusoidal network to make it easy to analyze and outperform the original one empirically. Most computation of the NTK analysis used the existing framework of NTK, but the results are interesting and striking enough. Besides, the theoretical results match exactly with the experiments, endorsing the correctness of theoretical analysis.
summary_of_the_review: The message that this paper delivers is simple but helpful to understanding the behavior of sinusoidal networks. It also provides some guidelines of choosing better initialization parameters to achieve better models empirically. In the meantime, from the technical perspective, it is a successful attempt to use NTK as an approximation technology to analyze the proposed model in the theoretical way.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors analyse the NNGP kernel and NTK of neural networks with sinusoidal activations. The NTK has an adjustable band-width parameter, which allows users to understand such kernel models in terms of a low-pass filter. A discussion on how the cutoff frequency is chosen is provided. The authors show that such kernels can be applied to implicit models and differential equations.
strength_and_weaknesses: **Strengths:**
- To the best of my knowledge, this is the first paper that studies in-depth the way in which infinite-width neural network theory can help inform neural networks with sinusoidal activations that are trained using gradient-based optimisers.

**Mathematical correctness:**
- Theorem 2. In particular, the phrase "approximately standard normal distributed" is not precisely mathematically defined anywhere in the paper before the theorem is presented. Either change remove the theorem status of this statement, or define "approximately standard normal distributed". More importantly: asymptotic normality will hold (due to the previously cited results of Lee et al. 2018). If all you need is Lee et al.'s result, then just state this.
- "We can thus observe that this kernel approximates a Gaussian kernel, which is a low-pass filter, with its bandwidth define by ω". If you really want a squared exponential, RBF kernel, you can obtain this using half sine and half cosine activations. You can observe that summing the two kernels cancels out some terms. For example, see Proposition 18 of "Results on Infinitely Wide Multi-layer Perceptrons".

**Minor:**
- Theorem 13 has typo in "shalow". Note this kernel is very similar to the kernel in "Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions", which uses cosine instead of sine (but the analysis is pretty much identical). Maybe it is identical to Lemma 17. It is perhaps also worth mentioning that it is straight-forward to handle non-zero mean weights for these activations in the NNGP setting, since the integrals also have a closed-form expression. In general, this work might be worth citing as a Bayesian counterpart to the gradient-descent approach considered here (with the other difference that cosine is replaced with sine, which is not too important).
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. The experiments are described in adequate detail from a reproducibility perspective. While the individual elements presented in this paper are not entirely novel, they are synthesised into a coherent and reasonably compelling story. 
summary_of_the_review: This paper studies neural nets with sinusoidal activations in an NTK framework. A key bandwidth parameter and its importance on some representative problems is given. While the mathematical analysis does not require a huge amount of mathematical creativity, the paper is well organised and presents a clear story. There are some minor issues surrounding mathematical clarity and discussion of previous work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper analyzes sinusoidal neural networks from the Neural Tanget Kernal (NTK) perspective.  This analysis leads to a number of important observations.  The most important finding, perhaps, is that their NTK approximates a tuneable low-pass filter.  This insight is subsequently used to develop guidelines for optimizing the performance of a sinusoidal network by tuning the bandwidths of its kernels according the maximum frequency present in the input signal.  The paper also suggests an initialization scheme for sinusoidal networks that leads to improved results.  The ideas developed in this work are evaluated using two tasks---1) learning implicit models and 2) solving differential equations---and the results suggest that the ideas developed in this work have merit.
strength_and_weaknesses: This is a well-written paper.  I quite liked the narrative structure of this manuscript.  

The paper begins by constructing a simplified sinusoidal network model that mimics the key characterstics of SIREN but is much more amenable to theoretical analysis.  In addition to providing mathematical reasoning that confirms that the simplified sinusoidal model used in this work is similar to SIREN, the paper also provides empirical results that show that the simplified network achieves performance similar to that attained by SIREN.

Section 4 shows that the kernals of the simplified network approximates a Gaussian kernel whose width can be tuned.  

Section 5 uses a toy example to show how network behaves as this "width" parameter value shifts.  It is not immediately obvious to me how to parse the results presented in Figure 3.  The following sentence confuses me, "We can see that due to the simple nature of the signal, containing only two frequencies, there are only three loss levels."  Why is this?  It would be useful for a reader like me to include a sentence for why there are only three loss levels for the current two-frequency problem setup.

Section 6 discusses how to tune the aforementioned "width" parameter.  The motivation being that this value is "crucial for the learning of the network."  This discussion presents a heuristic for setting this "width" parameter to one-eighth of the maximum frequency in the signal.  Results in Figures 3 seems to imply this heuristic.  This section of the paper is somewhat underwhelming.  Why one-eighth?  Why not, say, one-tenth?  Given that it is okay to select a slightly suboptimal value for this "width" parameter since the network is able to adjust it during training.  It is however clear that using too large a value may results in overfitting.

Section 7 and 8 presents results and conclusions.
clarity,_quality,_novelty_and_reproducibility: This is a well-written paper.  Sinusoidal neural networks are increasingly being used to learn implicit models, and the work presented in this paper sheds light on the inner working on these networks.  In addition, the work also presents guidelines that can help an interested reader design sinusoidal networks that exhibit better performance and achieve faster convergence.  This is all good news. 
summary_of_the_review: This paper advances our understanding of sinusoidal networks, consequently this paper will be of interest to the larger machine learning community, and in particular the ICLR community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors derive a simplified sinusoidal network, the Simple Sinusoidal Network (SSN) in order to analyze such architectures from the NTK perspective, and use the generated insights to create an initialization strategy. 

This work is in an interesting area that has enjoyed recent attention especially after the publication of Sitzmann et al. 2020. The work is, therefore, very timely.

Overall, the paper presents a well rounded and solid narrative: first, the creation of a simpler network for the purpose of analysis, with the additional theoretical demonstration that it can still be made equivalent to the original type of networks (SIREN). The analysis is novel and insightful. The insights feed back to the modeling step, to allow the authors to propose an effective initialization. The key strength is not only that the SSN ends up being very competitive despite its simplicity, but also that we now have a better understanding of sinusoidal networks as (in the NTK limit) are shown to approximate a low-pass filter with bandwidth that can be adjusted/optimized. 

It would be interesting to investigate some more extensive applications of the initialization scheme to SIRENs, as proposed by reviewer pDxg. But in any case, the authors provide relevant theoretical analysis for SIRENs, so all in all the paper is certainly complete and will be useful to the community. 
justification_for_why_not_higher_score: It would be interesting to investigate some more extensive applications of the initialization scheme to SIRENs, as proposed by reviewer pDxg. 
justification_for_why_not_lower_score: The narrative of the paper is very solid: motivation -> modeling -> analysis/insights -> feeding insights back to modeling (initialization) -> good empirical results. Well rounded paper that should be of interest to the community.
Decision: Accept: poster