Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies coordinate-MLPs, a class of multi-layer perceptrons that are trained to parametrize high-frequency signals over a low-dimensional domain.

The paper analyzes three main classes of coordinate-MLP, respectively based on random Fourier features, sinusoidal activations, and Gaussian activations. By proposing a new way of computing the Fourier transform of these MLPs (in the two layers case), the authors observe the effect of the main hyperparameters of the models on the spectrum.
Similarly, the authors study the effect of increasing the depth of the MLPs. 

The authors conclude that, in all three cases, changing hyperparameters to allow the model to represent higher frequencies has the effect of decreasing the spectral energy in the lower frequencies.
The authors use this result to propose a regularization scheme based on minimizing the trace of $\mathbf{J}\mathbf{J}^\top$, where $\mathbf{J}$ is the Jacobian of the penultimate layer. This has the effect of limiting the high-frequency components of the MLP associated with low-energy regions of the target signal. 

However, the authors also note that one can avoid computing the Jacobian by minimizing a much simpler loss designed to make the output of the penultimate layer smoother along randomly sampled directions. 

The paper is concluded with a series of experiments showing that the regularized MLPs can learn to generalize better than their unregularized counterparts, on signals sampled at variable intervals and on signals with specific spectral features (e.g., Fig. 7).
strengths_and_weaknesses: Overall, I really enjoyed the paper and I can safely recommend its acceptance. 
The contributions are entirely novel and the paper is very well-written and explained. 
The paper is likely to have a significant impact on the growing community of implicit neural representations/neural fields and gives non-trivial insights into well-known problems. 

I apologize that I cannot write a longer review, but I only have a few minor comments on the paper: 

- The numbers reported in Table 1 should be explained in the text or the caption. 
- The reasoning in lines 252-257 is not immediate to follow. I suggest the authors spend a few more words to facilitate the intuition of the result beyond the empirical observation. 
- The comparisons in Figure 4 are difficult to see, I suggest either increasing the size of the images or the resolution (or zooming in on the cutouts).
- Figure 6 could be improved by turning it into parallel line plots instead of scatter plots. 
- The authors should comment on the empirical differences between the regularization based on the Jacobian and the simplified regularization. An experiment showing the difference in practice (computational cost, quality of the learned signal, etc.) would be a valuable addition to the paper. 
- Typos: 
    - Line 47: Biship -> Bishop
    - Line 162: Fourier transform the activation -> Fourier transform of the activation
    - Line 288: Much more simpler -> Much simpler
questions: - In Section 5.2, it is not clear why increasing $a$ means decreasing the energy of the low-frequency components. Can the authors comment on this?
limitations: The limitations of the work are not really discussed. There's probably something to say about the fact that the theoretical analysis is conducted on 2-layer MLPs and the analysis is assumed to hold on deeper networks.
Also, the authors should comment on the empirical differences between the regularization based on the Jacobian and the simplified regularization.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper shows that coordinate MLPs lose some of their implicit regularisation properties when they are adapted to better represent high frequency signals. This suggests that implicit regularisation properties of neural networks are not necessarily a consequence of using SGD, but instead related to the network architecture. The authors suggest a new regularisation technique that allows coordinate MLPs to maintain high quality in their representation of low frequency signals, while still allowing improved representation of high frequency signals. The performance of the suggested explicit regularisation is demonstrated by multiple experiments.
strengths_and_weaknesses: ### Originality

Loss of implicit generalisation in coordinate MLPs has been observed before, but this paper provides some theoretical reasoning to why this happens. The paper also suggests a new method to help resolve this problem.

### Quality

Extensive references to prior work, consistent story throughout the paper, theoretical result of why implicit regularisation of coordinate MLPs deteriorates are mostly supported by proper derivations. For some findings, proper proofs are not provided but intuitive reasoning supported by maths is still given.

### Clarity

Very clear explanations of all relevant concepts, well readable without going into references also for a reader with less expertise in the specific topic. Experiments help to clarify the effects of the proposed method as well as the existing issues with coordinate MLPs. Contributions are very clearly written out.

### Significance

New method proposed with practical use in a field that has recently been popular.
questions: ### Q1

In Table1 what is difference of L-PSNR, R-PSNR and T-PSNR? I think I can roughly deduct what they are, but explicitly stating these would be helpful.

### Typos and other minor things
Line 147 typo: positonal

Line 162 typo: transform _of_ the activation?

Figure 2 caption overlaps with x-axis titles

Line 210 typo: follwing

Figure 4 caption typo: densly

Fig 7 caption: is able _to_ achieve

limitations: The authors comment on which of their results are novel, but the limitations of the proposed regularisation method are not included. It could be useful to include a comment on whether using the proposed method makes training the coordinate MLPs more difficult or requires tuning some training related hyperparameters. Potential societal impact of the work is not discussed, but this doesn't seem very relevant or necessary for this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies implicit regularization in implicit neural representation approaches, primarily coordinate-MLPs. The authors analyze failure modes of coordinate-MLPs by carrying out an analysis of shallow MLP models, investigating the effect of depth and hyperparameters on their spectral bias. A smoothness regularization method is introduced and evaluated on image encoding.
strengths_and_weaknesses: **Strengths**

* The authors carry out an extensive analysis of the behavior of various implicit neural architectures (sinusoidal, gaussian and RFFs) commonly used. Curated examples are provided in support of the theoretical investigation. 
* The proposed regularizer is conceptually simple and appears to improve PSNR across evaluations. 

**Weaknesses**

* Only images are considered. The paper would be stronger if the proposed regularized was tested on different modalities, such as encoding of audio samples. The experimental evaluation is generally limited - only 500 images from ImageNet (unclear how these are chosen), a subset of STL, and some synthetic data.  
* No code is provided. The experimental details provided are not sufficient for reproducibility.


Some minor typos and comments:
* l 162: Fourier transform OF the activation exists
* l 339: Bishop rather than Biship.
* l 528 (appendix), missing reference (??)
* 576 (appendix) erilest -> earliest
* Readability of Lemma A.1 proof could be improved by always specifying the integration domain, particularly the steps where Fubini's theorem is invoked.
questions: * Coordinate-MLPs are finding application even outside of implicit representation of data. It is common to provide "coordinates" as additional inputs to neural network architectures, or to use periodic activation functions (see for example current architectures for diffusion models (1), or neural operators (2, 3) ). I am curious if the authors think the proposed regularized could be helpful even in general deep learning tasks.

(1) Ho et al., Denoising Diffusion Probabilistic Models

(2) Li et al., Fourier Neural Operator for Parametric Partial Differential Equations

(3) Poli et al., Transform Once: Efficient Operator Learning in Frequency Domain
limitations: The authors provided a satisfactory discussion on limitations and comparisons with similar approaches (A.6 and A.7).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: One-liner: Authors study the spectral bias of INRs, and propose a regularizer to fix it.

This paper argues that popular neural network architectures for INRs---or coordinate-MLPs, as authors call it---tend to learn functions with "suppressed" low frequency components. To support this claim, the paper provides (i) Fourier transforms of the function expressed by (the two-layer versions of) these architectures which hints the effect of depth and hyperparameters on low-and-high frequency components, and (ii) empirical results showing that the spectra of the functions learned by these architectures depend on the hyperparameters and depth. Based on this observation, authors propose a (randomized) regularizer which aims to amplify the low-frequency component of the learned signal.
strengths_and_weaknesses: - A strength of this paper is its section 7, where the authors design a regularizer motivated by the spectral understanding of the network. Although I do not agree to some points (e.g., the tightness of the lower bound, as being sharp is different from being tight), I do find the approach original and interesting.
 
- Another thing that I like about this paper is the quality of writing. I appreciate the authors' effort on distinguishing what can be shown rigorously and what cannot be, e.g., as in Line 255.
 
- Also, the message conveyed in the paper is potentially very meaningful: "Conventional way of boosting the expressivity of INRs may be at odds with boosting the generalization performance, unlike as in conventional model architectures."

- A weak point is that some of the main claims are not properly formalized or ill-supported. As an example, authors argue that 'vanilla INRs are incapable of simultaneously generalizing well at both lower and higher ends of the spectrum,' or 'INRs do not naturally converge to solutions with rich spectrums in both low and high frequencies.' Here, I am not really sure what authors mean---I do not understand the ideal property the authors are looking for. From figure 1 (and later figures), I think that authors are actually looking for architectural bias that 'prefers low-frequency solutions whenever a perfect fit is possible,' instead of one that is frequency-neutral. This might be the reason why I was a bit puzzled when I saw Figure 6, where the authors claim that INRs have "suppressed" low freq. From the figure, I cannot really tell---the color distribution seems to be quite uniform on SIREN and RFF, given deep enough depth and large enough hyperparameters? I cannot imagine the "ideal heatmap" the authors have in mind, so that the SIREN/RFF has a relatively smaller low freq components. This part should be clear.
 
- In my opinion, theoretical contribution of this paper is rather weak. The paper does a nice job in presenting two-layer Fourier transforms of INRs, but I do not think they directly imply the main message. Especially, the arguments in section 6 (depth) is quite hand-wavy and not persuasive. In this aspect, I suggest that authors put more emphasis on describing the empirical observations and contributions.
 
- As theoretical analysis does not give any decisive conclusion, I think that the the experimental section needs to be more emphasized. Section 8 lacks a proper baseline---one should test the effectiveness of the proposed regularizer by comparing against existing regularizers, e.g., weight decay. In fact, it seems very likely that simply performing the weight decay may help boosting the low-frequency component of the learned signal, as model weights are what is being used as "frequency," especially in SIRENs.

- A nitpick is that it is very difficult to read the text on most figures if I print the paper.

- Nitpick#2: Line 75: "this effect is common to all the types of coordinate-MLPs." This cannot be true, as there are much more different architectures for INRs, e.g., multiplicative filter networks.

- Nitpick#3: I do not agree to the authors' point that previous mainstream works focused more on the contribution of SGD on the model bias than the architectural bias. In fact, while the term "implicit bias" or "implicit regularization" is being almost exclusively used for describing the work about how GD/SGD shapes the learned function, there are still many works on the impact of model architectures. For instance, there are plenty of works regarding ConvNet bias vs. Transformer bias, e.g., [ConViT](https://arxiv.org/abs/2103.10697). I do not even think that Zhang et al.'s main message was that implicit bias is "not" tied to the architecture.
questions: Here is an additional question, that I do not necessarily view as a weakness but very important to answer nevertheless.

- One of the critical architectural factor has not been discussed---width! From my understanding, width is one of the most critical factors about the implicit regularizations of overparameterized neural networks (not confined to INR context). In fact, I believe that width is often treated as a more important factor for generalization than depth. Indeed, I did observe from my own experimental results that widening the networks tend to have beneficial impact on the super-resolution capabilities of INRs. I wonder if the authors' analytical tools can be useful in studying the effects of width. This should help strengthening the main message of the paper.
limitations: I couldn't find any explicit discussions on limitations or potential negative impact, but I do not see this as a big drawback, because I also could not see a very clear potential negative societal impact of this work.
review_title: null

Meta-review: metareview: The paper initially received three positive reviews and a borderline reject one. After the rebuttal, all reviewers are voting for accepting the paper. The area chair agrees with their assessment and follows their recommendation.
confidence: Certain
Decision: Accept