Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: It is unclear what concrete contributions of this paper are. The paper does have quite a lot of equations, but it is hard to say that the derivations are upon a good problem. Experiments are severely lacking.
strength_and_weaknesses: Strength: lots of derivations.

Weakness: lack of experiments and competing results.

One fundamental issue: it has been long known that CNNs train better with whitened images -- which is equivalent to equalizing the images in the frequency domain beforehand. There is no room for sensitivity once proper preprocessing is done. The analysis of this paper seems irrelevant and may be in a wrong direction.

clarity,_quality,_novelty_and_reproducibility: Novelty and significance is below the standard for ICLR.

summary_of_the_review: I vote for rejection because the significance of the analysis is low.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a theoretical explanation for the observation of imbalanced frequency sensitivity in CNNs. The paper studies linear CNNs under weight penalties, and derives justification for the dependence of such CNNs' frequency sensitivity on the distribution of power in the data spectrum. Two empirical experiments involving actual non-linear CNNs trained on CIFAR are provided to support some predictions of the proposed theory, namely that adding more layers and increasing weight decay coefficient will both emphasize frequency sensitivity of CNNs to lower frequency components.
strength_and_weaknesses: **Strengths**

The paper posits an interesting theorem, and motivates it sufficiently well with some prior evidence. The writing is mostly clear, although the format is wrong. While I don’t think the main claim of the paper is correct, the much more restricted claim that is actually proved is interesting and can help explain some observations in practice.

**Weaknesses**

I have the following concerns/questions about the paper:

1) The main result in theorem 4.9 concerns a very specific case of linear CNNs, and as the paper itself points out is a quite straightforward extension of Tibshirani 2021. Now this in itself is not an issue, what becomes an issue is that the paper does not go much beyond this extension, for example considering the effect of size-limited filters and also the more complicated case of simple ReLU non-linearities, which would make the theory much more substantial.

2) The theoretical results really show the sensitivity caused by the norm penalties, rather than the CNN itself, since removing the penalties (lambda=0) will result in a “non-sensitive” CNN. In other words, the simplification of CNN structure has stripped it off its many inductive biases, and as a result exposing the inductive bias due to norm penalties, which are well-known already (although not explicitly discussed in the spectrum as far as I know). So it is misleading to claim “CNNs inherit frequency sensitivity from image statistics” where the theory only shows this happening due to norm penalties when CNN is extremely simplified.

3) As the paper itself points out in Section 6, the empirical experiments only on a low-frequency prominent dataset (CIFAR) do not provide any evidence for the main claim of the paper: “CNNs inherit frequency sensitivity from image statistics”. Repeating these experiments on datasets where different frequency bands are prominent (which are trivial to construct) are a strict requirement to support the main claim. Also, the empirical results should always report error bars to show statistical significance of the results (no figure currently shows error bars).

clarity,_quality,_novelty_and_reproducibility: The paper reads well and is clear for the most part. I think the novelty is limited since the main theorem does not go much beyond the straight forward extensions of known theorems, and the empirical results do not really support the theoretical extensions or the main claim of the paper (as detailed above). The results appear to be sufficiently reproducible.

**Typos**

The formatting is not for ICLR 2023.
1/p_i in Conjecture 4.13.
Figure 6 missing caption.
summary_of_the_review: The paper provides a theory explaining how linear CNNs under certain assumptions and weight penalties tend to have an inductive bias reflecting the spectral composition of the input data. However, the very restricted theorem, and lack of extensive empirical experiments supporting the theorem and the main claim, limits the paper’s significance and validity.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study the sensitivity of convolutional networks (CNNs) to perturbations in frequency components of input images. The authors consider linear convolutional networks. By applying the Fourier transform, in some simple settings, the problems can be reduced to,  essentially, a linear regression (or lasso). Then, tools from linear models (e.g. eigen-decomposition) can be applied to compute the closed-form predictor, which can be used to analyze how sensitive the linear predictor is to perturbation in different Fourier modes. 
strength_and_weaknesses: # Strength 

- The problem studied in this paper is quite interesting. 
- The paper is relatively easy to follow. 

# Weaknesses

- The setting is quite simple. The authors study linear convolutional networks, which is not that different from linear models. 

- Even for linear convolutional networks, the analysis is only restricted to a couple of simple cases (Theorem 4.9). E.g., channel size all equal to 1 or the depth L\leq 2. The general, more practical setting (e.g., multiple channels), which required new ideas beyond basic Fourier transforms, is not covered in the paper. 

- The reliance on Fourier transform also requires the kernel size ( 3 by 3 in practice) to be the same as the image size (e.g., 32 by 32 for Cifar 10). This restriction reduced the interest and relevance of the approach. Finding minimizers in the frequency domain  


clarity,_quality,_novelty_and_reproducibility: - The use of the notation $\cdot$ (`dot`) is confusing and non-standard. The Fourier transform of a `multi-channel` convolution is not pointwise multiplication, but it is a matrix algebra with entries of the matrix being a linear operator (==convolution in the paper). Though the authors correctly defined such mapping, the `dot` notation shouldn't be used here as it is non-standard. 

- Definition 4.6. Why introduce a new notation $C_p$ norm? As you mentioned, this is the Schatten p-norm and which is standard terminology. 
summary_of_the_review: update 

I thank the authors for improving the paper. In particular, the current form of theorem 4.9, if correct, is much stronger than the previous version. As such, I increase my score accordingly. However, I am not comfortable accepting the paper at the moment for the reasons below. 1. the paper is still in the linear convolutional setting, aka, Tibshirani, Gunasekar, etc., though the improvement is not as obvious as the previous version. 2. There are some significant changes in the main text of the paper, as well as the main result Theorem 4.9. Because of so, I would prefer the paper to go through a new review process with the proof of Lemma 4.11 carefully checked. 


------------------------------


Overall, the paper has several interesting observations. However, the contribution is incremental and does not go beyond existing work much, e.g., Tibshirani, Gunasekar, etc. The setting only covers some simple cases of linear convolutional networks. The obtained insights (e.g., sparsity, the predictor is more sensitive to low-frequency mode) seem not that new and can be derived from (deep) linear models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the authors develop a theoretical analysis of the sensitivity of convolutional neural networks to perturbations in frequency.  The reviewers found the motivation and problem setting compelling.  However, a common criticism was that the theory was applied in an overly simplified setting, i.e. linear CNNs, and as a result both lacked novelty over existing work (Tibshirani, etc.) and didn't satisfy the paper's claims.  In discussion, the authors presented an expanded analysis that seemed to please two of the reviewers, who both agreed that it could be convincing and raised their scores.  However, the reviewers still did not recommend an accept, because they felt the changes were too substantial to review and verify within the scope of the review period.  As a result the review scores are 3, 5, 5.  (Note that the review with the 3 score is terse and it's not clear that their review was easy to address or led to a productive discussion.).  As it stands, it seems clear that the paper was not ready for acceptance upon submission, but appears to have improved significantly as a result of the author's changes during the discussion period.  However, the change is too big for the reviewers to validate.  Therefore, the recommendation is to reject the paper but encourage the authors to continue to develop the theory and presentation for a future submission.
justification_for_why_not_higher_score: The paper was an obvious reject at submission time, due to lack of novelty and significance.
Decision: Reject