Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Summary of paper:** A series of empirical observations are made about the influence of scale of init on generalization (in particular, that a continuum of generalization performance from random to very good can be generated by varying only the scale of init) , and these effects are explained in detail for different activation functions. The authors also propose a measure of gradient alignment which they show correlates with generalization performance

**Pros/strong points:**
 - detailed explanations for each activation function provide nice insights
 - solid experiments

**Cons/weak points:**
 - overall clarity and presentation of information is the largest weakness in my opinion, although the writing is generally good. I think it just needs a few more passes, with an eye to making sure things are accessible/understandable/flow. Could be improved substantially just with formatting/subsections or something, e.g. italicizing key insights or making sub paragraphs where each gives a particular insight
 - some small things in related work

**Summary of review + recommendation:** Overall I think this is a good paper, and could be a very good paper with some "tightening up" and clarifications. The combination of things is too much for me to recommend acceptance out of the box, but the things are relatively small and I think easy to address, and I'd be happy to increase my score.

**Detailed review and Specific questions/recommendations:**
 - unclear what "large scale training" means
 - "engendering" is an unnecessary word there
 - observations about overparameterized models should be cited, e.g. Zhang et al.  and Arpit et al.
 - Chizat & Bach further observe (not observes), same incorrect pluralization with many citations (suggest checking the whole document)
 - background work portion of the intro misses works, some poorly explained / relationship to current work not discussed, and overall feels rushed. The Related Work section does a good job mostly though. I suggest moving the 3rd p of the intro into the related work, moving the first section of it about scale of init to the first sentence of Contributions. Merging them should get you some extra space for more experiments/larger figs.
 - Related work on inits should cite lottery ticket works (e.g. Frankle et al)
 - Geiger et al reference you describe what they do but not what to take away from it
 - extreme memorization should be bolded since it's a term you're defining (and make clear if you're proposing this term and if not, where it is from), but you then  define memorization the same way you define extreme memorization, making this term ("extreme") seem unnecessary. 
 - "from verylittle overfitting to perfectly memorizing the training set while making zero progress on test error" this sentence is unclear, makes it sound like "while..." applies to both of the 2 extremes. Suggest rephrasing.
 - I find 2nd bullet of contributions unclear about what is expected vs. what happens and what we learn from that 
 - in 3rd bullet briefly summarize the alignment measure. I suggest using a different/more precise term for this measure (e.g. gradient alignment) since just "alignment" means so many things already
 - In "related statistics" you don't mention if Chatterjee has a measure for coherence of gradients (I skimmed that paper and it seems not, but I'm not sure). If not, then maybe calling this a measure of gradient coherence would be appropriate? I googled it quickly and it seems that coherence means something specific in linear algebra and signal processing: (https://en.wikipedia.org/wiki/Mutual_coherence_(linear_algebra), https://en.wikipedia.org/wiki/Coherence_(signal_processing)) - maybe not necessary to comment on in the paper, but if the authors are familiar with this use of the term I'd appreciate a clarification of how it relates to the mentioned measures of alignment I'd be interested
 - "related statistics" should also mention Arpit et al critical sample ratio (which does take class information into account) and comment on differences (or if it's too different to include here, I'd appreciate an explanation of why)
 - Mention computational cost of the different measures of alignment
 - homogeneity is repeatedly mentioned without explanation (just a brief 1-line would do)
 - "fix the scale" ambiguous whether this means fix in place (at a particular value) or fix as in correct
 - seems obvious to me that the scaling would affect relus (especially in the absence of bias as your experiments say); if the scale is larger, fewer values are initialized near the non-linear region of the relus, meaning there are more 'dead relus' near the beginning (which I would guess up to a point could provide regularization, but past that point would just make learning slow and even unstable). Could the authors comment on this; do you think it's correct/relevant? How does it fit with the argument about homogeneity?
 - State important equations in words as well as math for clarity and ease of reading (as is done for eq.6; make sure this is done consistently, especially important for your proposed measures of alignment).
 - Conclusion discusses the results strangely, without mentioning the actual results (e.g. "making it particularly interesting" - why/how is it interesting, what are the implications for people using sin?, "the loss function plays a crucial role" what role? what things are good for what, what should I look out for?). The conclusion should stand on its own and summarize results, not reference them in a way that requires me to have read the whole paper to understand.
review_title: Good paper with nice experiments and insights that needs cleaning up/clearer presentation

final_score: rating: 7: Good paper, accept
review_content: ###
Summary:
This paper investigates the role of scale in generalization of neural networks. It shows that the initial scale of a 2-layers MLP, with sinusoid or ReLU activation, can control the memorization behavior of the network, from very little overfitting to complete memorization.
It then proposes an alignment measure which correlates with generalization for different initial scale. It shows that this alignment measure can capture generalization performances for other architecture such as ResNet or DenseNet on the CIFAR-10 dataset.

###
Reasons for score: 
Overall, I find the paper to be a bit borderline. 
The observation regarding the scale impacting generalization is novel and interesting as I would have assumed that large initial scale would lead to bad optimization rather than a lack of generalization.
However, all the experiments regarding the scale are carried out on a two-layers MLP models and it is not clear to me if similar conclusion would be true for deeper architecture.
 
 
###
 Pros:
- interesting observation regarding the impact of the initial scale on generalization
- clearly show the effect in a two-layers MLP with various activation and loss functions
- propose an alignment measure which have some promising correlation with generalization
 
###
Cons: 
- experiments investigating the impact of the initialization scales only for two layers MLPs
- Alignment is not compared with other generalization metrics in section 5.


review_title: Memorization and Scale Initialization

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ## Overview 

The paper studies how the generalization of the neural network trained with SGD is affected by the scale of the random initialization. Specifically, that when the scale of the initialization is big, the network overfits to the training set with bad performance on the test set. 
The paper also provides a hypothesis why does it happen -- that the graidents of difference examples are orthogonal in the "bad" mode and proposes a measure called "alignment" to predict the generalization reghime of the network.

The problem, tackled in paper is interesting and the paper itself is thought-provoking. 


The central model studied is 2 layer fully-connected neural network with {sin, ReLU} activation, where the 1st layer is initialized with variable (studies) scale and the 2nd - with Xavier init.
Both layers are without biases. The model is optimized with SGD w/o momentum and constant learning rate and the loss function is one of the classification losses: CE, hinge loss.

## Strong points

The paper formulates hypothesis and verifies it via series of controlled experiments. Besides "toy" 2-layer model, the similar results are get with more powerful architectures, like CNN, DenseNet and so on on the set of middle-sized datasets like SVHN and CIFAR.  
The paper clearly states its place among related works and proposes a useful metric for diagnosing model training. 

## Questions (and possible weak points


1. Figure 2 shows that the norm(grad)/norm(weight) decreases significantly with the scaling-up the initalization. This is hypothesized to be one of the problems with large scale initialization: the weights do not go far from the original (random) values. If this is the case and the problem, may be scaling up the learning rate with the initalization to keep the norm constant, would help?
Why is this an important question? Because the problem with large init is the bad gradient direction (as hypothesized in paper), then scaling the learning rate would not help. If, otherwise, the problem is the learning rate scale, then the work could be seen as indirect confirmation of the works about large-vs-small learning rate reghimes, e.g.
Lewkowycz et.al.  https://arxiv.org/pdf/2003.02218.pdf
Moreover, may be then we should study not the scale of init, but the ratio (init scale/lr). 


2. Following Q1, one also could use different learning rates for different layers depending on the scale of weights and/or activation. Yes, that would significantly complicate the experimental setup, but may lead to the different conclusions. E.g. it is commonly know that deep sigmoid networks are hard to train because of vanishing gradient issue. However, as it was shown in "Revise Saturated Activation Functions" Xu et.al (https://openreview.net/forum?id=D1VDjyJjXF5jEJ1zfE53), one could perfectly fine train such networks. The only thing, which is needed, is the proper rescaling of the learning rate per layer, up to ridiculous values such as 4^11.


3. (minor) why do you don't use bias? Bias-less NNs are less common and give worse results. 

4.  (just curious) Where is the actual learning with large scale init take place - in layer 1 or 2? If only one layer is actually trained, can we obtain the same results, when the non-training layer weights are frozen?


5. When examining Figures in appendix (e.g. 17, 20, etc), it looks like that proposed "alignment" measure can be used as alarm -- if it is low, the generalization is low, but the ranking of (test accuracy) and (alignment) is not really aligned. Any comments on that?

Overall, I like the paper, although would also like to have answers to my questions.

## Update after rebuttal

All my concerns have been addressed, including the possible alternative explanation of the experimental results. I strongly recommend the paper to be accepted.
review_title: Paper is interesting, but I am not sure about the experiment setup

Meta-review: This paper provides a clear and useful empirical study of how the initialization scale and activations function affects the generalization capability of neural networks. Previous works showing the effect of the initialization scale (Chizat and Bach (2018), Geiger et al. (2019), Woodorth et al. (2020)) had a more limited set of experiments. Moreover, here an extreme case is shown, wherewith sin activation function no generalization is possible at a large init scale (there the kernel regime is useless for generalization since the hidden layer output becomes very sensitive to any small perturbation in the input). Lastly, two alignment measures are suggested, which are correlated with the generalization across several architectures and initialization scales.

All the reviewers argued for acceptance, and one strongly so. I agree that the paper is sufficiently interesting and clear to be accepted. However, despite the high scores, I only recommend a poster and not spotlight/oral: I think the novelty of the empirical study is not groundbreaking, given the experiments in previous works, and the usefulness of the suggested measures are not completely clear without a thorough comparison against previously suggested measures.
Decision: Accept (Poster)