Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The main contribution of this paper is a study of how different loss minimization and generalization outcomes are realized: confusion (no loss minimization), memorization (loss minimization without generalization), fast generalization, and grokking (generalization, but significantly slower than the loss is minimized). There is a secondary result on the minimal training set size required to induce a representation in which different classes are linearly separable. All tasks studied in the paper are related to arithmetic operations like addition, with an appendix explaining that the setup studied in the paper can be used for all Abelian groups. The paper contains a thorough exploration of a toy system, which is used to analyze and illustrate the above-mentioned outcomes. Some results are included on a larger task with larger networks. 
strengths_and_weaknesses: - Strength: the exploration of the toy system is very thorough and offers an interesting ‘peek inside the black box’. It suggests very plausible mechanisms for how the learning process plays out.
- Weakness: the paper is quite limited in terms of bigger systems, which makes it difficult to assess how general the results are.
- Weakness: The current formulation of A1 (the answer to question Q1, when does generalization happen), while true, seems both a bit trivial and a bit vague. You say generalization happens ‘with a good representation’, or a representation whose ‘structure is appropriate for the task’. This answer leaves open the questions which representation needs to be good (is it the penultimate layer of the network? the input to the decoder? but where does the decoder start and the representation network end?) and when a representation is ‘good’ or a structure ‘appropriate’. Presuming that those questions are answered, it comes down to the statement that the mapping from the representation to the final required output is easy to learn. For the toy task you analyze you do go into the structure of the representation quite extensively, but since that analysis is limited to the toy setup, it is not obvious that the answers you find there generalize to harder / bigger tasks, deeper networks, and higher-dimensional representations. NB: A2 is only as meaningful as A1 is clear, which makes this issue more relevant.

questions: - Can you be a bit more explicit about the loss in the beginning? It would help reading the paper.
- If I read the paper correctly, you are saying that grokking happens when different parts of the network absorb information at different rates; in particular, when the last part (the decoder) is much faster than the preceding part. Is that a reasonable formulation? And if yes, is it possible to quantify that statement?
- What constitutes a part, for the purposes of the above question? (see also discussion of A1 in the ‘strengths and weaknesses’ section)
- How generally does your analysis apply? I can see from the appendices that it should generalize to other Abelian groups, but can you comment on different applications like image classification? The answers you give - linear representation, relative learning speeds of the decoder and upstream subnetwork - are not specific to the type of problem you study, so it would be very interesting (and make the paper much stronger) if you could also include results on other types of task.
- How strongly does the result for the critical training set size to obtain a linear representation depend on the simple effective loss you are using? Would it be possible to perform a similar analysis for image classification (or language modelling, or any other application you fancy)?
- Regarding the p=53 modulo addition experiment: is there a linear subspace in which you find the pizza wheel representation, like for the toy task? I understand it’s hard to visually inspect the 256D embeddings, but it might be interesting to see how many ‘effective dimensions’ there are - e.g. how many PCA components have significant explanatory power. It would be reasonable to expect a low-dimensional effective embedding, given how neatly t-SNE puts all the numbers in a circle. What happens if you take the two most explanatory principal components and make the same decoder plot there that you have for the toy task?
- How do your findings relate to the lottery ticket hypothesis? That also addresses generalization in unexpected situations - in overparameterized networks as opposed to late in training. Would having more highly overparameterized representation networks be another potential way to avoid grokking, given the lottery ticket hypothesis that bigger networks have higher chances of containing subnetworks with good initializations, that would therefore learn quickly?
- In the particle system analogy: What are the ‘internal particle interactions’? If they arise from the gradients computed from the loss, then they really originate ‘externally’ too, don’t they?

limitations: The main limitation comes from the size of the systems studied, as mentioned above. In principle the analysis could apply quite widely, but it is difficult to assess that from the experiments in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the grokking phenomenon under a physics-inspired "effective theory", which studies the phenomenon under toy settings. The grokking phenomenon refers to when generalization happens much later during training, long after training accuracy have reached 100%. The theory proposes the following explanations to grokking.
1) generalization is correlated to when the models learns a sensible representation of the inputs.
2) the critical training size (amount of data needed for generalization) relates to the amount of data needed to determine a good representation.
3) grokking represents a particular region in phase space of hyperparameters, and can be removed by using better hyperparameters.
strengths_and_weaknesses: ### Strength:

- The paper proposes an original view into the grokking phenomenon, which has been an interesting and unexplained phenomenon in the community. The phase diagram view into the landscape of training regimens illustrates interesting relationships between memorization, generalization, and grokking. One of the interesting questions about grokking is whether there is some magical inductive bias of gradient descent that leads to generalization after an abnormally long training period. The result of this paper would suggest that grokking is more of a pathological phenomenon that is the result of improper hyperparameter tuning. Though as I'll mention in the weakness section, I think that the support for this argument is not sufficient if it is indeed the conclusion of the authors.

- The paper proposes an explanation into the grokking phenomenon in the form of competition between representation learning and decoder overfitting. This seems plausible and is in line with a number of other works suggesting that later layers drive overfitting.

- Although the theory is based on a very simplified toy model, it provides an illuminating view into why a minimum number of training examples may be required for generalization. The predicted phase transition in Figure 4 is interesting to see.

### Weaknesses:

- One of the intriguing observations from Power et al [1] is that grokking can be seen both in unregularized models with sufficient data, or in regularized models with a small amount of training data. I think a sufficient explanation of the phenomenon needs to provide a justification for both behavior. However, the discussion around the effect of data size seems lacking. As far as I can tell, neither the theoretical nor the empirical analyses shed light on why more data speeds up generalization while less data can lead to grokking.

- Most of the results use an extremely toy setting. Only one set of experiments (modulo addition with varying learning rate and weight decay) is performed on the original grokking task from [1], while [1] contains a number of tasks and hyperparameter settings. Crucially the effect of data size is not studied, which was an important observation from [1]. It is unclear how much this theory really explains the original phenomenon, particularly given that the behavior of the phase diagram is qualitatively different between the setting in [1] and the toy model, namely that instead of models going from memorization to comprehension to grokking, in the real task setting it is going from memorization to grokking to comprehension. There is insufficient explanation to bridge this gap.
questions: - Is the analysis on "time towards the linear structure" independent of dataset size? Or can it shed light on the question of data size vs grokking / comprehension?

- What type of task characteristic makes a model most susceptible to grokking? If the mismatch between decoder learning speed and representation learning speed is key to grokking, would you expect that the four phases in the phase diagram to exist in general (e.g. for natural vision or nlp tasks)? If it's not general, what is it about the specific tasks and models studied in this work that leads to these different regions?
limitations: The authors address the limitations of the toy model. Though it is fine to use toy models to study these type of questions, I think that there is nonetheless insufficient work to show whether the insights from the toy model can actually generalize to other settings.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work studies the Grokking phenomenon -- a phenomenon in which generalization occurs long after mastering the training set. This work poses three questions with regard to this phenomenon:
1. When the generalization happens? and the answer is: when the representations become well-structured.
2. How the size of the training set affects this phenomenon? and the answer is: the critical size of the dataset is the amount required for learning a 'good' representation.
3. What leads to a delay in generalization? and the answer is: different phases of learning between "memorization" (perfect training acc but with lack of generalization) and "comprehension" (perfect training and generalization acc)

From a physics perspective, this work build the answers outlined above and provide supporting experiments.
strengths_and_weaknesses: **Strengths**:
- The problem is interesting and relevant: The Grokking phenomenon is a perplexing phenomenon which understanding it opens the doors to a better theory of generalization.
- The form of presentation is neat: I very much liked that the message is clear from the get go.
- The phase diagrams provide high-level intuitions: The same applies to the phase-diagrams that clearly distinguish between the 4 different phases.

** Weaknesses**:
- Some of the choices in the paper appears rather arbitrary and not fully justified (see questions below).
- At parts, it is difficult to follow certain arguments (at least for me): I am not from a physics back ground. For that it took me several wikipedia searches to grasp the meaning of some concepts in the paper. For example the term "effective" has a different meaning that the one that initially comes to mind.
- Very light on the related work: There has been a couple of papers that use tools from physics and provide phase diagrams for understanding the generalization of neural networks:
    + Generalisation error in learning with random features and the hidden manifold model by Gerace et al
    + Multi-scale Feature Learning Dynamics: Insights for Double Descent by Pezeshki et al
    + The Gaussian equivalence of generative models for learning with two-layer neural networks by Goldt et al
    + Statistical mechanics for neural networks with continuous-time dynamics by Kuhn et al
- Very limited experimental results.
questions: 1. The choice of Parallelogram-ratio for quantifying the quality of a representation is rather arbitrary. What is the idea behind using such a quantity?
2. Why don't you model the actual dynamics of GD on the training loss? Instead, you derive the dynamics of the 'effective' loss (defined in Eq. 5) which again appears arbitrary.
3. By defining the 'effective' loss, in essence the learning dynamics of the encoder is now decoupled from the decoder. I am right?
4. I have a hard time understanding lines 148-153. Would you explain why the third eigenvalue is the one that dominate the dynamics?
5. The analogy in lines 197-201; Don't you agree that it is the decoder that encourages the representations to have a structure rather than their "internal interactions". If the is not decoder, what forces the representations to change structure?
limitations: No negative societal impact is foreseen by this reviewer.
Authors talk about the limitations of the theory in line 160-165.
review_title: null

Meta-review: metareview: There was a consensus among reviewers that this paper should be accepted. In particular, reviewers felt, that the contribution of studying the rate at which different parts of a neural net absorb information, and the effects it has on learning and generalization is a worthwhile one that would enrich the literature and that the paper is a solid contribution to the NeurIPS literature.
confidence: Certain
Decision: Accept