Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The submitted paper studies the differences in the behavior of typical neural networks and the predictions made by the Neural Tangent Kernel (NTK) framework by showing that the NTK evolves and adapts to the current task during training. The experiments show that the predictions of NTK framework correctly rank tasks in terms of hardness but over- and under-estimate their hardness in different cases. The paper then studies the role of sample size and the directions where NTK evolves.
main_review: The main finding of the paper, that NTK evolves through the training to align its top eigenvectors to the task, has already been shown in recent work. There are still new and interesting results about how NTK evolves and its potential for overfitting and I'm leaning towards acceptance.

Adaptation of NTK to the task has been recently shown by [1] (also see [2] for an empirical study). Those papers also talk about benefits of alignment for convergence and generalization. I have not read those two papers in depth and I'm happy to discuss if the authors believe the submitted paper is showing a different phenomenon.

The most interesting result to me was Figure 8 that shows NTK alignment can hurt generalization. While some previous works describe NTK alignment as a sort of implicit regularizer and also the generalization bound in Eq (2) suggests that alignment helps generalization, Figure 8 shows that the story is more complicated and allowing the network to evolve its kernel can hurt generalization. Do the authors have an explanation for this apparent paradox? I suspect that in small data regime the NTK itself overfits to the training data so that its alignment increases on train data but stays constant (or maybe even decreases) on test data. A plot like Figure 6 but evaluated on both train and test data and in a setting where NTK evolution hurts generalization can clear this up.

Another interesting finding is Figure 7 (and the replications in the appendix) where NTK alignment happens mostly in a single direction, although I'm not sure how this would extend to more general cases. For example, if the target function was aligned with two eigenvectors of the initial NTK, would the evolution be in those two directions or still in the direction of one eigenvector?

The experiments highlight the role of alignment on convergence while the theory in the introduction only motivates alignment with a generalization bound. Bringing an alignment-dependent convergence analysis beside the generalization bound can help with motivating alignment. An example is Theorem 4.1 in [3] (that paper is about ranking of tasks in terms of hardness and does not study evolution of NTK).

Minor comments:

In some experiments like section 3.1 the eigenvector has to be binarized to be used as training labels. Any idea how much this binarization changes the direction of this vector?

It looks to me that the curve in Figure 7 (left) becomes equal to lambda^2. Mentioning this in the text can help with understanding the plot. For the curve on the right, it is good to mention that phi_i still refers to eigenvectors of the initial NTK if this is the case.

The choice of K=50 for Figure 6 looks arbitrary and the result might be different for another K. Can one use the scalar measure in Lemma 1 to get rid of K?


[1] https://arxiv.org/abs/2008.00938

[2] https://arxiv.org/abs/1910.08720

[3] https://arxiv.org/abs/1901.08584

Update: increased the score as the comment about related work is addressed.
limitations_and_societal_impact: The authors address limitations and societal impact.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Disclaimer: I'm only superficially familiar with the related literature, so this is rather low-confidence review.

The paper investigates whether the study of linearized neural networks via the neural tangent kernel is insightful for understanding generalization in deep learning. It argues that this is indeed the case based primarily on empirical evidence. Overall I am not fully convinced by the significance of the results in this paper, so that I am leaning towards a reject, but I would expect the other reviews to be more informative.
main_review: Overall, I think this is a polished and well-written paper. Considering the growing body of recent works on the NTK the topic of the submission is clearly of interest to the community. In particular the paper attempts to address recent concerns over the NTK not being informative regarding the generalization capabilities of finite-width neural networks.

To this end, the paper presents the following results:
- Based on labels generated from the eigenfunctions of the NTK, both linearized and finite-width NNs decrease in performance with increasing complexity of the learning taks. Further it observes that similarly on the recently proposed neural anisotropy directions performance decreases concurrently with increasing task complexity. These results complement previous works which have used the NTK to predict task complexities and gives them a slightly stronger theoretical foundation. However, even though the experiments are based on carefully chosen synthetic labels, the support is primarily empirical, which seems somewhat incremental and not nearly as strong as any theoretical guarantees.
- Next, the paper finds that neural networks primarily outperform their linearized counterparts in the large data regimes and argues that this is due to the distance of the optimum from initialization, which impacts the accuracy of the linearization approximation. It also notes that the alignment of the labels with the top eigenfunctions of the NTK increases through training.

While the results in this paper seem to fill some gaps in the literature, they feel somewhat incremental and mostly appear to add to previous observations rather than making novel ones of their own. Given this incremental nature I find that the predominantly empirical analysis lacks significance. If the authors think that this understates their contribution, I would encourage them to clarify what avenues of future research their work opens up specifically. The discussion at the end remains rather vague on this and (to my taste) focuses too much on summarizing the paper.
limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies the training dynamics of both neural networks and their linear approximations, and show that the finite-width NTK aligns to the target function during training, which in turn induces rapid convergence to a particular solution.
main_review: While this submission is well-written and covers an interesting avenue of research, most of its content has previously been published in a AISTAT2021 paper: Baratin et al., namely:

 - the dynamical bias of linear models towards functions that are aligned with the kernel (which is a standard result)
 - the increase in alignment of the NTK to the target during training of a neural network
 - the consequence of this increase in alignment on (linear) generalization guarantees

Therefore, depending on the NeurIPS policy regarding prior work, it looks to me that this work cannot be considered an original contribution. I am putting a low score only for that reason.

A comment regarding the experiment in section 4.1/fig 5: As you increase the training set size while keeping the minibatch size fixed, you also increase the number of minibatches in an epoch, thus there are more training iterations for the same number of epochs. Given that, I am not so surprised by the increasing l2 distance to initialization. Do these different behaviours between linear and non-linear dynamics occur because of the increasing number of examples, or because of the increasing number of training iterations?


Reference:
Baratin, George, Laurent, Hjelm, Lajoie, Vincent, and Lacoste-Julien, Implicit regularization via Neural Feature Alignment, AISTATS2021 https://arxiv.org/abs/2008.00938
limitations_and_societal_impact: yes
review_title: null

Meta-review: This paper explores what we can learn about neural network generalization through linearization, such as the empirical NTK. There were a number of supportive sentiments, but also concerns, discussed at length in the rebuttal period. One of the key concerns was novelty with respect to Baratin et. al. After discussion, it was understood that there are still novel contributions, such as the relative complexity of tasks being predicted with the empirical NTK, some experiments demonstrating the superiority of linearity model, and how alignment can hurt generalization. The final version of this paper should _carefully_ discuss Baratin et. al, and other related work, and address all of the reviewer questions, including experiments that were part of the rebuttal in the paper.

Decision: Accept (Poster)