Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The paper proposes a framework for observing the eigenvector dynamics of EigenPro 3's training data in its implementation of kernel methods. The framework shows that significant change in the training data eigenvectors typically curtails early in training, suggesting bias towards equilibrium.
strengths_and_weaknesses: ## Strengths

- Provides a clear introduction to the motivation for understanding kernel methods and their application to deep learning.
- Explains the concept of EigenPro 3 and how it facilitates large-scale training based on preconditioned gradient descent.
- Discusses the relevance of the Neural Tangent Kernel and how it relates to the behavior of fully-connected neural networks.
- Provides an accessible framework for observing the eigenvector dynamics of EigenPro 3's training data and finding empirical evidence of change in the training data eigenvectors.
- Provides an easy-to-use Python implementation of the framework.
- Discusses the significance of the findings for gaining intuition into the inner workings of large-scale kernel training methods.

## Weaknesses

- The submission could benefit from more specific details on the results and their implications for the field.
- The submission could provide more context on the significance of the findings for the development of kernel-based machine learning algorithms.
- While the proposed framework for observing eigenvector dynamics in EigenPro 3 and RFMs is new and valuable, it does not necessarily introduce a fundamentally new idea to the field. However, it still contributes to the growing body of knowledge on kernel methods and their application to deep learning, and provides a useful tool for understanding the inner workings of large-scale kernel training algorithms.
- Not anonymized properly, the code implementation is available in a public GitHub repository.
suggested_changes: Overall, the submission is well-written and provides a clear overview of recent work on kernel methods in machine learning, focusing specifically on the EigenPro 3 algorithm and its use in Recursive Feature Machines (RFMs). However, there are a few suggestions that the author(s) could consider to improve the submission:

- While the submission provides a clear overview of recent work on kernel methods, it is not immediately clear what the authors' contributions are. It would be helpful to explicitly state the contributions of the paper in the introduction or abstract.
- The submission assumes some familiarity with the Neural Tangent Kernel (NTK) and Recursive Feature Machines (RFMs). It would be helpful to provide more context and background information for readers who may not be as familiar with these concepts.
- The submission briefly mentions that the authors "explore the dynamics of eigenvectors in training data in EigenPro 3" but does not provide much detail on what this entails or how it relates to the larger discussion on kernel methods in machine learning. Providing more detail on this would be helpful for readers.
- The submission uses some vague or imprecise language at times (e.g., "significant change," "apparent bias towards equilibrium"). Using more specific language would make the submission clearer and more precise.
- Consider including more concrete results: While the submission mentions that the authors found that "significant change in the training data eigenvectors typically curtails before five iterations" in the case of RFMs, it does not provide much detail on this or any other results. Including more concrete results would make the submission more compelling and informative.
- The submission mentions that the authors provide an "easy to use Python implementation" of their framework, but does not provide more details on the implementation description. Providing more detail on the implementation and/or a link to the code repository would be helpful for readers.
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: This study evaluates the dynamic change of eigenvectors in NTK and RFM and finds RFM achieves optimal performance in 5 iterations. 
strengths_and_weaknesses: Strengths

1. The paper is clearly written with comprehensive literature review and math details.

2. The experiment design is valid. 

3. Python code is provided to replicate the result. 
		
Weaknesses

1. The authors described the observation clearly but didn't explain why it happen. For example, what are the potential reasons for the "repulsive and attractive effects"?

2. The authors mentioned "in accordance with findings that RFMs achieve optimal performance in five iterations", what are those findings? Please provide references. Why is 5 the magic number? What are potential reasons? 
suggested_changes: Suggestions

1. For visualization, I would suggest using two different colors for RFM and NTK results separately. 

2. If you have room, I would suggest highlighting specific broader impact of this work. 

3. A potential minor mistake: I think the notation should be $y_i \in \mathbb{R}^n$ in the first sentence of Section 2. 

4. Please kindly explain why 5000 was used as model centers and 2000 was used as subsamples. 

5. Please explain the math formula of how change is measured in figures. 

6. It may be worth plotting change after 5 epochs when claiming "we did not observe significant change in training data eigenvectors after five iterations."
review_title: null

Decision: Invite to present