Out-of-Distribution Generalization in Kernel Regression
Authors: Abdulkadir Canatar
Blake Bordelon
Cengiz Pehlevan
Venue: NeurIPS 2021 Poster (2021)
Keywords: kernel regression
generalization
out-of-distribution

Abstract: In real word applications, data generating process for training a machine learning model often differs from what the model encounters in the test stage. Understanding how and whether machine learning models generalize  under such distributional shifts have been a theoretical challenge. Here, we study generalization in kernel regression when the training and test distributions are different using methods from statistical physics. Using the replica method, we derive an analytical formula for the out-of-distribution  generalization error applicable to any kernel and real datasets. We identify an overlap matrix that quantifies the mismatch between distributions for a given kernel as a key determinant of generalization performance under distribution shift. Using our analytical expressions we elucidate various generalization phenomena including possible improvement in generalization when there is a mismatch. We develop procedures for optimizing training and test distributions for a given data budget to find best and worst case generalizations under the shift.  We present applications of our theory to real and synthetic datasets and for many kernels. We compare results of our theory applied to Neural Tangent Kernel with simulations of wide networks and show agreement. We analyze linear regression in further depth.


Reviews:
[{"review_id": "vCeiVtQzycc", "reviewer": "Reviewer_Asg9", "summary": "The paper applies the so-called \"replica method\" to tackle the generalization issue for kernel regression where there is a distribution shift between training data and testing data. It presents the derivation of kernel regression by using an alternative perspective on RKHS functions. Empirical studies on real datasets including MNIST have been shown. Discussions with the neural tangent kernel are presented.\n\nDespite the interesting idea, the clarity of the paper can be improved to better appreciate the ideas and presentations. Moreover, the significance and impact of the work is also unclear, making it difficult to appreciate the advancement from existing results.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies the generalization issue for kernel regression. It presents an alternative perspective on deriving the objectives as well as interpret the kernel.\n\nDespite the interesting idea, the criticism on the paper is associated with the clarity of presentation and significance of impact to the community.\n\nIt is not easy to read and follow the ideas presented in the paper as it is not self-contained. Even after reading the appendix, it is still unclear what is \"replica method\" and how is this useful to improve the overall learning scheme for kernel regression. \nmore ideas are introduced in the Experimental section which induces more doubts and concerns. For instance, the beneficial and adversarial matrix in Fig 1 b): is this related to adversarial attack during learning procedure? \nFig 3 b) shows the double descent phenomenon while not clearly explained.\n\nIn terms of significance and impact, it is unclear how the overall setting improves the kernel regression when testing on cases with distribution shift. The regularised kernel-based regression objective is based on scalar-valued RKHS functions and squared loss. How does this approach objective different from learning a conditional mean embedding? With conditional mean embedding, distribution shift can be easily incorporated. Results on this approach have been studied in \n Grunewalder et, al. \"Conditional mean embeddings as regressors\"\nMoreover, Fukumizu et,al. \"Kernel Bayes rule\" is another useful reference in terms of manipulating conditional distributions and joint distribution in the kernel setting. \nI am curious how the proposed approach compares with the existing ones.\nIn addition, is the result only valid for energy-based models like Gibbs distribution presented? it will be nice to discuss what may happen beyond. \n\nminor issues:\n\n- \\langle \\rangle are used for both RKHS norm (e.g. Eq(1)) as well as the norm w.r.t. distribution/samples (e.g. Eq(2)) which are incoherent.\n\n- sample size notation P is non-conventional and especially can be confusing when distribution p(x) is used.\n\n- in Eq.(6), is Z the normalizer/partition function? otherwise, function O can be better explained and notated.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation is mainly the clarity of presentation and bringing up the significance/advancement from existing work (detailed in the main review). \n\nSocial impact is not an issue here but more discussions on how the distribution shift can be connected to social issues, such as fairness machine learning or privacy-preserving concerns in machine learning can be useful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "agXw5DibLpg", "reviewer": "Reviewer_wKiN", "summary": "This paper proposes asymptotic convergence results of kernel regression in the out-of-distribution setting via replica tricks. Some findings and examples are given. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I’m not an expert on statistical mechanics/physics though I know replica methods used in the machine learning community. I haven't checked the proofs in the appendix and will just trust authors in the correctness of the proof. \n\n**Main issues:**\n\nNevertheless, this paper corresponds to eigenvalue decomposition for $R^M \\times R^M$ matrix (with infinite $M$ and infinite feature dimension $D$) is still questionable to me. \nAs indicated by random matrix theory, matrix in spectral norm in high dimensions is quite different from the low-dimensional case. Besides, this paper does not strictly distinguish the difference between the integral operator and kernel matrix. In fact, such difference would be non-negligible: “The large eigenvalues of kernel matrix approximate the corresponding eigenvalues of kernel operator very well. The small eigenvalues are away from the smallest eigenvalues of kernel operator, but still relatively large.” Refer to the following paper:\n\n*Ma, C., Wu, L. and Weinan, E., 2020. The slow deterioration of the generalization error of the random feature model. Mathematical and Scientific Machine Learning, 2020.*\n\nProposition 1 is not enough and I suggest the authors to provide further analysis. For example, quantitative analysis/discuss the gap between the training/test distribution via some metric, e.g., MMD or total variance norm, and then study how such gap effects the third term in Eq. (7). It would be nice to provide error bounds in probability.\n\nIn Section 4, the authors consider linear regression with the linear kernel for Gaussian data, and thus the kernel eigenvalue equation can be solved exactly. This is a common way in replica methods. But this appears a little “overclaim” on application to any data, any kernel, any distribution throughout the paper.\n\n\n**Minor issues:**\n\nAll kernel eigenvalues are assumed to be non-zero, which means the RKHS is dense in L^2(X). This paper assumes the target function \\bar{f} lies in a RKHS, which is fair and standard in learning theory. In this case, if my understanding was right, “N” should be “M” in line 86?\n\nIt would be better if the definitions of some notations and expressions follow the machine learning community. For example, “generalization error” -> “excess risk” in line 58. They are in fact different and have different meanings in learning theory. Besides, I suggest the authors to add “P” in the loss term in Eq.(1) as the empirical error:\n$$\\frac{1}{2P} \\sum_{\\mu=1}^P (f(x^\\mu) - y)^2 + \\lambda || f ||_{\\mathcal{H}}^2$$\n\nThis setting is common in statistical learning theory though [14,15] is without $P$. \nWith $P$, it is natural to assume that $\\lambda = O(P^{-a})$ with $a \\in (0,1]$, refer to\n\n*Cucker, Felipe, and Ding Xuan Zhou. Learning theory: an approximation theory viewpoint. Vol. 24. Cambridge University Press, 2007.*\n\n\nTypo:\n\nThe test propability meansure should be $\\tilde{p}(x)$ in line 135.\n\nThe transpose operation is missing for $\\Phi$ in Eq. (SI.4).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "it's enough.", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "MedvvJA-jPY", "reviewer": "Reviewer_6Lb8", "summary": "This paper studies the out-of-distribution generalization error in kernel ridge regression (KRR) with any choice of regularization parameter $\\lambda$ (including the case of interpolation), and as a general functional of the training and test distributions using the replica method in statistical physics. In addition,\n\n-- they provide explicit closed-form expressions for the case of the linear model.\n\n-- using the neural tangent kernel (NTK), they empirically evaluate their asymptotic formulas on modified samples of the MNIST test dataset to provide best-case training distributions, which can sometimes be different from the one that exactly matches.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The setting of transfer learning in kernel ridge regression is well-motivated and an important topic for research, especially in the context of the recent discovery of the NTK regime, which increases the relevance of kernel analyses in the context of modern ML. To the best of my knowledge, I have not seen precise asymptotic analyses of the transfer learning setting in KRR, and so I think this paper will be of interest to the theoretical ML community. The experiments done on MNIST with a 3-layer NTK are also interesting.\n\nThe paper would benefit from a more detailed comparison with the results already derived for the case where the training and test distributions are the same (e.g. citation [14] in the submission), and a discussion of the additional technical non-trivialities that arise in the analysis for the out-of-distribution case. Additionally, are there settings that are more general than the linear model where the expressions could potentially be made closed-form?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impacts of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "KCK1iLqtJN2", "reviewer": "Reviewer_U6wb", "summary": "This paper studies the generalization capabilities in kernel regression in the particular case where the distribution of training and testing is different. \nThey quantify the mismatch between distributions for a given kernel and show that its a key determinant of generalization performance. \nThey also propose how to optimize training and test distributions for a given data budget to find the best and worst case generalizations under a shift.\nThe authors investigate how in real data this hypothesis can be confirmed and show how learning rate with training set size changes with training distribution and cases where choosing it different than test distribution helps.\nTheir approach provides analytical tool for having more precise predictions.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is a well-written and well-motivated paper. The problem of out-of-distribution generalization of neural models is a well-studied area and this work provides valuable insight in this area.    \n\nQuestions for the authors:\n\n- The effectiveness of this approach is in knowing what the test distribution look like. What do you think about the potential of the analysis you did for when the test distribution can be estimated but is not known? In NLP problems for instance, the distribution of output\n language can be to an extent estimated.\n\n- You show on MNIST dataset that the beneficial measure has lower generalization error while the adversarial measure has higher generalization error than the matched case. You use gradient ascent to find adversarial test measures. Does the type of adversarial samples (as opposed to less or more challenging/natural cases) influence the conclusion?\n\n- What are the limitations (computation-wise) of this approach? How well do you think it's adaptable for large (real world) datasets. In case of machine translation for example, the training data can be a couple of million samples.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper gives a detailed study for out-of-distribution generalization in kernel regression, going substantially beyond known results in this particular problem, which are both of independent interest and also may help serve as a baseline of understanding for similar results in other methods. Some concerns did come up in the reviews both in terms of the framing – which seem to be generally resolved in the discussion phase, though please make sure that these are reflected in the final version of the paper – and in particular for the style of result in Proposition 1, which is worth thinking a little further about. Overall, though, this will be a nice contribution to the conference.
Decision: Accept (Poster)