Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper deals with 3 theoretical properties of ridge regression. First, it proves that the ridge regression estimator is equivalent to a specific representation which is useful as for instance it can be used to derive the training error of the ridge estimator. Second, it provides a bias correction mechanism for ridge regression and finally it provides proofs regarding the accuracy of several sketching algorithms for ridge regression.
 
The paper addresses an important problem and puts itself nicely in context of previous work. However, it comes across as three papers stapled together, that were submitted to some journal and have now been put into ICLR format. Most of the important results are in the appendix. The main body of the paper is just a smattering of theorems with some text flowing around them and too much notation. There is little or no intuition provided for the proofs in the paper. Moreover, the connection between the 3 theoretical properties of ridge regression studied is also unclear. There could very well be 2 or 3 conference papers written out of this one paper. 

As far as the technical merit is concerned, I checked some theory and it appears correct, however some things were unclear. For example, the Theorem 2.1 is proven under a random design setting; how would the proven ridge representation look under a fixed design setting? Or can we even prove something in that case?

I think the paper definitely has some merit but the presentation makes it hard to assess it. 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents a theoretical study of ridge regression, focusing on the practical problems of correcting for the bias of the cross-validation based estimate of the optimal regularisation parameter, and quantification of the asymptotic risk of sketching algorithms for ridge regression, both in the p / n -> gamma in (0, 1) regime (n = # data points, p = # dimensions). The authors derive most of their results exploiting their (AFAICT) new asymptotic characterisation of the ridge regression estimator which may be of independent interest. The whole study is complemented by a series of numerical experiments.

I am recommending this paper to be accepted for publication at ICLR. The paper is clearly written, makes several solid theoretical contributions, and recommends a simple and practical bias correction for CV-based estimates of the optimal ridge regulariser. While ICLR is biased towards deep learning focused publications, the work of Belkin, Hsu, Ma, Bartlett, Hastie, Montanari, Rakhlin, Liang and others (apologies to everyone whose name was omitted---it is for the sole reason of brevity) has recently shown that we can learn non-negligible amount about neural networks from study of linear models.


Comments:

- Most of the examples in the paper focus on the regime gamma < 1. Would you expect the observed behaviours to be significantly different when gamma > 1? I am asking specifically because of [1] which has found that the bias of the risk estimate obtained via cross-validation is often most extreme when p >> n, which makes me wonder about what would an experiment like those in fig.2 look like in the p >> n regime?

- I was somewhat confused when I first read the statement of thm.2.1. In particular, the definition of asymptotic equivalence requires (roughly speaking) that any series of projections of the difference between the two random vectors converges to zero a.s. However, within the theorem, you introduce Z without much explanation which confused me because not every Z ~ standard normal would be asymptotically equivalent. I needed to look at the proof to understand how Z is “coupled” with hat(beta), which I think should not be necessary. If possible, I would either say that there exists a (series of) Z (all standard normal) s.t. the asymptotic equivalence holds, or add some other clarification (possibly in the form of a footnote).

- When you are citing a book, please consider citing exact pages or at least chapters/sections (e.g., when citing the exact shortcut from Hastie et al. (2019)).

- In the definition of asymptotic equivalence (starting at the bottom of p.2), did you mean to assume limsup ||w|| < \infty a.s. (or is limsup not needed here)?


References:

[1] Tibshirani, R. J., & Tibshirani, R. (2009). A bias correction for the minimum error rate in cross-validation. The Annals of Applied Statistics, 822-829.
review_title: Official Blind Review #2

Meta-review: The paper studies theoretical properties of ridge regression, and in particular how to correct for the bias of the estimator. 

The reviewers appreciated the contribution and the fact that you updated the manuscript to make it clearer.

I however advise the authors to think about the best way to maximize impact for the ICLR audience, perhaps by providing relevant examples from the ML literature.
Decision: Accept (Spotlight)