Ridge Regression: Structure, Cross-Validation, and Sketching
We study the following three fundamental problems about ridge regression: what is the structure of the estimator? how to correctly use cross-validation to choose the regularization parameter?and how to accelerate computation without losing too much accuracy?We consider the three problems in a unified large-data linear model.We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise.We study the bias of-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction.We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate.Our results are illustrated by simulations and by analyzing empirical data.
We study the structure of ridge regression in a high-dimensional asymptotic framework, and get insights about cross-validation and sketching.A theoretical study of ridge regression by exploiting a new asymptotic characterisation of the ridge regression estimator.