On the properties of random feature methods
Keywords: benign overfitting
kernel methods
supervised learning
random features

In this thesis, we study the theoretical properties of the random feature methods, which are initially proposed to address the scalability issues of the kernel methods. Although empirically efficient, existing theoretical analysis of random feature methods often provide pessimistic bounds. Hence, this thesis starts with theoretical investigation of the generalization properties of random feature methods. Our first goal is to understand the statistical consistency of random feature estimators in various kernel learning frameworks. By adopting the kernel matrix approximation perspective, we demonstrate that random feature estimators asymptotically converge to the same estimators as the original kernel methods. This consistency property is valid for both kernel ridge regression and distribution regression settings.
Next, we explore the computation and statistical accuracy trade-off of random feature methods. We provide the first unified risk analysis of learning with random feature approximation for both regression and classification under various loss functions (such as Lipschitz continous loss and the 0-1 loss). We express the trade-off through the regularization parameter and the number of effective degrees of freedom. For regression problems, we theoretically demonstrate that random feature methods can provide us computational gain without incurring any prediction accuracy loss. Our study of standard random feature estimators improves the existing bounds on the number of features required to guarantee the minimax risk convergence rate. In addition, we show that a data-dependent sampling scheme of the random features based on leverage scores can further reduces the required number of features. For the classification problem with Lipschitz continuous loss and 0-1loss, we first improve the existing bound on the required number of random features to achieve the minimax optimal learning rate. We then demonstrate that under appropriate assumptions on the data generating distribution, it is possible to obtain a fast learning rate under both standard and leverage score sampling scheme. As ridge leverage scores are expensive to compute, we devise a simple approximation scheme which provably reduces the computational cost without loss of statistical efficiency. Our empirical results illustrate the effectiveness of the proposed scheme relative to the standard random features method.
Finally, we investigate the recent unusual benign overfitting phenomenon: overparameterized models has zero training loss but near-optimal prediction accuracy. Recent studies on benign overfitting reveal that the learning curve of the overparameterized model displays the \textit{double descent} behavior, contradicting classical learning theory. Regarding the random feature model as a simple two-layer neural network with fixed first layer, we examine the conditions under which benign overfitting occurs. We adopt a new view of random feature and show that benign overfitting arises due to the noise which resides in such features (the noise may already be present in the data and propagate to the features or it may be added by the user to the features directly). The feature noise plays an important implicit regularization role in the benign overfitting phenomenon. In addition, we show that the overparameterized models may achieve the optimal learning rate in the minimax sense depending on the feature noise decay rate. Our study also reveals that the excess learning risk curve undergoes a multiple descent, instead of a double descent behavior.