ICML 2025

Random Feature Representation Boosting
Nikita Zozoulenko 1 Thomas Cass * 1 Lukas Gonon * 1 2
Abstract
We introduce Random Feature Representation
Boosting (RFRBoost), a novel method for con-
structing deep residual random feature neural net-
works (RFNNs) using boosting theory. RFRBoost
uses random features at each layer to learn the
functional gradient of the network representation,
enhancing performance while preserving the con-
vex optimization benefits of RFNNs. In the case
of MSE loss, we obtain closed-form solutions to
greedy layer-wise boosting with random features.
For general loss functions, we show that fitting
random feature residual blocks reduces to solving
a quadratically constrained least squares problem.
Through extensive numerical experiments on tab-
ular datasets for both regression and classification,
we show that RFRBoost significantly outperforms
RFNNs and end-to-end trained MLP ResNets in
the small- to medium-scale regime where RFNNs
are typically applied. Moreover, RFRBoost offers
substantial computational benefits, and theoretical
guarantees stemming from boosting theory.
1. Introduction
Random feature neural networks (RFNNs) are single-
hidden-layer neural networks where all model parameters
are randomly initialized or sampled, with only the linear
output layer being trained. This approach presents a compu-
tationally efficient alternative to neural networks trained via
stochastic gradient descent (SGD), avoiding the challenges
associated with non-convex optimization and vanishing/-
exploding gradients. Despite their simplicity, RFNNs and
related random feature models have strong provable gen-
eralization guarantees (Rahimi & Recht, 2008b; Rudi &
Rosasco, 2017; Lanthaler & Nelsen, 2023; Cheng et al.,
2023), and have demonstrated state-of-the-art performance
∗Equal last authors.
1Department of Mathematics, Imperial
College London, UK 2School of Computer Science, University of
St. Gallen, Switzerland. Correspondence to: Nikita Zozoulenko
<n.zoz...