ICML 2025

Adjusting Model Size in Continual Gaussian Processes:
How Big is Big Enough?
Guiomar Pescador-Barrios 1 Sarah Filippi 1 Mark van der Wilk 2
Abstract
Many machine learning models require setting
a parameter that controls their size before train-
ing, e.g. number of neurons in DNNs, or inducing
points in GPs. Increasing capacity typically im-
proves performance until all the information from
the dataset is captured. After this point, compu-
tational cost keeps increasing, without improved
performance. This leads to the question “How big
is big enough?” We investigate this problem for
Gaussian processes (single-layer neural networks)
in continual learning. Here, data becomes avail-
able incrementally, and the final dataset size will
therefore not be known before training, prevent-
ing the use of heuristics for setting a fixed model
size. We develop a method to automatically ad-
just model size while maintaining near-optimal
performance. Our experimental procedure fol-
lows the constraint that any hyperparameters must
be set without seeing dataset properties, and we
show that our method performs well across di-
verse datasets without the need to adjust its hy-
perparameter, showing it requires less tuning than
others.
1. Introduction
Continual learning aims to train models when the data ar-
rives in a stream of batches, without storing data after it has
been processed, and while obtaining predictive performance
that is as high as possible at each point in time (Ring, 1997).
Selecting the size of the model is challenging in this set-
ting, since typical non-continual training procedures do this
by trial-and-error (cross-validation) using repeated training
runs, which is not possible under our requirement of not
storing any data. This is a crucial parameter to set well,
1Department of Mathematics, Imperial College London, UK
2Department of Computer Science, University of Oxford, UK.
Correspondence to: Guiomar Pescador-Barrios <glp22@ic.ac.uk>.
Proceedings of the 42 nd Int...