Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Authors: Guiomar Pescador-Barrios
Sarah Lucie Filippi
Mark van der Wilk
Venue: ICML 2025 spotlightposter (2025)
Keywords: continual learning
gaussian processes
adaptive model capacity
variational inference
inducing points
sparse approximations
bayesian nonparametrics
Primary area: probabilistic_methods
TL;DR: The paper addresses the challenge of determining model size in machine learning, focusing on online Gaussian processes. It presents a method for automatically adjusting model capacity during training while ensuring high performance.

Abstract: Many machine learning models require setting a parameter that controls their size before training, e.g. number of neurons in DNNs, or inducing points in GPs. Increasing capacity typically improves performance until all the information from the dataset is captured. After this point, computational cost keeps increasing, without improved performance. This leads to the question "How big is big enough?" 
We investigate this problem for Gaussian processes (single-layer neural networks) in continual learning. Here, data becomes available incrementally, and the final dataset size will therefore not be known before training, preventing the use of heuristics for setting a fixed model size. We develop a method to automatically adjust model size while maintaining near-optimal performance. 
Our experimental procedure follows the constraint that any hyperparameters must be set without seeing dataset properties, and we show that our method performs well across diverse datasets without the need to adjust its hyperparameter, showing it requires less tuning than others.

Adjusting Model Size in Continual Gaussian Processes:
How Big is Big Enough?
Guiomar Pescador-Barte Barrioss
Sarah Filippi Mark van der Wilk" 2
Abstract
since if the model is too small, predictive performance will suffer. One solution could be to simply make all continual learning models so large, that they will always have enough capacity, regardless what dataset and what amount of data they will be given. However, this "worst-case" strategy is wasteful of computational resources.
Many machine learning models require setting a parameter that controls their size before train- ing, c.g. number of neurons in DNNs, or inducing points in GPs. Increasing capacity typically im- proves performance until all the information from the dataset is captured. After this point, compu- tational cost keeps increasing, without improved performance. This leads to the question "How big is big enough?" We investigate this problem for Gaussian processes (single-layer neural networks) in continual learning. Here, data becomes avail- able incrementally, and the final dataset size will therefore not be known before training, prevent- ing the use of heuristics for setting fixed model size. We develop method to automatically ad- just model size while maintaining near-optimal performance. Our experimental procedure fol- lows the constraint that any hyperparameters must be set without seeing dataset properties, and we show that our method performs well across di- verse datasets without the need to adjust its hy- perparameter, showing it requires less tuning than others.
A elegant solution would be to grow the size of the model adaptively as data arrives, according to the needs of the problem (see Figure 1 for an illustration). For exam- ple, if data were only ever gathered from the same region, there would be diminishing novelty in every new batch. In this case, one would want the model size to stabilise, with growth resuming once data arrives from new regions. In this paper, we investigate principle for determining how to select the size of a model so that it is sufficient to obtain near-optimal performance, while otherwise wasting min- imal amount of computation. In other words, we seek to answer the question of "how big is big enough?" for setting the size of models throughout continual learning.
We investigate this question for Gaussian processes, since excellent continual learning methods exist that perform very similarly to full-batch methods, but which assume fixed model capacity that is large enough. We provide a criterion for determining the number of inducing variables (analo- gous to neurons) that are needed whenever a new batch of data arrives. We show that our method is better able to maintain performance close to optimal full-batch methods, with a slower growth of computational resources compared to other continual methods, which either make models too small and perform poorly, or too large and waste computa- tion. One hyperparameter needs to be tuned to control the balance between computational cost and accuracy. However, single value works similarly across datasets with differ- ent properties, allowing all modelling decisions be made before seeing any data.
1. Introduction
Continual learning aims to train models when the data ar- rives in stream of batches, without storing data after it has been processed, and while obtaining predictive performance that is as high as possible at each point in time (Ring, 1997). Selecting the size of the model is challenging in this set- ting, since typical non-continual training procedures do this by trial-and-error (rosss-validation) using repeated training runs, which is not possible under our requirement of not storing any data. This is crucial parameter to set well,
Our approach benefits from Bayesian nonparametric per- spectives, by separating the specification of the capacity of the model, which should be able to learn from an unbounded amount data, from the specification of the approximation, which determines how much computational effort needed to represent the solution for the current finite dataset. Our approach relies the variational inducing variable approx-
Department of Mathematics, Imperial College London, UK 'Department of Computer Science, University of Oxford, UK. Correspondence to: Guiomar Pescador- Barrios <glp22@ @ uc.ac.uk>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Datasct
Datasct
Dataset 3
0-
2
10
5
10
Inducing Points
30
20
2
10
4
10
8
Batch
Batch
Batch
Exact GP Mean
Past batch
Current hatch
x
Future hatch
VIPS
Figure 1. Three continual learning scenarios with different capacity requirements. Top: Three consecutive batches for 1) a growing input
space, 2) i.i.d. samples from uniform distribution, and 3) narrow-range samples with occasional outliers. Bottom: Number of inducing
points selected using the VIPS algorithm at each batch. When selecting model size with our method, we observed: 1) a linear increase, 2) after initial training,
halt in growth, and low model size until it encounters outliers.
imation of Titsias (2009), which leads to an interpretation that our method automatically adjusts the width single- layer neural network. Our hope is that such perspectives will be useful for the development of adaptive deep neural networks, which would allow models to be trained on large amounts of data beyond the limitations of storage infras- tructure, or learning from streams of data that come directly from interaction with an environment.
remains.
Bayesian continual learning methods have been developed because the posterior given past data becomes the prior for future data, making the posterior a sufficient quantity to estimate (Murphy, 2023, $19.7). For the special case of snnear-in-the-aare parameters regression models, the posterior and updates can be calculated in closed form, leading to continual learning giving exactly the same result as full- batch training. most cases (e.g. for neural networks), the posterior cannot be found exactly, necessitating the afore- mentioned approximations that as the sufficient quantity (Nguyen et al., 2018; Rudner et al., 2022; Chang et al., 2023b).
2. Related Work
approximate posterior
The foundational problem that makes training with gradient descent insufficient for continual learning is catastrophic forgetting, where previously acquired knowledge is lost in favour of recent information (McCloskey Cohen, 1989; Goodfellow et al., 2013). Many solutions have been pro- posed in the literature, such as encouraging weights to be close to values that were well-determined by past data (Kirk- patrick et al., 2017; Schwarz al., 2018), storing subsets or statistics of past data to continue to train the neural network in the future (Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017), and approximate Bayesian methods that balance un- certainty estimates parameters with the strength of the data (Bui et al., 2017; Nguyen et al., 2018; Rudner al., 2022: Chang al., 2023b). Within continual learning, many different settings have been investigated, which vary in dif- ficulty (Farquhar& & Gal, 2018). Across these tasks, the gap in performance to a full-batch training procedure therefore also varies, but despite progress, some gap in performance
Even with a perfect solution to catastrophic forgetting (e.g. the case of senear-in-hhe-paraaee regression mod- cls), continual learning methods face the additional diffi- culty of ensuring that models have sufficient capacity accommodate the continuously arriving information. In continual learning, it is particularly difficult to determine fixed size for the model, since the number of data or tasks are not yet known, and selecting a model that is too small can significantly hurt performance. Growing models with data can fix this problem. For example, Rusu et (2016) extend hidden representations by a fixed amount for each batch. Yoon et al. (2018) argue that extension by fixed amount is wasteful and should instead be data dependent, specifically by copying neurons if their value changes too much, and adding new neurons the training loss doesn't tt
2

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
reach a particular threshold. Kessler et al. (2021) propose to use the Indian Buffet Process more principled way to regularise how fast new weights are added with tasks. Maile et al. (2022); Mitchell et al. (2023) develop similar ideas of automatically adjusting model size to the normal batch-learning setting. We agree with the aims of these papers, but note that they rely on thresholds that need to be tuned to the dataset characteristics, c.g. tolerances on the training error which depend on the amount of intrinsic noise in the dataset. We seek unifying training principle that is broadly applicable to a wide range of datasets, without knowing their properties priori.
leading to wasted computation. Galy-Fajou & Opper (2021) propose OIPS (online inducing point selection), which de- termines M through threshold on the correlation with other inducing points, and is currently the only adaptive method for GPs, although we show that it still relies on tuned hyperparameters to work.
In this work. we propose to select the capacity of the vari- ational approximation by selecting an appropriate toler- ance in the KL gap to the true posterior. This criterion works within the same computational constraints exist- ing GPs continual learning methods, adapts the capacity to the dataset to minimise computational waste while re- taining near-optimal predictive performance. Our method has a single hyperparameter that we keep fixed to a sin- gle value, which produces similar trade-off's between size and performance across benchmark datasets with different characteristics.
Growing model capacity with dataset size was one of the main justifications for research into (Bayesian) non- parametrie models (Ghahramani, 2013; 2015). This ap- proach defines models with infinite capacity, with Bayesian inference naturally using an appropriate finite capacity to make predictions, with finite compute budgets. Gaussian processes (GPs) (Rasmussen& & Williams, 2005) are the most common Bayesian non- non-parametric model for supervised deep neural
3. Background
learning, and are equivalent to infinitely- networks (Neal, 1996; Matthews al., 2018) and linear-in- the-parameters models with an infinite feature space (Mer- cer, 1909). Their infinite capacity allows them to recover functions perfectly in the limit of infinite data (van der Vaart & van Zanten, 2008), and their posterior can be computed in closed form. These are strong principles for providing high-quality solutions to both catastrophic forgetting and ensuring appropriate capacity, and therefore make GPs an excellent model for studying continual learning.
3.1. Sparse Variational Gaussian Processes
We consider the typical regression setting, with training data consisting of N input/output pairs (X,: Yn -= Xn E RD, yn R. We model y/, by passing x, through func- tion followed by additive Gaussian noise yn. = fx,,) + en, En ~ NO0 o²), and take Gaussian process prior on f ~ GP -)) with zero mean, and a kernel & with hyperparameters 0. We wish to find the posterior (for pre- diction) and marginal likelihood (for finding 9). We need approximations, despite these quantities being available in closed form (Rasmussen & Williams, 2005), since they have computational cost of O(N3) that is too high, and require all training data (or statistics greater in size) to be stored, both of which are prohibitive for continual learning. Varia- tional inference requires a lower O M²) computational and O(N memory cost by selecting an approximation from a set of tractable posteriors:
However, developing practical continual learning in GPs is not as straightforward as it is in finite dimensional linear models, because (for N datapoints) the posterior requires 1) an intractable )(N3) computation to compute it exactly, and 2) storing the full training dataset, which breaks the re- quirements of continual learning. Sparse variational induc- ing variable methods solve these problems (Titsias, 2009), by introducing small number of M inducing points that control the capacity of the posterior approximation. In cer- tain settings (e.g. bounded inputs), this approximation is near-exact even when M < N (Burt et al., 2019; 2020). This property has allowed continual learning methods to be developed for GPs that perform very closely to full-batch methods (Bui al., 2017; Maddox et al., 2021: Chang et al., 2023a), provided M is large enough.
x(()) = f J (1)
(1)
= =N (f();; K.kk
k(,,)) - KukKiKKnn -))
(2)
As in neural network models, automatically selecting the capacity M is an open problem, with several proposed so- lutions. Kapoor et al. (2021) acknowledge the need for scaling the capacity with data size, and propose VAR-GP (Variational Autoregressive GP) which adds a fixed number of inducing points for every batch. However, this number may be too small, leading to poor performance, too large,
with [Kuuls] k(z,,,,), [k kK k(, 2,), Z [zm and (u) Nuu m, S). The variational pa- rameters m. S, Z and hyperparameters 0 are selected by maximising the Evidence Lower Bound (ELBO) (Hensman et al., 2013). This simultancously minimises the KL gap [L(((f) (Iy(f 0)] between the approximate and true GP posteriors (Matthews et al., 2016; Matthews, 2017), and maximises an approximation to the marginal likelihood of
3

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
the hyperparameters:
as:
p( (e0n)p(yylDDpse
Nn,
CELBO =
M.yn.y.B8. p(f10m))(ymm11 p(fivo. 0 Ou)p(y.l0o) p(f(0))
KT (a(u) || p(u[0))
(3)
Plyn. (0,)
The variational approximation has the desirable properties (van der Wilk, 2019) of 1) providing measure of discrep- ancy between the finite capacity approximation, and the true infinite capacity model, 2) arbitrarily accurate approx- imations if enough capacity is added (Burt et al., 2019; 2020), and 3) retaining the uncertainty quantification over the infinite number of basis functions. In this work, we will particularly rely on being able to measure the quality of the approximation to help determine how large M should be.
We denote the new variational distribution as qn (n) p(f=b bb, 8n (b)))() where b fZ,) and 0, is the new hyperparameter setting. The KL divergence between the exact and approximate posterior at the current batch is given by:
(L(ga(J) II 0n)]
=log Noln))
p(y.1oo)
ff (nllf) log
p(
0o) df qn (f)(((()))))
3.2. Sparse Gaussian Processes are Equivalent to Single-Layer Neural Networks
The posterior distribution p(fly., 0o) is not available, how- ever by multiplying its approximation (f( in both sides of the fraction inside the log, we obtain:
The equations of the predictor (eq. (2)) show the strong con- nection between Sparse Gaussian processes and neural net- works. The kernel k(: 2;) forms the nonlinearity, with the weights being parameterised as K.l m. For inner product kernels [xx Z) = a (Zx) like the arc-cosine kernel (Cho & Saul, 2009), the connection becomes stronger, with as the input weights. This construction also arises from other combinations of kernels and inter-domain inducing vari- ables (Dutordoir al., 2020; Sun et al., 2020), and has also showed equivalences between deep Gaussian processes and deep neural networks with activations similar to the ReLU (Dutordoir et al., 2021). As consequence, our method for determining the number of inducing variables in a sparse GP, equivalently finds the number neurons needed a single-layer neural network. As such, we hope the ideas presented in this work can inspire similar mechanisms for adaptive neural networks.
KL (f) 11 p(f No. Nn, Fn,
= log ((yn:yo|0.)
6().1o.)
(4)
(n((f) log p( (00))p(( On (nv(()))))))
(apppyyllgooll (f) df +
where $ = - gn (f) log p(fly..0o) df. We cannot com- dol.))
pute & due to its dependence on the exact posterior, so we drop it and use the remaining term as our "online ELBO" training objective, which simplifies as:
E := f annl)) log p(bl0n) on (b)p((a(0.)
df, (5)
3.3. Online Sparse Gaussian Processes
since qo( = (fza a, 0,) (a) and similarly for q (f).
In this work, we use the extension of the sparse variational GP approximation to the continual learning case developed by Bui et al. (2017). Here, we present modified derivation that clarifies 1) how the online ELBO provides an estimate to the full-batch ELBO, and 2) when this approximation is accurate.
Maximising 2 will accurately minimise the KI. to the true
posterior when & is small, which is the case when the old approximation is accurate, i.e. (f( = p(flyo. 9.) for all values of f(x,) (with + = 0 in the case of equality). In our continual learning procedure, we will keep our sequence of approximations accurate by ensuring they all have enough inducing points, which will keep &b small.
We update our posterior and hyperparameter approxima- tions after each batch of new data (X,. While we do not have access to data from older batches (X., yoJ. the parameters specifying the approximate posterior (f) p(Jzala, 0.)g.(a (a) are passed on. We construct this approx- imate posterior as in eq. (1) but with a = f(Z)) and the old hyperparameters 9,- Given the "old" (f),) online sparse GPs construct a "new" approximation qr, (f) of the posterior for all observed data p(fly you Yn, 6n), which can be written
To get our final bound, we perform change of variables for the variational distribution (a) N(aa; ma, S.) to use the likelihood parametrisation (Panos et al., 2018):
Na m., Da) 0, K'a)
1(a)p(a 0,) (a) = T[N(;; ma, Da)N 0, Ky )a da
(6)
"a(a;, 0, D, + KKa))

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
where D, (S ) -1 and m, = 1 m, are the variational parameters, K is the covariance for the prior distribution p(a (,) and ((a) == (a: m,, D.). In this formulation, the variational parameters m., D effectively form a dataset that produce the same posterior as the original dataset, but which we have chosen to be smaller in size, M- < N. This makes our online ELBO from eq. (5)
4. Automatically Adapting Approximation Capacity
We propose a method for adjusting the capacity of the ap- proximation MD to maintain accuracy. We propose to keep inducing points from old batches fixed, and select new induc- ing points from each incoming batch, with their locations set using the "greedy variance" criterion (Fine & Schein- berg. 2001; Foster et al., 2009; Burt et al., 2020). While optimising all inducing points does lead to strictly better approximation, we avoid this for the sake of simplicity, The question remains: To achieve a level of accuracy, "how big is big enough?" To answer this, we consider the online ELBO as a function of the capacity E [(Mb), and propose a threshold after which to stop adding new inducing variables.
-= C= plg + (1) (a)) - -KL (b) || pbb log. (a; 0, Knn + + (7)
which has the nice interpretation of being the normal ELBO,, but with an additional term that includes the approximate likelihood I(a) which summarises the effect of all previous data.
While Σ is all that is needed train the online approxi- mation, it differs from the true marginal likelihood by the term Dogp(yy000) (0)). To approximate it, we could drop the term log.N (a; 0, K + Da) from Σ, since this term also approximates log p(y. (0o), with equality when the posterior is exact, but with no guarantee of being lower bound.
4.1. Online Log Marginal Likelihood (LML) Upper Bound
The problem of selecting a sufficient number of inducing variables is also still open in the batch setting. One possible strategy is to use an upper bound to the marginal likelihood (Titsias, 2014) to bound (L(((f)llp((l)) I u L. and stop adding inducing variables once this is below tolerance a. To extend this strategy to online learning, we begin by deriving an online upper bound, as a counterpart to the online ELBO from eq. (8). We follow the same strategy as Titsias (2014), by considering the highest possible value that our lower bound can attain. While in full-batch inference this is equal to the true LML, in our case this is obtained by keeping the inducing inputs from the previous iteration, and adding each new datapoint to the inducing set:
Although L is a useful training objective for general likeli- hoods, the regression case we consider allows us to analyti- cally find q(b) (Bui et al., 2017) (see App. E), resulting in the lower bound
È = log.N O, K K bf + Egg + A
2 tr D (Kaa- - Qaa)] 1
tr tr(Kr Qi))
(8)
202
where K;p = =Krb Kabl" T Eg- = diag( (o Dal).
[yn U.Sa--ma and
L == -(NN,+ = log.N (ý;; O, Kit + Eg) +Da (9)
1 log
|Sal
Ma log(22)
A
2
K na
Dal
2
K
Kta
with K
Using properties of posi-
1
mm T' s mma + m 7
"D.'' ma.
Kaf Kaa
2
2
tive semi definief matrices, we derive an upper bound (MM to eq. (9):
with Q+ Kn bb Kbr. Qua K Kba All covariances are computed using the new hyperparameters on except for Kn, which is the covariance for the prior distribution p(a 0o). Finally, M₁ = |a| is the number of
log. (y: 0, KFF + Egg)
(N + Ma)
<
log log(2T)
- log (Kpo KTb K bf + Egl
inducing points used at the previous batch. For calculating č in each batch, the computational complexity is O(N, M3 + Mg) and the memory requirements are (MM) where Mp is the total number of inducing points for the current batch.
2
"" KroKkbbK + +++2i/) -1
= Ü(M)
To create fully black-box solution, we still need to specify how select the hyperparameters On the number of in- ducing variables Mr. and the inducing inputs Z,, We will always select 0n by maximising L using L-BFGS. To select the locations Zn. we use the "greedy variance" criterion (Fine & Scheinberg, 2001; Foster et al., 2009; Burt et al., 2020). This leaves only the number of inducing variables Mp to be selected.
where 1 = tr Lr(K#² QFF) and QF = Kip K Kr be and M is the number of inducing points used to calculate the bound (which can be unequal to Mp).
4.2. Approximation Quality Guarantees
Adding inducing points will eventually increase C until it reaches L (Bauer et al., 2016; Matthews, 2017; Burt et al.,
5

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Algorithm I Vegas Inducing Point Selection (VIPS)
Input: X, = [=)
Ma i=1 Zo { } m==1= ir o, on. kernel R:; (0n), threshold parameter 8.
Output: Updated set Z, = Z, U {xmm =m where Z Mp.
Initialise Zn = Z..
while (M) - [(M) - (M) - Lnoise (at,eeff,, (2,0) do
Select x = argmaxxex,, k(x,x, x) (b(x) K K kt()..
Add x to the set of inducing points: Zn = Zn U {x} end while
2020). If we add inducing points until ( S a we can guarantee the following:
to negative the code length that the model requires to encode the dataset. Intuitively, our desire to select an a such that our method captures high proportion (e.g. 95%) of all the information in each batch, so that we can compress within small fraction of the optimal variational code. By itself, this is not well-defined since the code length depends on the quantisation tolerance for our continuous variables. To (11) avoid this, we take an independent random noise code as our baseline, and select a to be within some small fraction of the optimal variational code, relative to the random noise code. We want capture high proportion of all the additional information that our model provides, relative to the noise model, i.e. we want our threshold to be:
Guarantee. Let M be a fixed integer and M, be the number of selected inducing points such that (MM) - ((Mb)= S a.
Assuming that on Go. we have two equivalent bounds:
(f(gn(f) 11 p(flyo. Yn. 0,)] S a v
(10)
(ff 11 g.) Sa
where v -f [f) log C, 8.) of and (f) =
z z- (o f) represents the variational distribution associated with the optimal lower bound L = [(N Ma)) with 2 denoting the marginal likelihood that normalises (.).
N,
a = =5 - Lroisee ; Laoise
Σ n=1
log N (Nnn 6?)
Proof. We cease the addition of points when ((Mb) > ((M -a. a. Given that (M) 2 L" and assuming 6,, 9..
where j and 8² are the average and variance of the obser- vations for up to the current task and 0 is a user-defined hyperparameter. We validate that this approach leads to values of 8 giving consistent behaviour across a wide range of datasets, which allows it to be set in advance without needing much prior knowledge of the dataset characteris- tics.
the rest follows from algebraic manipulation of eq. (4). See App. C for the complete proof.
0
The first bound shows that if L is near zero, the KL to the true posterior is bounded by a. While V depends on the true posterior and therefore cannot be computed, the posterior in the previous iteration was exact, L would be equal zero. The second bound shows that we are guaranteed to have actual approximation qn (f) be within a nats of the best approximation that we can develop, given the limitations of the approximations made in previous iterations.
Calculating this threshold is intractable for large batch sizes N₂. However, if we change our stopping criterion to the more stringent upper bound
ā = ((((MM) Luokse)
(12)
and increase M for calculating i as MD is increased for calculating E. we obtain the same guarantees as before but at a lower computational cost. However, this strategy is only worthwhile for very large batch sizes N,. due to the importance of constant factors in the computational cost. In the common continual learning settings we investigate N,, is small enough to allow computing L.
4.3. Selecting a Threshold
In this final step of our online learning method, we must specify a heuristic for selecting a that does not require know- ing any data in advance, while also working in a uniform way across datasets with different properties. A constant value for a does not work well, since the seale of the LML depends strongly on properties such as dataset size, and ob- servation noise. This means that tolerance of 1 nat (Cover, 1999) may be appropriate for a small dataset, but not for large one.
The procedure for our inducing point selection method is detailed in Algorithm I and we give further details in App. B.I. We name our approach Vegas Inducing Point Selection (VIPS), drawing an analogy to Las Vegas Algo- rithms. These methods guarantee the accuracy of the output, however, their computational time fluctuates for every run (Motwani Raghavan, 1995).
As a principle for selecting the threshold, we take loose inspiration from compression and MDL (Grünwald & Roos, 2019), which takes the view of the ELBO being proportional
6

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
5. Experiments
Map =
Mupo 15
Muta #22
Mitzo = 29
We evaluate the performance of our adaptive inducing point selection method VIPS in a range of streaming scenarios where we assume the total number of observations is un- known. In all cases, the variational distribution, noise and kernel hyperparameters are optimised using the online lower bound (Eq. (8)). We provide further details in App. D.
*
M M-13 10
0.75
M- 20
RMSE
M 30
0.50
VIPS
Exact
0.25
2
Continual learning scenarios pose unique challenges. It is impossible to pre-determine memory allocation due to un- known input space coverage. Additionally, cross-validation for hyperparameter tuning is not feasible as it would require storing all data. Thus, an effective method must 1) have adaptive memory that can grow with the demands of the data, 2) work with hyperparameters that can be set before training. The first part of our experiments demonstrates the benefits of using an adaptive model capacity in continual learning. The second part highlights the necessity of a sin- gle, pre-tunable hyperparameter that works across diverse settings. We benchmark VIPS against two other inducing point selection methods streaming UCI datasets and real-world robotics application.
Batch Number
(a) Accuracy on a synthetic datasct.
Map =13 Map 229
Mapa -42
Mapa
Mapa
Oracle
Time (s)
10
Heuristic
VIPS
10'
10"
5
10
15
20
Batch Number
(b) Training time comparison on "naval" dataset.
Figure 2. (a) Performance comparison of fixed memory ap- proaches (bluc curves with M 10, 20, 30 inducing points) and VIPS, with M (shown at the top) inducing points at cach batch. (b) (log) Time taken to train the online GP model on the "naval" dataset divided into 20 batches with fixed size (oracle: M = 100, heuristic: M- 1000) and VIPS, our adaptive method.
5.1. Model size and data distribution
Figure 1 shows VIPS's 's ability to adapt across three datasets with different characteristics, each divided into ten batches, illustrating how input distribution drives model growth as more data is seen. In the first dataset, each batch intro- duces data from new parts of the input space. Since each batch is equally novel, the model size grows linearly. In the second, the data remain within fixed interval, leading to diminishing novelty in each batch, and a model size that converges to a fixed value. The third dataset mixes narrow- range uniform samples with occasional batches sampled from a heavy-tailed Cauchy distribution. This leads the model size to converge, with sporadic growth when novel data is observed.
the test root mean square error (RMSE) and compare to an exact GP model with access to all current training data as a benchmark. Figure 2a shows that fixed size models M = 10, 20 lose accuracy with more data whereas fixed model M 30 and VIPS match exact GP performance. At this point, we can select the model with M = 30 with the benefit of hindsight. However, at the start of training, we could not have known that this size would have been sufficient at the point of testing. VIPS not only automatically ends up with this optimal size, but also avoids computational waste by using fewer inducing points in earlier batches.
5.2. The impact of model capacity on accuracy and training time
Training cost comparison: We use the "naval" UCI dataset divided into 20 batches, compare the training time of models with fixed and dynamic size. We compare best- case fixed-size model with the smallest number of induc- ing points that would still give near-optimal performance. For "naval", this is M = 100, which would have to be set by an "oracle". We also include a typical heuristic, of M = 1000 (around 1/10th of the total data points), that would ensure sufficient capacity to handle different data patterns and complexities. Finally, we test our adaptive method, VIPS, which adjusts the number of inducing points as needed while achieving near-exact performance for the dataset. As shown in Figure 2b, VIPS results in lower com- putational time by only growing its model size as new data is seen, beating even the oracle method.
Here, we compare a fixed to dynamic model size, with inducing points selected according to the greedy variance criterion (Burt et al., 2020), demonstrate the challenges of a fixed model size. Since the total dataset size and data characteristics are not known at the start of training, select- ing a fixed size will either lead to a model that is too small and underperforms, or a model that is too large and wastes computational resources. Adaptive size solves these issues.
Performance comparison: We train on synthetic dataset divided into four batches, using fixed model size, and our adaptive stopping criterion, VIPS. We test three fixed sizes: M- = 10, 20, and 30 inducing points. We record
7

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Table 1. Mean (std) of inducing points in final hatch across train- ing/test splits at operating point selected to achieve 10% RMSE threshold. Cross (X) indicates failure to meet accuracy threshold and "Max." " denotes reaching maximum capacity limit. Values for OIPS are in italics, as the operating point was selected based on only five datasets, making it less stringent than other methods.
Table 3. Mean (std) of inducing points in final batch across train- ing/test splits at operating point selected to achieve 10% RMSE threshold. Cross (X) indicates failure to meet accuracy threshold and "Max." denotes reaching maximum capacity limit. Values for OIPS are italics, as the operating point was selected based on only five datasets, making it less stringent than other methods.
DATASET
CV
OIPS
VIPS (OURS)
DATASET
CV
OIPS
VIPS (OURS)
CONCRETE
492(54)
102)
234(116)
CONCRETE
492(54)
383(90)
451(77)
SKILLCRAFT
739(34)
174(16)
134(9)
SKILLCRAFT
1091(107)
236(38)
195(18)
KIN8NM
6316(10)
6458(8)
1904(41)
KIN8NM
6469(7)
6523(9)
5065(46)
NAVAL
49(1)
X
57(1)
NAVAL
903(8)
X
509(15)
BLEVATORS
3130(
298(8)
291(4)
ELEVATORS
3 3133(140)
386(47)
(87(24)
BIKE
MAX. 7000
1964 137)
650(17)
BIKE
MAX. 7000
3916(122)
2794(148)
Table 2. Mcan (std) of inducing points in final batch across train- ing/test splits at operating point selected to achieve 10% RMSE threshold in all but one dataset. x Gray shows cases exceeding threshold, with percentages showing % difference to full-batch GP relative to noise model.
Table 4. Mean (std) of inducing points in final batch across train- ing/test splits at operating point selected to achieve 10% NLPD threshold in all but one dataset. X Gray shows cases execeding threshold, with percentages showing % difference to full-batch GP relative to a noise model.
DATASET
CV
OIPS
VIPS (OURS)
DATASET
CV
O1PS
VIPS (OURS)
CONCRETE
184(74)
240(102)
178(95)
CONCRETE
303(78)
383(900
434(79)
SKILLCRAFT
226(22)
174(16)
133(0)
SKILLCRAPT
295(13)
236(38)
177(13)
KIN8NM
5334(27)
6458(8)
1563(39)
KINBNM
5685(26)
(523(99
4281(69)
NAVAL
23(())333%%
(16(1)3155288%
48(3)
NAVAL
227(6)-2406%
X20(11-133.87
X373(199-11.
ELEVATORS
836(27)
298(8)
(81(4)
ELEVATORS
1099(50)
386(47)
438(17)
BIKE
4895(28)
1964(137)
X 4549(14)-10.. 10.8%
RIKE
5563(36)
3916(122)
2296(138)
cause large numbers of inducing points to be used for other datasets. This simulates a realistic setting, where one cannot know what the optimal hyperparameter value is for each dataset beforehand, and penalises methods that are hard to tune, due to large variations in the required hyperparameter values across datasets.
5.3. Continual learning of UCI datasets
We compare our method, VIPS, to two other inducing point selection approaches: Conditional Variance (CV) and OIPS (Galy-Fajoo & Opper, 2021) (see details in App. B). We use six datasets from the UCI repository (Dua & Graff, 2017), simulating continual learning scenario by sorting the data along the first dimension and dividing it into batches. When evaluating inducing point selection methods, we must con- sider the trade-off between accuracy and model size: while selecting more inducing points improves model accuracy, it also increases computational cost.
Table 1 shows the number of inducing points for each method at the operating point that achieves the accuracy threshold within 10% RMSE
full-batch GP. VIPS re- quires fewer inducing points in the majority of datasets. CV often results in larger models, reaching the 7000-point limit Bike. OIPS performs inconsistently, matching VIPS on smaller datasets but oversizing on larger ones. For the Naval dataset, OIPS failed to meet the accuracy constraint across all tested hyperparameter values. While extending the hy- perparameter range might have improved accuracy, it would have led to even larger models. Therefore, we selected an operating point that satisfied our conditions for five out of six datasets, treating Naval an outlier. Table 2 shows the results when this exception is applied to all methods. We observe a similar trend, with VIPS achieving the smallest model sizes. Tables and 4 report analogous results for the threshold within 10% NLPD of a full-batch GP. The same pattern holds. Figure 3 summarises these outcomes across
Our goal is to minimise computational cost, while satis- fying the constraint that we reach a fixed target accuracy threshold (e.g. within 10% RMSE of a full-batch GP). To benchmark our methods, we select a single hyperparameter across datasets, so that our accuracy constraint is always met, while otherwise using the fewest inducing points. We refer to this hyperparameter this operating point, we then measure the number of induc- ing points that the method chooses for each dataset, with a lower result being better. If across datasets, there is a large variation in hyperparameter values needed to achieve the ac- curacy constraint, selecting the most stringent criterion will
the operating point. For
8

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Comparison at 10% RMSE Threshold
Frequency Minimmn
4-
Learning paths 1, 2, 4, sequentially with VIPS.
Dutliers
Outhir Outlier Allowance
CIPS
Comparison at 10% NLPD Threshold
Froquencc Minimumm ModelSios
(b) Learning path 3 with VIPS. M: 134, RMSE: 7.55.
3-
Figure 4. A small robot is used to perform sequential estimation of magnetic field anomalies. The strength of the magnetic field given by 10 pT
Outlies
Outlier
None
to 90 μT, (a) shows the final estimate the magnitude field learned sequentially through different paths. (b) show the outcome of learning a single path continuously with the black dots representing the chosen inducing points.
Outlier Allowance
OPPS
VIP
(b)
Figure 3. Number of datasets where each method achieves minimal model at different outlier allowances. A "win" assigned to the dataset with the smallest model which satisfies the (a) RMSE (b) NLPD thresholds 10%. Counts represent absolute wins. Higher counts indicate better method robustness across datasets.
points and learning hyperparameters without prior tuning in two scenarios: 1) sequentially mapping multiple robot trajectories and 2) processing single path in continuous batches. Figure shows the final magnetic field estimates produced by VIPS both experimental settings. Compara- tive results for CV and OIPS are provided in App. D.4. The results show that VIPS robot moved, allowing for accurate modelling of the path and increasing its capacity dynamically. In contrast, we observe that CV increases its capacity uncontrollably while OIPS provides a less accurate estimate of the magnetic field due choosing fewer inducing points.
outlier allowances, with VIPS more frequently achieving minimal model size. App. D.3 provides additional results under different thresholds.
able to grow its memory the
Our results show that VIPS performs consistently across datasets, allowing single pre-set hyperparameter to be used. In contrast, OIPS and CV require tuning, as no sin- gle hyperparameter works well for all datasets. This a limitation in continual learning, where the whole dataset is unavailable in advance.
6. Discussion
In this work, we propose method to dynamically adjust the number of inducing variables in streaming GP regres- sion. Our method achieves a performance close to full-batch approaches while minimising model size. relies on sin- gle hyperparameter to balance accuracy and complexity, and we demonstrate that our method performs consistently across diverse datasets. This reduces the need for exten- sive hyperparameter tuning and eliminates the requirement to pre-define model size, thereby addressing a significant bottleneck traditional methods. While our current appli- cation focuses on GPs we hope this extendable for the development of adaptive neural networks.
5.4. Continual Learning Magnetic Field
We apply the optimal hyperparameters found the previous section, selected under the 10% RMSE threshold. to the ex- perimental setup from Chang et (2023a) using real-world data from Solin al. (2018), where robot maps magnetic field anomalies in 6m x 6m indoor space. This scenario simulates an expanding domain, where the model needs to continuously learn new areas without uncontrollably increas- ing its capacity. Our goal is to test whether the methods can form representations by progressively spreading inducing

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Acknowledgements
Cover, T. M. Elements of information theory, pp. 14, John Wiley & Sons, 1999.
GPB is supported by EPSRC through the Statistical Ma- chine Learning (StatML) CDT programme, grant no. EP/S023151/I. GPB acknowledges the support of G- Research for her participation in the conference.
Dua, D. and Graff, C. UCI machine learning repository, 2017. URL http://////// /atp:///arcvee.e/eeee ics. ucc.edu/mil /ml
Dutordoir, V., Durrande, N., and Hensman, J. Sparse Gaus- sian processes with spherical harmonic features. In In- ternational Conference Machine Learning, pp. 2793- 2802. PMLR, 2020.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Dutordoir, V., Hensman, J., der Wilk, M., Ek, C. H., Ghahramani, Z., and Durrande, N. Deep neural net- works point estimates for deep Gaussian processes. Advances in Neural Information Processing Systems, 34: 1443-9455, 2021.
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J.. Devin, M., Ghemawat, S., Irving, G., Isard, M., al. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Op- erating Systems Design and Implementation (OSDI), pp- 265-283. USENIX Association, 2016.
Farquhar, S. and Gal, Y. Towards robust evaluations of continual learning. arXiv preprint arXiv: 180500933,, 2018.
Fine, S. and Scheinberg, K. Efficient svm training using low- rank kernel representations. Journal of Machine Learning Research, 2(Dec):243-264, 2001.
Bauer, M., van der Wilk, M., and Rasmussen, C. E. Under- standing probabilistic sparse Gaussian process approx- imations. Advances in neural information processing systems, 29, 2016.
Foster, L., Waagen, A., Aijaz, N., Hurley, M., Luis, A., Rinsky, J., Satyavolu, C., Way, M. J., Gazis, P., and Sri- vastava, A. Stable and efficient Gaussian process caleu- lations. Journal of Machine Learning Research, 10(4), 2009.
Bui, T. D., Nguyen, C., and Turner, R. Streaming sparse Gaussian process approximations. Advances in Neural Information Processing Systems, 30:3299-3307, 2017. ISSN 1049-5258.
Galy-Fajou, T. and Opper, M. Adaptive inducing points selection for Gaussian Processes. In Continual Learning Workshop, July 2021.
Burt, D., Rasmussen, C. E., and Van Der Wilk, M. Rates of convergence for sparse variational Gaussian process re- gression. International Conference on Machine Learn- ing, pp. 862-871. PMLR, 2019.
Ghahramani, Z. Bayesian -anaamrmesssics and the proba- bilistic approach to modelling. Philosophical Transac- tions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 37 (11(984):201 10553, 2013.
Burt, D. R., Rasmussen, C. F.,, and van der Wilk, M. Con- vergence of sparse variational inference in Gaussian pro- cesses regression. The Journal of Machine Learning Research, 21(1):5120-5182, 2020.
Ghahramani. Z. Probabilistic machine learning and artificial intelligence. Nature, (7553)))) 44944522-444444 2015.
Chang, P. E.. Verma, P., John, S., Solin, A., and Khan, M. E. Memory-Bate -Bssed dual Gaussian processes for sequential learning. In International Conference Machine Learn ing, pp. 4035 4054. PMLR, June 2023a.
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic for- getting in gradient-based neural networks. arXiv preprint 131 2013.
Chang, P. G., Durán-Martin, G., Shestopaloff, A., Jones, M., and Murphy, K. P. Low-rank extended kalman filtering for online learning of neural networks from streaming data. pp. 1075-1071, 2023b.
Grünwald, P. and Roos, T. Minimum description length re- visited. International journal of mathematics for industry, 11(01):193001,, 2019.
Hensman, J., Fusi, N., and Lawrence, N. D. Gaussian processes for big data. In Proceedings of the Twenty- Ninth Conference on Uncertainty in Artificial Intelligence, pp. 282-290, 2013.
Cho, Y. and Saul, L. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
10

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Kapoor, S., Karaletsos, T., and Bui, T. D. Variational auto- regressive Gaussian processes for continual learning. In International Conference Machine Learning, pp. 5290- 5300. PMLR, 2021.
Mercer, Xvi. functions of positive and negative type, and their connection the theory of integral equations. Philo- sophical transactions of the royal society of London. Se- ries A, containing papers of a mathematical or physical character, 209(441-44) -458):415 4444,6,, 1909.
Kessler, S., Nguyen, V., Zohren, S., and Roberts, S. Hier- archical indian buffet neural networks for bayesian con- tinual learning. In Uncertainty in artificial intelligence, PMLR, 2021.
Mitchell, R., Mundt, M., and Kersting, K. Self expanding neural networks. arXiv preprint arXiv:2307.0m 04526, 2023,
pp.
Motwani, R. and Raghavan, P. Randomized Algorithms.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinsker Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35113356, 2017.
Cambridge University Press, 1995.
Murphy, K. P. Probabilistic Machine Learning: Advanced Topics, chapter 19.7. MIT Press, 2023.
Neal, R. M. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 1996.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-290) 2017.
Nguyen, C. V., Li, Y., Bui, T. D., and Turner, E. Varia- tional continual learning. In International Conference on Learning Representations, October 2018.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning, Advances in neural information processing systems, 30, 2017.
Panos, A., Dellaportas, P., and Titsias, M. K. Fully scalable
Gaussian processes using subspace inducing inputs. arXiv 2018.
Maddox, W. J., Stanton, S., and Wilson, A. G. Conditioning sparse variational Gaussian processes for online decision- making. Advances in Neural Information Processing Systems, 34:6365-633 6379, 2021.
preprint
Rasmussen, C. E. and Williams, C. K. Gaussian Processes for Machine Learning. MIT Press, November 2005. ISBN 9780262182539.
Maile, K., Rachelson, E., Luga, H., and Wilson, D.G. When, where, and how to add new neurons to anns. In Interna- tional Conference on Automated Machine Learning, pp. 18-1. PMLR, 2022.
Ring. M. B. Child: A first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
Rudner, T. G., Smith, F. B., Feng, Q. Teh, Y. W., and
Matthews, A. G. d. G. Scalable Gaussian process inference using variational methods. PhD thesis, 2017.
Gal, Y. Continual learning via sequential function-space variational inference. In International Conference on Machine Learning, pp. 18871-18833 PMLR, 2022.
Matthews, A. G. dl G., Hensman, J., Turner, R., and Ghahra- mani, Z. On sparse variational methods and the Kullback- Leibler divergence between stochastic processes. In Ar- tificial Intelligence and Statistics, pp. 231-239. PMLR, 2016.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. arXiv preprint arXiv:1606.0461 (1666.04671, 2016.
Matthews, A. G. d. G., van der Wilk, M., Nickson, T., Fujii, K., Boukouvalas, A., León-Villagrá, P., Ghahramani, Z., and Hensman, GPflow: A Gaussian process library us- ing TensorFlow. Journal of Machine Learning Research, 18(40):1-6, apr 2017.
Schwarz, J., Czarnecki, W., Luketina, J., Grabska- Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. Progress & compress: A scalable framework for con- tinual learning. In International conference on machine learning, pp. 4528 4537. PMLR, 2018
Matthews, A. G. d. G., Hron, J., Rowland, M., Turner, R. E., and Ghahramani, Z. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
Solin, A., Kok, M., Wahlstrom, N., Schon, T., and Sarkka, Modeling and interpolation of the ambient magnetic field by Gaussian processes. IEEE Transactions on Robotics, 34(4): 2018. ISSN 1552-3098. doi: 10.1 PRO.2018.220 283002.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
Sun, S., Shi, J., and Grosse, R. B. Neural networks as inter-domain inducing points. In Third Symposium on Advances Approximate Bayesian Inference, 2020.

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Titsias, M. K. Variational learning of inducing variables in sparse Gaussian processes. In Proceedings of the Twelfth International Conference Artificial Intelligence and Statisties (AISTATS), volume 5 of Proceedings of Ma- chine Learning Research, pp. 567-574. PMLR, 2009.
Titsias, M. K. Variational inference for Gaussian and deter- minantal point processes. In Workshop Advances in Variational Inference (NIPS), 2014.
van der Vaart, A. and van Zanten, J. Rates of contraction of posterior distributions based Gaussian process priors. The Annals of Statistics, 36(3):1435-1463, 2008.
van der Wilk, M. Spurse Gaussian process approximations and applications. PhD thesis, 2019.
Yoon, J., Yang, E.. Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR 2018. International Conference on Learning Repre- sentations, ICLR, 2018.
12

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
A. Code
The methods discussed this work, along with the code to reproduce our results, are available online at https: github. com/ omarpe scador/vips All experiments and figures were done in Python using libraries TensorFlow v.2.12 (Abadi et al., 2016) and GPflow v.2.9 (Matthews et al., 2017). Both packages are under Apache License 2.0. The implementation of the online lower bound (see eq. (8)) by Bui al. (2017) is available on GitHub under Apache license 2.0. For the "greedy variance" selection criterion and algorithm, we use the implementation provided in Burt et al. (2020) available under Apache license 2.0.
B. Adaptive Inducing Points Selection Methods
In sections 5.3 and 5.4 we compare our method, VIPS, two other adaptive approaches: Conditional Variance (CV) and OIPS (Galy-Fajou & Opper, 2021). The key distinction between these approaches lies in their selection strategy of inducing points. OIPS uses unified selection process (Algorithm 2), where the choice of both location and number of inducing points is determined by single metric. In particular, each point x; in the training set X,, is evaluated individually, and those that meet the threshold condition are added to the inducing set Z.. In contrast, both CV and VIPS use a decoupled selection process (Algorithm 3), which separates two decisions: where to place inducing points and how many to include. These methods iteratively select the best candidate for the next inducing point from a pool of potential locations, continuing until stopping criterion is met, allowing for more control over the model size. VIPS and CV share the same greedy variance selection criterion but differ in their candidate pools and stopping criteria. Table summarises the properties of these methods, including the fixed memory approach of Bui et al. (2017) for comparison. The next three subsections provide detailed descriptions of each method's s specific implementation.
Algorithm 2 Unified inducing set selection process done at each batch.
Input: New batch of data X., current set inducing points Zold- Output: Updated set Z,
Initialise Zn = Zoid
for X; E X, do
if auditiin__ciitcrinnss; criterion(x:) < threshold then
Z, Zn {x,}
end if end for
Algorithm 3 Decoupled inducing selection process done at each batch.
Input: New batch of data X,, current set inducing points Zold-
Output: Updated set Z,,
Set candidate pool Xpoo based method (see Table 5) and initialise Z, accordingly. while coappiin__rrttrronnn,,,
threshold do selection_criterion223 criterion(Xpoas Z) { [Greedy variance criterion
Xbc = selection,
Z, = Zn U [Xbest}
end while
Table 5. Properties of inducing points selection method for updating an online GP regression model.
METHOD
MODEL SIZE
SELECTION POOL
SELECTION CRITERION
STOPPING CRITERION
BUI AL. (2017)
FIXED
X..
GRADIENT OPTIMISATION
M CONSTANT
OIPS (GALY-FAJOU & OPPER, 2021)
ADAPTIVE
Xx.
max ku(x) S p GREEDY VARIANCE
max ku(x) S P
COND, VARIANCE (CV)
ADAPTIVE
{ Zoo X
tr (K Qa) S 1. Ú C E
VIPS
ADAPTIVE
(X.)
GREEDY VARIANCE
13

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
B.1. Vegas Inducing Point Selection (VIPS) Algorithm
To select the location of our new inducing points use the location selection strategy "greedy variance" proposed in Burt et al. (2020). This strategy iteratively selects points from a set based preference criterion until a stopping condition is met. In particular, it chooses the location of the next inducing point to maximise the marginal variance in the conditional prior p(fzulu), This is equivalent to maximising diag K Qur.. In continual learning, Chang et al. (2023a) use the "greedy variance" criterion by defining {Z., J as the selection pool from which inducing point locations are selected and maintaining a fixed number of inducing points. Similarly, Maddox et al. (2021) extends the "greedy variance" criterion to heteroskedastic Gaussian likelihoods and also uses a fixed memory approach. In our case, we tested the location strategy with our stopping criterion using both (Z., },, and {X,) as candidates pool for the locations of the inducing points. We did not find a substantial difference between the methods and hence opted for the simpler version where we keep the old inducing point locations fixed and choose the new set of inducing points from among the locations in {X.).
Algorithm 4 presents an inducing point selection method using our stopping criterion combined with the location selection strategy "greedy variance". The method takes as input a value for the hyperparameter 0n. In practice, we will set 0, O. to select the number of inducing points; the hyperparameter 0n is inferred by optimising (MMb) once the inducing point locations Zn has been chosen. In Algorithm 1, (MM is used to calculate the stopping criterion. However, in practice, since for the continual learning settings investigate N,, is small enough, we will use ((M N,,) L This value is calculated once at the beginning of the process. The algorithm's complexity depends on the number of inducing points Mp. used to compute the lower bound ((Mb) at each iteration. The computational complexity for calculating C at each batch is MB, + MG), and the memory requirement is (MM)), where Mp represents the total number of inducing points the current batch.
Algorithm 4 Vegas Inducing Point Selection (VIPS)
Input: X,, = / Z.
P. ã, Onr kernel ks,,, -0,,), threshold parameter 6. {I J Ma
Output: Updated set Zn = Zo U {xem² =m'=1' where Zn| Mb.
1M
Initialise Zn = 2..
while ((M) - [(Mb)= S 81L(MM- - Lnoise (ft, (f, σ) do
Select x = argmaxxex, k(x,x) x) (b(x) K (x).
Add x to the set of inducing points: Zn = Zn U {x}. end while
B.2. Conditional Variance
The implementation of the Conditional Variance method presented in Algorithm 5. This method uses the "greedy variance" strategy that iteratively chooses the location of the next inducing point. As stopping criterion, it uses the quantity tr (K+ Q+). ) In this algorithm,, new inducing points are longer added once tr (K+ Qu) falls below chosen tolerance value 17. Although. this approach was mentioned in Burt et al. (2020), this stopping criterion has not yet been tested in the literature. The hyperparameter 7) is determined by the user.
Algorithm 5 Conditional Variance (CV)
Input: X,, = |
No, Zo {zm Ma Oz kernel, k(,, 10n), threshold 17. where [Zn| Mb.
Output: Updated set of inducing points Zn [ 1M" U {xmm } mm==1 Ma-
Initialise location selection pool: Xpao! Zo U Xm-
Initialise Z, arginaxxex,, Xpool k(x,x).
while tr (K# Q+) do
Select x = argmax X k(x, x) (x)x) (x)
Add x to the set of inducing points: Z,, = Zn U (x}. end while
14

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
B.3. Online Inducing Point Selection (OIPS)
Galy-Fajou & Opper (2021) introduced the Online Inducing Points Selection (OIPS) algorithm, which iteratively adds points from X,, to the set of inducing points. The algorithm assesses the impact of each new point the existing inducing set, based on covariance threshold. A point is added if the maximum value of (x) falls below user-defined threshold p. While the original algorithm implicitly assumes unit kernel variance, Algorithm 6 presents our implementation where we scale the threshold by the kernel variance of to p ozp. This modification ensures the threshold comparison remains valid for arbitrary kernel variances.
Algorithm 6 Online Inducing Point Selection (OIPS)
Input: X, {x, J Zo {Zm Im= kernel function k(+, -(,)). kernel hyperparameters On (including variance of)). acceptance threshold () < p < 1. Output: Updated set Zn - Z, U (xmi 1M where Zn. = Mb.
Initialise Zn = Zo. Initialise p = p 0.
for all x, X,, do d maxj (k (x; (n))) Vzj Zn. if then
Add x, to the set of inducing points: Z, = Zn U {xi}. end if end for
15

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
C. Proof of Guarantee
Guarantee. Let M a fixed integer and Mo, be the number of selected inducing points such that (M) - E(Mb) S a.
Assuming that ,n der we have two equivalent bounds:
(f(gn(f) ll p(fly.. Nn, Bo)] S a +
(13)
(f) |1 q. (f)) S
(14)
where = qn(ff) log
2.3 df and (f( = -- (o()p())) represents the variational distribution associated po)) with the optimal lower bound L = L(N, + Ma), with Z denoting the marginal likelihood that normalises 97 (J)-
Proof. We cease the addition of points when ((M) [(Mb) < Since ((M) 2 L", then -L(Mb) -Q(M)
Lr (4) can be bounded as:
p(yn. vo.on)
kL(ga(f)
= log
C + +b
plyo(0.)
(15)
p(yn, Yo (n))
I log
+a- - L- +
ply.10.)
where =
= f ( log O(f df. Let (f) = - (of))() be the variational distribution associated with (0(
L I(N, + Ma). Then, by expanding the true posterior and multiplying by the variational distributions g, (f) on both sides of the fraction inside the log, we obtain:
[ (n( log 9o(f) as p(fly,, 0o)
/ (n(())on log 2o(F)p(y,100) as
p(yalf)p(f10.)
f (n(f)) log go(f)p(y.(0.) (
df
(16)
p(y.lf)p((10.) (J)
/ (n(())og log gafffp(y.10.)
(f)
p(yolf)p(f10 Z Z-1 E-1ggffplyym1 df
(f)
Qn (f) log
df + log 2 log p(y.10,)
p( P(. 0,)
Using the above expansion for 4, eq. (15) becomes.
(L(gx(f) |l p(fly:: yn. 0mn)
$ log p(yn, Yo On)
+ a - - + +
p(y.10.) 0o)
(17)
$ log Po aa- - f (nn((((flo) p(flyn 0o)
(f(
p(yn n. 3o10,)
df + log Z -log
0(y.
ply.
Assuming that 0,, 0.. the above can be simplified to
kL(gn(f) Il p(flyo. Yn. 0n)l sa / (f)
(5) p(flyn. Noi 0n)
df
(18)
Again by multiplying by gn (f) both sides of the fraction inside the log, we obtain:
(i) gn df +
kf)(gn(()) p(flyo. 0o)] S (n(()) log (f) P(fly((f........ 0w)] s KL(g,(F) p(fly. Nn. 0m)] kf)gn(f) || (.)
p(flyn, Yo. 0,) (f)
(19)
kflgx(f) |1 (f) a.
16

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
D. Further Experimental Details and Results
For all experiments and methods, we use the L-BFGS optimiser.
D.1. Model size and data distribution
For the synthetic dataset, we generate random noisy observations from the test function f(x) = sin(2x) + cos(52). We used a Squared Exponential kernel initialised with lengthscale 0.5 and variance 1. The noise variance was initialised to 0.5. For VIPS, we use 0.05.
Dataset 1: We use N 500 observations uniformly distributed from 0 to 10. The data is ordered and divided into ten batches.
Dataset 2: We simulate scenario where small batches of data are received but the data is distributed across the input space. We N 150 observations uniformly distributed from 0 to 10. The data is shuffled and divided into ten batches.
Dataset 3: We simulate a scenario where only outliers are encountered from time to time and the rest of the data is concentrated around a small part of the input space. We use two sets of data: the first set is sampled from a uniform distribution from to with N 1000 and the second set is sampled from Cauchy distribution with mean of μ with N 300. The data is divided into ten batches, where the first few batches only contain observations from the 4 to range, and the Cauchy observations are observed in the latter batches.
Datasct
Datasct 2
Dataset
VWmM
W
10
12
12
10
Excac GP Mean
Data Points
Figure 5. Plot of the three datasets considered in Section D.
D.2. The impact of model capacity accuracy and training time
D.2.1. ACCURACY COMPARISON
For the synthetic dataset, we generate 1000 random noisy observations from the test function f(x) = sin(2x) + cos (5z). We used a Squared Exponential kernel initialised with lengthscale 0.5 and variance 1.0. The noise variance was initialised to 0.5. The performance was measured on test grid of 500 points. For VIPS, use ō = 0.05.
D.2.2. TRAINING COST COMPARISON
This experiment was performed on an Nvidia RTX 6000's GPU on high-performance computing cluster. We used Squared Exponential kernel with hyperparameters initialised to 1. The noise variance was initialised to 0.1. The dataset was divided into 20 batches, and we recorded the time training per batch. For VIPS, we use = 0.05.
D.3. UCI datasets
These experiments were performed on an Nvidia RTX 6000's GPU on high-performance computing cluster. We used a Squared Exponential kernel with hyperparameters initialised to for all datasets. The noise variance was initialised to 0.1. We consider six UCI (Dua & Graff, 2017) datasets of different characteristics: Concrete (1030, 8), Skillcraft (3338, 19), Kin8nm (8192, 8), Naval (1 (11934, 14), Elevators (16599, 18), and Bike (17379, 17). The smaller datasets (< 12000) were
17

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
divided into 20 batches, and the larger (> 12000) into 50 batches. The batches were created by sorting the data points by the first dimension.
Increasing the model size enhances performance until all relevant dataset information is captured; beyond this point, only computational costs increase. Therefore, when comparing inducing point methods, it is essential to consider the size-performance trade-off, rather than focusing solely on performance gains. This trade-off is typically controlled by a hyperparameter of the model. However, in continual learning, traditional cross-validation for tuning is not practical since it would require storing all past data, As a result, a good method needs to use pre-set hyperparameters and still perform well on different datasets. We compare our method, VIPS, with two other adaptive approaches: Conditional Variance (CV) and OIPS. Our goal is to determine if a single hyperparameter for each method can perform consistently well across different datasets. To do this, we evaluate various hyperparameter settings for each method, with ranges 5 E [0.005, 0.2] for VIPS, 5 E [0.005, 1.0] for CV, E [0.80, 0.999) for OIPS.
For each method, we identified hyperparameter values that achieved a particular accuracy threshold (either RMSE or NLPD) within a percentage of 5% or 10% of the full- full-batch GP relative to a noise model. For example, for 5% RMSE, the accuracy threshold is calculated as: RMSExxact + 0.05 IRMSEncise RMMEExact] Among the hyperparameter values meeting these criteria, we select the one that results in the smallest model size. For CV and VIPS, this corresponds to the largest hyperparameter value, while for OIPS, it corresponds to the smallest hyperparameter value (see Table 5 for summary of the methods).
With the optimal hyperparameter selected, all methods achieve the desired performance threshold. Therefore, we can compare the methods based on their model size, where smaller sizes are preferred. As explained in the main part of the paper, we did not find hyperparameter that satisfied the accuracy thresholds for OIPS in the Naval dataset. For this method, we selected the hyperparameter configuration that satisfied our conditions for five out of six datasets, treating Naval as an outlier. This decision was made considering that OIPS already selected larger models with this less stringent hyperparameter (for example 80% of the data points for the Kin8nm dataset) and extending the hyperparameter range would have led to even larger dataset sizes.
Since such dataset-specific limitations are common in practical applications, and to ensure a fair comparison across methods, we evaluated configurations that allowed for either one or two datasets to be outliers for each method. To report performance, we calculated the difference with the full-batch GP metric relative to noise model. For each dataset and batch, we computed the relative RMSE percentage as
R.MSE RMSE 100
RMSE%
|RMSE RMSE
and similarly for the NLPD metric. The test set for each batch consists only of data from the current and previous batches.
Figure 6 shows a summary of the results, presenting the absolute counts for smallest model size. Tables 6, 7, 8 and 9 provide details for each threshold and outlier choice. The results reveal consistent patterns across different accuracy thresholds and metrics. VIPS achieves the smallest model size while maintaining performance within thresholds, particulary when all datasets are considered.
Allowing outlier datasets reduces model sizes across all methods. This comes with performance degradation the particular outlier. We observe that this degradation is often higher for CV and OIPS. The Naval dataset proved particularly challenging for these methods. This is probably due to its almost noiseless nature. While VIPS came close (missing the threshold by 0.48%), none of the methods achieved the 5% target NL.PD threshold within the tested hyperparameter range. Given the dataset's 's near-zero noise characteristics, we chose not to expand the hyperparameter range to avoid excessive model sizes.
The results for OIPS and CV suggest that their hyperparameters may need to be tuned for each dataset. As mentioned, this is not possible in continual learning where data is not available in advance. In contrast, across all configurations, VIPS demonstrates consistent behaviour, minimising model size in the majority of cases and hence reducing the need for dataset-specific tuning. Based on our experimental results, we recommend using 8 = 0.035 for VIPS, which represents the average of successful parameter values when all datasets were considered. This value should provide near-exact performance across different scenarios while minimising model size.
18

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Comparison 5% RMSE Threshold
Comparison 10% RMSE Threshold
Frequency for Minimum Model Size
Frequancy for Mininuum Model Size
Outliers
Outlier
Outliers
Outlier Allowance
Outlier Allowance
OIPS
VIPS
VIPS
(a)
(b)
Comparison 5% NLPD Threshold
Comparison 10% NLPD Threshold
Prequeeccy for Minimum Madel Stre
Proquencc Gor Mininuum Model Sizze
2-
Outliens
Outlier
Outliers
Outlier
Outlier Allowance
Outlier Allowance
VIPS
VIPS
(c)
(d)
Figure 6. Number of datasets where each method achieves minimal model size different outlier allowances. "win" is assigned to the dataset with the smallest model which satisfies the accuracy thresholds. Counts represent absolute wins. Higher counts indicate better method robustness across datasets.
19

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Table Mean (std) of inducing points and RMSE difference with full-hatch GP for the final batch different outlier allowances the selected operating point to achieve a 5% RMSE threshold. x Gray highlights cases that exceed the threshold. and bold text identifies smallest model size. 'Max.' denotes reaching model's maximum capacity limit. For none, one, and two outlers, selected hyperparameters were: VIPS 0.02, 0.065, 0.14, CV 0.005, 0.06, 0.5 and OIPS p 0.955, 0.935.
DATASET
OUTLIERS
VIPS
CV
OIPS
M
RMSE %
M
RMSE %
M
RMSE %
NONE
434(79)
3.67%
492(54)
3.28%
CONCRETE
ONE
317(101)
4.24%
321(71)
4.65%
383(90)
4.89%
TWO
X163(95)
X10.43%
X70(5)
X x13344%
x257(106)
X 88.84%
NONE
177(13)
0.14%
739(34)
0.02%
SKILLCRAFT
ONE
135(1)
0.21%
312(15)
0.04%
236(38)
0.07% 0.02%
TWO
133(0)
0.13%
130(56)
1.45%
185(21)
NONE
4281(69)
0.15%
6316(9)
0.11%
KIN8NM
ONE
2513(54)
1.35%
5685(26)
0.13%
6523(9)
0.12%
TWO
1356(42)
4.97%
5334(27)
0.13%
6499(10)
0.12%
NONE
373(19)
0.40%
49(1)
2.39%
NAVAL
ONE
86(3)
1.37%
X23(7)
X79.08%
X 2
X X407 2207.22%
TWO
43(2)
2.67%
x21(5)
X 336.08%
200(1)
X 447.222
NONE
438(17)
1.19%
3133(140)
0.31%
ELEVATORS
ONE
312(5)
1.33%
1194(42)
0.85%
386(47)
1.19%
TWO
276(4)
1.55%
536(22)
1.43%
305(13)
1.49%
NONE
2296(138)
4.61%
MAX. 7000
0.58%
BIKE
855((19)
8.73%%
5563(36)
1.02%
3916(122) 2438(130)
2.19%
TWo
x498(10)
X 11.24%
3940(41)
1.80%
4.23%
Table Mean (std) of inducing points and RMSE difference with full-batch GP for the final batch at different outlier allowances at the that exceed the threshold, and bold text identifies
selected operating point achieve 10% RMSE threshold. Gray highlights the smallest model size. 'Max.' denotes reaching the model's maximum capacity limit. For none, one, and two outliers, the selected 0.93, 0.845.
hyperparameters VIPS 0.095, 12, 0.2, CV r 0.005, 0.13, 0.6 and OIPS
DATASET
OUTLIERS
VIPS
CV
OIPS
RMSE
RMSE %
M
RMSE
NONE
234(116)
7.66%
492(54)
3.28%
CONCRETE
ONE
178(95)
9.81%
184(74)
9.96%
240(102)
9.03%
TWO
x111(37)
X 12.93%
x3(2)
126.811%
108(66)
X 200.7%%
NONE
134(0)
0.12%
739(34)
-0.02%
SKILLCRAFT
ONE
133(0)
0.13%
226(22)
0.20%
174(16)
0.04%
TWO
133(0)
0.12%
17(0)
7.93%
138(1)
0.06%
NONE
1904(51)
2.55%
6316(9)
0.11%
KIN8NM
ONE
1563(39)
3.77%
5334(27)
0.13%
6458(8)
0.11%
TWO
949(25)
8.17%
3408(32)
0.19%
5053(93)
0.14%
NONE
57(1)
2.25%
49(1)
2.39%
NAVAL
ONE
48(3)
2.57%
X
323.90%%
(16(1)
315.28%
38(1)
5.07%
X x
553.119 %
13(1)
407.6%
NONE
291(4)
1.41%
3133(140)
0.31%
ELEVATORS
ONE
281(4)
1.51%
836(27)
1.11%
298(8)
1.49%
TWO
267(2)
1.57%
287(18)
2.52%
269(1)
1.59%
NONE
650(17)
9.98%
MAX. 7000
0.58%
BIKE
ONE
549(14)
X 10.82%
4895(28)
1.29%
1964(137)
5.18%
TWO
X 40007)
X 12.50%
2582(66)
4.22%
594(11)
10.00%
20

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Table Mean (std) of inducing points and NLPD % difference with full-batch GP for the final batch different outlier allowances the selected operating point to achieve a 5% NLPD threshold. X Gray highlights cases that exceed the threshold, and bold text identifies the smallest model size. 'Max.' denotes reaching the model's maximum capacity limit. For one, and two outliers, the selected hyperparameters
were: VIPS
0.005, 0.015, CV f = 0.025, 0.100 and OIPS p 0.975, 0.960.
DATASET
OUTLIERS
VIPS
CV
OIPS
M
NLPD %
M
NLPD %
M
NLPD %
ONE
514(74)
2.27%
417(63)
4.28%
490(73)
4.72%
CONCRETE
TWO
451(77)
4.43%
(227(84)
115.4%
X410(88)
X 8.34%
ONE
291(34)
0.60%
439(29)
0.60%
488(119)
0.33%
SKILLCRAFT
TWO
195(18)
0.78%
260(16)
0.81% %
332(82)
0.51% %
ONE
5640(73)
0.85%
6194(12)
0.88%
6547(8)
0.88%
KIN8NM
TWO
5065(46)
0.94%
5334(27)
0.90%
6539(8)
0.88%
ONE
X X1389(18)
X 5.48%
X27(5)
X X4654.12%
X26(I)
115.29%
NAVAL
TWO
X 509(15)
X9.99%
x28(4)
X 88146.44%
220(1)
133.87%
740(24)
1.74%
1814(88)
1.23%
1869(257)
1.34%
BLEVATORS
TWO
(24(
2.02%
916(21)
1.91%
642(133)
2.01% %
ONE
5064(334)
3.99%
MAX. 7000
2.13%
6794(79)
2.55%
BIKE
TWO
x2794(148)
88.88%
4895(28)
3.54%
5131(64)
4.05%
Table Mean (std) inducing points and NLPD difference with full-batch GP the final batch at different outlier allowances the selected operating point achieve 10% NLPD threshold. X Gray highlights cases that exceed the threshold, and bold text identifies the smallest model size. 'Max.' denotes reaching the model's maximum capacity limit. For one, and two outlicrs, the selected hyperparameters were: VIPS 0.015, 0.02, 0.05, CV 0.001, 0.07, 0.5 and OIPS p 0.95, 0.935.
DATASET
OUTLIERS
VIPS
CV
OIPS
M
NLPD %
M
NLPD
M
NLPD
NONE
451(77)
4.43%
492(54)
2.32%
CONCRETE
ONE
434(79)
5.68%
303(78)
9.64%
383(90)
9.56%
TWO
349(86)
9.04%
x3(2)
X 111..11%
x247(101)
X 15.73%
NONE
195(18)
0.78%
1091(107)
-0.34%
SKILLCRAFT
177(13)
0.78%
295(13)
0.71%
236(38)
0.57%
TWO
141(4)
1.10%
X x54(45)
X x8.03%
185(21)
0.68% %
NONE
5065(46)
0.94%
6469(7)
0.87%
KIN8NM
ONE
4281(69)
0.34%
5685(26)
0.91%
6523(9)
0.88%
TwO
2953(71)
1.89%
5334(27)
0.90%
6499(10)
0.88%
NONE
509(15)
9.99%
903(8)
6.17%
NAVAL
ONE
X373(19)
11.922
X27(6)
224058799 %
X20(1)
X 133.87%
Two
122(6)
22.955%
x22(7)
6012.82%
20(1)
X 133.87%
NONE
487(24)
2.02%
3133(140)
0.82%
ELEVATORS
ONE
438(17)
2.11%
1099(50)
1.73%
386(47)
2.68% % 4.84%
Two
332(8)
2.54%
352(29)
3.76%
305(13)
NONE
2794(148)
8.18%
MAX. 7000
2.04%
BIKE
ONE
2296(138)
9.92%
5563(36)
2.95%
3916(122)
5.50%
Two
X 11037(23)
17.166%
2582(66)
9.61%
2438(130)
9.66% %
21

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
D.4. Magnetic anomalies
The data used in this experiment is obtained from Solin al. (2018) and is available on GitHub. The objective of this task to detect local anomalies in the Earth's s magnetic field online, caused by the presence of bedrock and magnetic materials in indoor building structures. For this purpose, small robot with 3-axis magnetometer moves around an indoor space of approximately 6 meters by 6 meters and measures the magnetic field strength. Out of the 9 available trajectories, we use trajectories 1, 2. 3, 4, and 5 (with n 8875, 9105, 9404, 7332, 8313, respectively) for the experiments. Specifically, we use trajectories 1, 2, 4 and for the first experiment and trajectory 3 for the second.
We use the experimental setup proposed in Chang et al. (2023a). The proposed model applies GP prior to magnetic field strength, given by GP (0 Mat (x,x') (in nT ), where the kernel consist of constant kernel and Matérn-v/2 kemnel. The model assumes the spatial domain is affected by Gaussian noise with variance of The initial variance for the constant kernel is set to 500, and the Gaussian likelihood is initialised with noise variance of 0.1.
Our aim for this experiment is to test the optimal hyperparameters identified in the previous section for the 10% RMSE threshold for each adaptive method in a real-world setting. The setting simulates an ever-expanding domain, where the robot is not confined to a predefined area. In this context, the model continuously learns new parts of the space. Therefore, method that works will need sufficiently expand the model's size to accommodate new data without letting it grow uncontrollably.
In the first experiment, aim to sequentially learn the paths taken by the robot using trajectories 2, and 5, i.e. an entire path will correspond to one batch. We investigate whether the method can adapt to changes in the environment and adjust the number of inducing points accordingly. During this process, we concurrently learn the hyperparameters où o², L, and of. As a test set, we use trajectory 3. Figures 7, 9, 11 show the temporally updating field estimate over batches alongside corresponding path travelled in each batch.
In the second experiment, we focus on the streaming learning of trajectory 3. The trajectory is split into 20 batches. We compare the number of inducing points selected and the estimate obtained by the three methods, Conditional Variance (CV), OIPS and VIPS (ours). Detailed learning of the path for each method is shown in figures 8, 10 and 12. As test set, use trajectories 1, 2, 4, and 5. We observed how Conditional Variance chooses an excessive number of inducing points, indicating that its hyperparameter needs tuning, which is impractical in the continual learning setting. OIPS chooses the least inducing points; as seen the figure, the number of inducing points is more concentrated at the beginning of the path and becomes sparse towards the end. When compared to the learning estimates derived from the paths in the previous experiment, we see that both estimates differ significantly, suggesting that OIPS fails to adequately add enough capacity to capture the changes in the environment. VIPS provides the middle ground, selecting moderate number of inducing points that effectively balance accuracy and memory size. This choice allows VIPS to maintain robust estimate of the magnetic field obtained when compared to learning by paths without excessive computational overhead.
In the last two sections, when compared to both alternative approaches, VIPS achieved the best trade-off between performance and model size without requiring hyperparameter tuning, making it the preferred method among the three.
After path #1
After path #2
After path #4
After path #5
Figure 7. VIPS (Ours). A small robot with wheels is used to perform sequential estimation of magnetic field anomalies. We show the estimate of the magnitude field learned sequentially after travelling the path shown in dotted line. The degree of transparency represents the marginal variance.
22

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Batch 1, M =
Batch 5, M 32.
Batch 10,
Batch 15,
Batch 20,
Figure& 8. VIPS (Ours). small robot with wheels is used perform sequential estimation of magnetic field anomalies. Data is collected continuously as the robot moves along the path. The inducing points are represented as black dots and the line represents the travelled part of the path. We indicate the batch number and number of inducing points (M). Final RMSE 7.55.
After path
After path
After path #4
After path
Figure Conditional Variance. small robot with wheels used perform sequential estimation of magnetic field anomalies. We show the estimate of the magnitude field learned sequentially after travelling the path shown a dotted line. The degree of transparency represents the marginal variance.
Batch 1, M, = 12.
Batch 5, M 597.
Batch 10, M = 2206.
Batch 15, M = 4071.
Batch 20, M = 5785.
Figure Conditional Variance. small robot with wheels used perform sequential estimation magnetic field anomalics. Data collected continuously as the robot moves along the path. The inducing points represented black dots and the line represents the travelled part of the path. We indicate the batch number and number inducing points (M). Final RMSE = 10.66.
After path #1
After path #2
After path #4
After path #5
Figure 11. OIPS. small robot with wheels is used to perform sequential estimation of magnetic field anomalies. We show estimate of the magnitude field learned sequentially after travelling the path shown in dotted line. The degree of transparency represents the marginal variance.
23

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Batch I, . = 10.
Batch 5, M = 27.
Batch 10, = 43.
Batch 15, M = 59.
Batch 20, M = 76.
Figure OIPS. A small robot with wheels used to perform sequential estimation magnetic field anomalies. Data is collected continuously as the robot moves along the path. The inducing points represented as black dots and the line represents the travelled part the path. We indicate the batch number and number inducing points (M), Final RMSE = 9.42.
D.5. Model size and Data Distribution with Larger Data
We reproduce the three continual learning scenarios from App. D.1, increasing the dataset size to 500 data points and the input dimensionality to three, and compare how the three methods (VIPS, OIPS and Conditional Variance) scale in this setting. This setup aims to simulate practical scenarios where an exact GP would be infeasible, and sequential approach is required instead. We use Squared Exponential kernel initialised with lengthscale 1.0 and variance 1.0. The noise variance set to 0.2. We evaluate all methods at the operating points corresponding to the 10% RMSE thresholds determined from the UCI experiments (App. D.3). The data is divided into 50 batches of 1000 points each.
Dataset 1:
Data points are sampled uniformly from [0, 20 with batches ordered such that new regions of the input space discovered gradually. Model size is expected to grow linearly more data is observed.
Data points again sampled uniformly from [0, 20]8, but batches shuffled at random. This represents setting where the entire input space is observed from the start. As result, the novelty of each batch decreases over time, and the model size is expected to plateau.
Dataset 2:
Most data is concentrated in a narrow region, with occasional outliers. Ninety per cent of the points are drawn from a truncated Cauchy distribution on [5, 15]s centred at 10, and the remainder uniformly from [0, 40) This mimics a scenario where data dense but includes some outliers. Therefore, the model size should remain small until the region with the outliers observed.
Dataset 3:
Figure 13 shows how the number of inducing points evolves more data is observed for each method across the three datasets. VIPS adapts across all settings, increasing model size only when needed. OIPS shows similar trends but typically adds up to twice as many inducing points as VIPS. CV grows rapidly even when unnecessary (e.g. Dataset 2). Note that for CV we impose a cap of 5000 inducing points.
The datasets were designed to be well-approximated with relatively few inducing points, and all methods attain near-exact accuracy. These results show that even larger scale, VIPS maintains a compact model, using fewer than a thousand inducing points in all cases. contrast, the results for CV and OIPS suggest that these methods may require hyperparameter tuning to control model growth in larger settings.
Dataset
Dataset
Datasel
5000
Nombher of Induciing
#000
4000
4000
3000
3000
9003
1000
1000
1000
10000
20010)
30000
80000
50000
10000
200000
30000
40000
50000
10000)
20000
20000
40000
50000
Cumulative Dataset Size
Cumulative Datasct Size
Cumulative Dataset Size
Conditional Vaiasce
CIPS
VIPS
Figure 13. Number of inducing points used by each method as function of cumulative data size, across the three synthetic datasets. VIPS consistently uses fewer inducing points and only increases model size needed. OIPS follows similar trend but typically uses twice many inducing points as VTPS. CV grows unnecessarily and reaches the imposed cap of 5000 points.
24

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
E. Derivation the optimal form of variational distribution
The derivations follow Bui et al. (2017). We start by rewriting the online lower bound,
/ afanlf) [og p(al0.)g.(b) p(ble.)9.
غ =
(20)
/
- / df log 9la
We can find the optimal Jn by setting the derivative of E with respect to gn (b) to zero,
dā dg(b)
/ dfo bb) logp(yn (f) log
p (a Gold 9n (b) p (b Onew (a(a)
0,
(21)
this gives us,
Topt (b) x p(b(On) exp (ʃ |daapia b, 0n) log go(a)
+ dfp(( | b, 1o) (1) 1
p Ooia
Let q(a) .V (a; m. Sa) and (a Oou) N (a; O, a. and denoting D, (S K==²) Qu Ko.KbKbr
and Qaa Kab Kba, the exponents in the optimal gopt (b) can be simplified as follows:
Ea = dap(a b, 0m) log q(a)
(a
0.)
/ daN (a: Kab Kab K Qa)
Sa.
log
(a
ma) SSa (a (mmma)
ma) + T TK-la
2
K
na
[Sal
2 daN (a; Kank Qa)
log
ma T S
ma + a (K Sm)) a a 2a 1 1 ma
K
[Sa]
2 daN (a; Kabk Qa)
log
m a m, a D a 2a 2aTS S ma
K'
daW (a: Kab b. Qa) log A(D. "maaa a, Da) + An
(22)
Nog (DaS mai Kark, b, Da) D (Kaa Qaa)l + Aa
E2 [ dfp(( b,0,) logply...
[ dEN (P:KK, Qr) log.N Ny:.W(y:f,o'))
1
Nogg (y: Krr b. 0²
tr (Krr - Q
202
Sa
24 - -log
m S S m m s + Ma log(2mr,
K'
D,
where A. is constant term with respesct b and M, is the dimension of a. The optimal form of gn is then given by:
(b) x p(blO.)W (y. Kr Kobbb Eg)
(23)
where
Kno
0
K
FEg
(24)
D
K.b
D
After normalisation, we have
P(b]o.) (s, K K bb
b. Egg)
(25)
ddbp(b(0,)) W (y. K K Eg)
25

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Substituting the above results into the lower bound, have
Kt = log (8: O, Kt, Kbt + +Mig) + A. 2
(Kaa tr Daa Qua)) tr tr( - Qu)..
(26)
2x²
E.1. Derivation of L
Recall that
Kt + Σg) + Dn. L = N (9: 0
(27)
K
Ktn
with it
Kaf
Kaa
The first term can be lower bounded using Jensen's inequality as,
log.N (5; 0, Kpp + Eg)
(28)
2 log (5; 0, Ki K Egg) tr (Igg (K Ki KibK
where Kib KibK simplified as follows:
Di
of is the Nyström approximation of K fp. The trace will be small when b = (f(xn, a)} and can be
tr
(Kpt
Ki
[[
o
K
Kta Kaa
K Kr
Kb 1)
ir
(29)
D=
Kat
K KabKnb Kbr
Kab Kba Kba
= tr (D, (Kan Kab Kba)) Kba)) + σ tr(Kæ K Kn
which recovers the expression for 2-A.. --A..
F. Online Upper Bound Implementation
In this section, we provide efficient forms for practical implementation of the online upper bound t. As the second term constant we focus on the first term,
ú (N + Ma) log(2+t=
1 log Kip K Kt + +gg
1 "" (Kio KibKK a Eg)
- ".
(30)
2
This term is an upper bound for the first term C log.N (y: 0, Kit + Σg) + Da.
F.. Determinant term
Letting Kbb = LLL and using the matrix determinant lemma, we can rewrite the determinant term as
log Kgp K
1--KK K L. K of + Eg = log Egg + log + L, K bi 2i
(31)
=N log of log Dal log L, 2 K
I|
Let D I+ L, Kb KIOLD" Note that,
1 a
K Kab
Kpp Kpp
Krt
Kbn
D
KorKnD + KbaD, Kab
(32)
o
KorKnD + KKba SKab Kab Kbak Biak'sKK Kab.
26

---

Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Therefore,
+ Lbb 9
L + L K K - LbbLo
(33)
F22 Quadratic term Given the quadratic term,
1 " (Kip bb bi + + Egg) -1 g.
Letting Dg 11 + Eg and by Woodbury's formula, we obtain:
+10)= = -Gg-- K Kb
Ki Eig "Ky"Kibbo b1b²)²²)
-1
Lb
L 5-1
bi
We have,
y 8
o3 + y'y y + (DS ma) T (Da +t) (D.Sa (ma)
(34)
Letting II Kr 5-1 K Ip= T where,
bf 'g
2 Ktb
kv Kab
offt
0
Knb
o3 +1 KorKn Kba (D,
'Kab
Kab
(Da + +1) -1
Kab
and letting è = Kpp 2
offt KEry + Knn (Da -1) " mm). we obtain,
8'E ' Kop -
K (I+LI'K,n) S- L K -T)
L, K - - c''LT D
(35)
Putting this back into the upper bound:
0h (N Ma)
log(2 2 log oz log [Dal
log
[D
5 " y - L' "1''""D1- L,- (36)
2
The upper bound for L is therefore
a = (N + Ma)
1 " (K K K bf + I1 [11))) -1 y + An
log(2rr) log K
ib
K
bî + Egl
2
2
1
log (2gT))o)))
log [D]
(D.S 'ma'maa
(Da +t+ (DSa'mm. (37)
2
2 +
1
c
Lp
-T
''D'L
c
1
log
Sa
1
m
T
ma
m I T ma-
2
K
2
2
27


Reviews:
[{"review_id": "h2Hpoggz3v", "reviewer": "Reviewer_9EX8", "summary": "The submission proposes a method that dynamically adjusts the model size (i.e., the number of inducing points in a sparse Gaussian process) while maintaining near-optimal performance in a continual learning setting, where data is presented as a stream and data storage is not allowed. The proposed method requires only a single hyper-parameter (the threshold) to balance accuracy and complexity.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging feedback and for recognising the relevance of our work. We appreciate your positive comments on the clarity of our writing and experimental design. \n\n**Q1: Optimizing the inducing points, rather than selecting them from the training data, may lead to better solutions. Is it straightforward to implement this in the current model formulation?**\n\nYes, it is straightforward to implement with our formulation. As our method builds on the variational framework of Bui et al. (2017), which itself builds on Titsias (2009), the inducing point locations can be jointly optimised with the other hyperparameters of the model using the streaming variational lower bound, just as in the batch setting. In this work, we opt to select inducing points locations from the data for simplicity. That said, our framework is compatible with gradient-based optimisation of inducing locations if desired.\n\n**Q2: Related to the item Other Strengths and Weaknesses, is it straightforward to extend the current base prediction model to a (sparse) deep (multi-layered) Gaussian model?**\n\nThe extension is not straightforward. In the deep case, there is no analogous bound to Titsias' collapsed bound, which the base model in Bui et al. (2017) builds upon. However, we are actively working on extending VIPS to deep models. \n\n**The proposed approach is not a generic model adaptation method but is limited to a specific model (sparse Gaussian Processes). Additionally, GPs and sparse GPs are limited in their ability to model highly nonlinear or non-Gaussian data distributions.**\n\nWe appreciate your observation. While our current focus is on sparse GPs, we view VIPS as a first step towards adaptive size in more general settings. In our work, we provide a principled criterion that adjusts model size to maintain accuracy with incoming data. In particular, since GPs and NNs share structural similarities, we hope that the ideas introduced here can inspire similar mechanisms for adaptive neural architectures. We are actively exploring such extensions in ongoing work.", "claims_and_evidence": "Main claim: The submission develops a criterion for model size adjustment based on existing variational bounds and demonstrates its performance by comparing it to existing inducing point selection methods using USI datasets and robot data.", "methods_and_evaluation_criteria": "The evaluation metric used is the number of inducing points learned by the methods, which makes sense given that the target performance is consistent across all methods. Therefore, a smaller number of inducing points is preferred. However, it seems that the UCI datasets used in the experiments do not include very large-scale data, and interpreting the results based solely on the real-world (robot) data may be challenging for those who are not familiar with this specific dataset or task.", "theoretical_claims": "I did not verify each proof individually, but the methods used to derive the bounds appear technically sound.", "experimental_designs_or_analyses": "The experiments appear to be conducted correctly, and the text is well-written, making it easy to understand the experimental results and the interpretation the authors intended to convey,", "relation_to_broader_scientific_literature": "The idea of adaptive model size presented in the submission is relevant to current trends in the machine learning community, as continual learning (dealing with streaming data) has gained significant attention.", "essential_references_not_discussed": "It appears that the submission covers the essential references related to the research topic.", "other_strengths_and_weaknesses": "Strengths\nThe idea of dynamically adjusting the model size in continual learning appears novel, and integrating this approach within Gaussian Processes is promising\nThe use of a single hyper-parameter may reduce the need for extensive fine-tuning.\n\nWeaknesses\nThe proposed approach is not a generic model adaptation method but is limited to a specific model (sparse Gaussian Processes). Additionally, GPs and sparse GPs are limited in their ability to model highly nonlinear or non-Gaussian data distributions.", "other_comments_or_suggestions": "It seems that the reference for Conditional Variance (CV), used as a baseline, is missing in the text.", "questions_for_authors": "Q1: Optimizing the inducing points, rather than selecting them from the training data, may lead to better solutions. Is it straightforward to implement this in the current model formulation?\n\nQ2: Related to the item Other Strengths and Weaknesses, is it straightforward to extend the current base prediction model to a (sparse) deep (multi-layered) Gaussian model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FhXs4jHN7V", "reviewer": "Reviewer_YXRf", "summary": "In a streaming data setting, where access to previously observed batches of data is not available, one cannot use Gaussian process methods with non-degenerate (i.e., full-rank) kernels. A very popular approach is to approximate the full Gaussian process with a variational approximation, in which the posterior is computed using a fixed-size set of inducing points. Nonetheless, in the streaming setting, a poor choice of the number of inducing points can lead to either poor performance or wasted computational resources.\n\nThe authors propose a new criterion based on a previously known online version of the variational ELBO. Specifically, when the gap between the true posterior and the online approximate posterior becomes sufficiently large, the model capacity is increased to alleviate this gap. The correctness of this criterion depends on the quality of the approximation at the previous step.\n\nThis proposal is evaluated using synthetic data, UCI datasets in a streaming setting, and a real-world dataset collected in a streaming fashion.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and your suggestions. We appreciate your positive feedback. \n\n**Connection of our work with NNs**\n\nAs we note in response to other reviewers, we view VIPS as a first step towards adaptive size in more general settings. Since GPs and NNs share structural similarities, the aim of Section 3.2 was to introduce these ideas as a foundation for future work on adaptive neural architectures. We are actively exploring such extensions in ongoing work. \n\n**It is unclear to me how this work could interact with alternative approaches, such as the expanding memory approach of Chang et al. (2023).**\n\nWe believe our approach is complementary to Chang et al. (2023). Both approaches aim to retain a selected set of points (whether data or inducing points) to ensure a good approximation. Chang et al. (2023) achieve this by expanding their memory sequentially, adding a fixed number of data points at each step. In contrast, our method dynamically adjusts the number of inducing points to maintain a desired approximation quality. One promising direction could be to adapt our criterion to guide memory growth in their framework; this is an extension we are currently considering.\n\n\nWe will also take your other suggestions into account for the camera-ready version.", "claims_and_evidence": "The authors' claims are validated by experimental results. They assert that their method achieves results close to the exact non-streaming GP (Sec. 5.2) and produces models with smaller footprints (Sec. 5.3). The results are easy to understand and well explained.", "methods_and_evaluation_criteria": "The evaluation criteria consider real-life constraints of streaming datasets, assess different hyperparameter combinations, and compare only those that meet a specific RMSE or NLPD threshold. This evaluation should fairly assess the methods while accounting for their predictive distributions and the problem's constraints.", "theoretical_claims": "I have briefly checked the correctness of the theoretical claims (Sec. 4.1 and 4.2), and they appear valid given the assumptions; specifically, that the quality of the previous iteration affects the global quality of the current iteration.", "experimental_designs_or_analyses": "Yes, as discussed in the evaluation criteria, their analysis is sound and follows best practices.", "relation_to_broader_scientific_literature": "This paper fits into the field of sparse online Gaussian processes, a domain where the contributions by Csató and Opper (2001) are well known in the literature on sparse GPs. Increasing the capacity of the sparse model on the fly is an important problem, and the solution presented by the authors is, to the best of my knowledge, the first that addresses this by upper-bounding the gap, in terms of KL divergence, between the approximate model and the full GP model. It is unclear to me how this work could interact with alternative approaches, such as the expanding memory approach of Chang et al. (2023).", "essential_references_not_discussed": "Given the paper's focus on methods that do not use memory or replay buffers, the references discussed seem appropriate to me.", "other_strengths_and_weaknesses": "The focus on reducing the number of hyperparameters and using already established criteria for inducing point selection is a strength of this paper, as hyperparameters and their selection can be a significant limitation on the applicability of Gaussian process methods.", "other_comments_or_suggestions": "The authors forgot a stray quotation mark in their Impact Statement.\n\nIn the authors’ introduction to Gaussian processes, I would suggest stating that infinite-width neural networks are a well-known subclass of GPs rather than implying that GPs are either that or something else.\n\nWhile the discussion in Section 3.2 is quite interesting, none of the authors' experiments use inner-product kernels or further explore the connection with neural networks.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tI0NttHAac", "reviewer": "Reviewer_y4Kp", "summary": "The paper introduces a new criterion for determining the number of inducing variables in a context of continual learning with single-output GP regression models. The general idea is to automatically adjust model size while maintaining near-optimal performance, but without the need of seeing future data points.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and thorough review. We appreciate your perspective on the significance of our work.\n\n**[Q] Focus on regression problems.**\n\nAs you noted, the main reason is the theoretical guarantees and the bound from Titsias (2014), which was derived for GP regression with Gaussian likelihoods. In this case, the variational KL divergence can be made arbitrarily small by increasing the number of inducing points. This no longer holds in classification, where the likelihood is non-Gaussian\n\n**[Q] Validity of Eq. (5) without $\\Phi$ term**\n\nEq. (5) is no longer a bound on the marginal likelihood of the full dataset, so the theoretical rigour of the original bound is, strictly speaking, lost when omitting the $\\Phi$ term. However, it remains a valid lower bound for the new data, given the posterior carried from previous batches. As we add more inducing points, $\\Phi$ approaches zero, and in the limit, Eq. (5) recovers the full bound.\n\nThis is where our argument becomes empirical: we show that by adding enough inducing points, we can keep $\\Phi$ small enough for Eq. (5) to behave as a proper ELBO during training, and that this is sufficient to maintain performance on the full dataset.\n\n**[Comment] The last term of Eq. (7) could be ignored from an optimization point of view.**\n\nFrom an optimisation point of view, yes, this is right. \n\n**[Q] Must $\\mathcal{U}$ vary with $M$ in the same way as $\\mathcal{L}$?**\n\nWe are free to use a different number of inducing points to calculate $\\widehat{\\mathcal{U}}$ compared to what we use in the lower bound. Using more inducing points for the upper bound leads to a stricter stopping criterion, allowing us to stop adding points to the lower bound sooner. This is useful since only the inducing points used for the lower bound are retained for future batches. So, by increasing computation for the upper bound slightly, we ultimately reduce the number of inducing points retained lowering the overall cost.\n\n**[Q] How do $\\mathbf{q(a)}$ and $\\mathbf{q(b)}$ mix for the third iteration $\\mathbf{c}$?**\n\nIn our algorithm, we keep the old inducing point locations $\\mathbf{Z}_o$ fixed and choose the new set of inducing points from among the locations in the new batch $X_n$. Consider three batches $\\mathbf{X}_1, \\mathbf{X}_2, \\mathbf{X}_3$ with inducing variables $\\mathbf{a}, \\mathbf{b}, \\mathbf{c}$. For the first batch, we construct the variational approximation $q(\\mathbf{a})$ using inducing points $Z_1 \\subset \\mathbf{X}_1$. For the second batch, we keep $\\mathbf{a}$ and select new inducing points $Z_2' \\subset \\mathbf{X}_2$, forming $\\mathbf{b} = \\mathbf{a} \\cup f(Z_2')$. The updated variational posterior is:\n\n$$\nq(\\mathbf{b})\\propto p(\\mathbf{b})  \\mathcal{N} ( \\hat{\\mathbf{y}} ; \\mathbf{K}\\_{\\hat{\\mathbf{f}} \\mathbf{b}} \\mathbf{K}\\_{\\mathbf{b}\\mathbf{b}}^{-1}\\mathbf{b}, \\Sigma\\_{\\hat{\\mathbf{y}}} )\n$$\nwhere $\\mathbf{K}\\_{\\hat{\\mathbf{f}} \\mathbf{b}} = \\begin{bmatrix} \\mathbf{K}\\_{\\mathbf{fb}} \\\\ \\mathbf{K}\\_{\\mathbf{ab}} \\end{bmatrix}$ , $\\Sigma_{\\hat{\\mathbf{y}}} = \\mathrm{diag}([\\sigma_y^2 \\mathbf{I},\\, \\mathbf{D}\\_{\\mathbf{a}}])$, $\\hat{\\mathbf{y}} = \\begin{bmatrix} \\mathbf{y}_n \\\\ \\mathbf{D}\\_{\\mathbf{a}} \\mathbf{S}\\_{\\mathbf{a}}^{-1} \\mathbf{m}\\_{\\mathbf{a}} \\end{bmatrix}$. For the third batch, $q(\\mathbf{b})$ now summarises all past information. We select new inducing points $Z_3' \\subset \\mathbf{X}_3$  and form: $\\mathbf{c} = \\mathbf{b} \\cup f(Z_3')$ which leads to the update:\n$$\nq(\\mathbf{c})\\propto p(\\mathbf{c})  \\mathcal{N} ( \\hat{\\mathbf{y}} ; \\mathbf{K}\\_{\\hat{\\mathbf{f}} \\mathbf{c}} \\mathbf{K}\\_{\\mathbf{c}\\mathbf{c}}^{-1}\\mathbf{c}, \\Sigma\\_{\\hat{\\mathbf{y}}} )\n$$\nwith updated kernel matrices and statistics, using $q(\\mathbf{b})$ instead of $q(\\mathbf{a})$.\n\n**[Q] Is the noise hyperparameter fixed, as stated in the Experiments section?**\n\nThank you for pointing this out. This is a typo: the noise hyperparameter is not fixed. The text should read, \"the variational distribution, noise and kernel hyperparameters are optimised [...]\". We will correct this. \n\n**Figure 2A**:  Thank you for the suggestion. We will emphasise that $M$ shown at the top is for VIPS using a subscript $M_{vips}$.\n\n**Stress testing**: Thank you for your suggestions and for clearly indicating that additional experiments were not required at this stage.  Please see our response to Reviewer Me8M regarding larger datasets.\n\n**Technical challenges in Bui et al. (2017). Does VIPS do better?**\n\nAs you pointed out in your review, our bound in Eq. (7) is a more interpretable version of the online lower bound introduced by Bui et al. (2017) using Panos et al. (2018) reparametrisation, so both methods share the same properties. The novelty in our method is that, unlike Bui et al. (2017), which used a fixed number of inducing points and heuristically retained 30% of the old ones, we propose a principled way to decide how many and which new inducing points to add to maintain the approximation quality.", "claims_and_evidence": "Some points of strength that I consider relevant for this subsection on claims and evidence:\n- Very interesting approach to continual learning, particularly with the focus on computational resources -- as stated mainly in the intro (column 2, pp 1). \n- Correct identification of issues and challenges, limitations of current SOTA methods, and in general, a nice willingness to provide rigorous continual learning methods for GP regression close to full-batch performance.\n- I do believe problem statement + literature review + method proposal is of highest scientific quality. Additionally, the work builds the methodology on top of 3 well-recognised solutions: \"Titsias GP bound\" from Titsias et al. (2014), streaming (sparse) GPs from Bui et al. 2017, and Burt et al. (2019, 2020) line of research to optimally find the number M of inducing points.", "methods_and_evaluation_criteria": "I will add in this current section all the questions/points of curiosity that I would like to hear about from the technical, methodological and theoretical sides. Additionally, I see the utility, and personally liked, the following decisions taken:\n\n- Use of the re-parametrization from Panos et. al (2018), to obtain an extra likelihood parametrization on the old inducing points, such that the online ELBO in Eq. (7) is more interpretable and later allows to build Eq. (8) for the regression case considered.\n- The way of selecting the threshold $\\alpha$ inspired by Grunwald & Roos, (2019) is certainly nice and inspirational in this case. Also the way that later the noise model is used as the baseline.\n- There is a key point on the clarity of the authors when it is stated in section 4.3 that the strategy for selecting $\\alpha$ varies depending on if the batch $N_n$ is very large or not. Quite interesting indeed.", "theoretical_claims": "Some questions on claims and details that I did not find clear enough or I did not understand very well while reading it:\n\n- **[Question]** What is the main reason behind the focus on (single-output) regression problems only? The streaming sparse method of Bui et al. (2017) was applicable to both classification and regression problems.. I do see that it must be due to the guarantees and the bound build on top of the exact optimal bound of Titsias et al. (2014) that (as long as I remember) was developed only for GP regression with Gaussian likelihoods. Is there any other reason?\n- **[Question]** Between Eq. (4) and Eq. (5), the term $\\Phi$ is omitted as it cannot be computed. I see that some comments on its properties are later added, but is Eq. (5) still a bound really? Is the theoretical rigour of the bound kept in this omission of the $\\Phi$ term?\n- **[Comment]** To me, in Eq. (7) the last term $-\\log \\mathcal{N}(\\mathbf{a}|...)$ could be ignored from an optimization point of view right? As it does not depend on $\\mathbf{b}$, new variational parameters or new hyperparameters.\n- **[Question]** The way and reasons in which $M$ is augmented for both $\\mathcal{L}$ and $\\mathcal{U}$ in Eq. (12) look a bit like mysteries. $\\mathcal{U}$ must vary wrt $M$ in the same way as $\\mathcal{L}$? Am I missing some details here?\n- **[Question]** How do $q(\\mathbf{b})$ and $q(\\mathbf{a})$ mix together for the third (let's say $\\mathbf{c}$) iteration? I am missing a bit the algorithmic point of view, and how the method 'refresh' itself for each new batch from a variational parameter point of view.\n- **[Question]** From the description in the 1st paragraph of the Experiments section, I am assuming that the likelihood noise hyperparameter is fixed. What happens if not? Does this produce issues to the stability of the method?", "experimental_designs_or_analyses": "I do like the way experiments were designed, the empirical results and the perspective brought in both Figure 1 (types of data distribution in the batches) and Figure 2 (continual learning GP vs full-batch exact GP). \n\nSome points that make me feel concerned somehow:\n- Figure 2A is a bit confusing, since we have in the same vertical axis curves with M=8 and M=10 from the legend. Maybe there could be a better way to show this info without intersecting in the same points, curves with different M values.\n- Continual learning is a problem that deals a lot with the idea of an \"unstoppable\" flow of input data, such as one never should keep data points in memory, revisit them, etc, etc. From Figure 1 both time and performance are fantastic, but I don't really perceive that the method has been tested in \"stress\" situations (i.e. 1k, 10k batches for instance). Additionally, such long time/term analysis would have been great wrt the memory allocation. (This is not a call for additional experiments in the rebuttal, just a comment/suggestion of improvement).\n- To me, the work inherits a lot the spirit of Bui et al. (2017). However, the main weakness or point of technical struggle of that method was the management of old variational parameters, inducing points and hyperparameters. Does VIPS do better somehow? Experiments are not saying something new to me in this direction, at the moment.", "relation_to_broader_scientific_literature": "Good review of literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S8vmONwRCV", "reviewer": "Reviewer_Me8M", "summary": "The paper addresses the problem of choosing an appropriate number of inducing points for a sparse Gaussian process in the context of continual learning, where batches of data are observed sequentially, such that the total number of data points is not known before training, which prevents the use of heuristics that depend on the size of the dataset. An \"online ELBO\", which lower bounds the marginal likelihood based only on the current batch of training data, is introduced and used to optimize noise, kernel, and variational parameters. A corresponding upper bound is derived and used to dynamically increase the number of inducing points. Empirical experiments demonstrate that the proposed strategy leads to efficient usage of resources while maintaining performance.\n\n---\n## Update after rebuttal\nI am satisfied by the rebuttal response and I continue to support acceptance of this submission. I maintain my score of 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and your clear recommendations for improvement. We appreciate your positive feedback on our work.\n\n**Suggestions on main results presentation and is there any particular reason why you only include NLPD results in the appendix?**\n\nThank you for your suggestions on which results to include in the main paper. Space constraints were the main reasons to place some of the results on the appendix but we agree that including them in the main text will be valuable. \n\n**Data sorted by the first dimension: Why not sort by the last dimension instead / what if the data is not sorted at all?) Do you have any specific reason for this? And how does this compare to batches which are simply sampled uniformly at random?**\n\nThe reason we sorted the data along the first dimension is to follow the experimental setup used by Chang et al. (2023) for the UCI experiments, but any other dimension could have been used. If we were to sample batches uniformly at random, we would expect a behaviour similar to the middle column of Figure 1. After a few initial batches, enough of the input space would be covered causing the number of inducing points to asymptote to a particular value.\n\n**All considered datasets [...] seem to be quite small...** \n\nWe agree with the reviewer that adding larger-scale datasets could improve the paper. The main reason for not including them was that finding suitable large-scale real-world datasets for where GPs are an appropriate model to use is challenging. However, we were considering including a large-scale synthetic dataset in the final version, which would allow us to evaluate the scalability of our method in such scenarios.", "claims_and_evidence": "The main claim of the paper is to \"develop a method to automatically adjust model size while maintaining near-optimal performance\" in the context of continual learning, where model size refers to the number of inducing points in sparse Gaussian processes. This is achieved by introducing an online ELBO and an online upper bound on the log marginal likelihood which (together with a baseline noise model) are used to select the number of inducing points such that the approximation error will be below a certain threshold.", "methods_and_evaluation_criteria": "The proposed method is specifically designed for the problem at hand and makes sense. In terms of experiments, the paper uses the popular combination of a simple 1D toy experiment and standard UCI regression benchmarks, which is acceptable but arguably somewhat underwhelming. The proposed method is compared to two alternatives for automatic selection of the number of inducing points, which is also appropriate, given that there is not a lot of existing work on this topic (as far as I know). Additional interesting results on a real-world dataset are provided in Appendix D.4. I encourage the authors to include this experiment in the main paper.", "theoretical_claims": "The main theoretical claim is a \"Guarantee\" in Section 4.2 which upper bounds the KL divergence between the actual and the optimal variational posterior over the latent function after observing the latest batch of training data. The argument given in line 291 right below the statement is sensible. I did not check the complete proof in Appendix C.", "experimental_designs_or_analyses": "- For the experiment in Section 5.3, the data was sorted along the first dimension to create batches to simulate continual learning. This seems quite arbitrary (why not sort by the last dimension instead / what if the data is not sorted at all?).\n- All considered datasets (including the real-world data used by the experiment discussed in Appendix D.4), seem to be quite small. In particular, they would be small enough to fit an exact Gaussian process on a modern GPU. I acknowledge that the number of inducing points was comparatively small (a few dozens or hundreds), but this may not faithfully represent a scenario where continual learning would actually be necessary because the whole dataset becomes too large to keep track of.", "relation_to_broader_scientific_literature": "Selecting the number of inducing points for sparse Gaussian processes is an unsolved problem and relevant to the whole research area of sparse Gaussian processes. While there are a few approaches in the literature, many practitioners simply choose an arbitrary value based on the amount of available computational resources. This paper provides a principled way of selecting the number of inducing points in the context of continual learning, and demonstrates empirically that the proposed method performs better than existing alternatives.", "essential_references_not_discussed": "I do not know of any essential reference which is not currently discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n- clear definition and motivation of the addressed research problem\n- principled solution with theoretical arguments, which also seems to work well empirically\n- detailed manuscript and appendix with thorough descriptions, pseudocode, derivations, etc.\n\nWeaknesses:\n- experiments only consider somewhat small datasets which might not be realistic for continual learning", "other_comments_or_suggestions": "- I encourage the authors to include (some) NLPD results and the experiment from Appendix D.4 in the main paper\n- Figure 3 currently uses quite a lot of space in the main paper without providing a lot of information (low \"information density\")", "questions_for_authors": "1. It seems like all experiment results in the main paper only consider the RMSE, whereas Gaussian processes are celebrated for their ability to quantify uncertainty, which can be evaluated using e.g. predictive log-likelihood. Is there any particular reason why you only include NLPD results in the appendix? I find this particularly relevant because sparse Gaussian processes are known for struggling with good uncertainty predictions.\n2. For the experiment in Section 5.3, the data was sorted along the first dimension and divided into batches to simulate continual learning. This seems quite arbitrary (why not sort by the last dimension instead?). Do you have any specific reason for this? And how does this compare to batches which are simply sampled uniformly at random?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Four knowledgeable reviewers recommend Accept and I agree. Reviewer Me8M recommends including a large-scale experiment, and the authors seem to agree it would strengthen the results. Please include such an experiment in an appendix.
Decision: Accept (spotlight poster)