"How Big is Big Enough?'' Adjusting Model Size in Continual Gaussian Processes

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-02 11:07:04

Abstract: For many machine learning methods, creating a model requires setting a parameter that controls the model's capacity before training, e.g. number of neurons in DNNs, or inducing points in GPs. Increasing capacity improves performance until all the information from the dataset is captured. After this point, computational cost keeps increasing, without improved performance. This leads to the question "How big is big enough?" We investigate this problem for Gaussian processes (single-layer neural networks) in continual learning. Here, data becomes available incrementally, and the final dataset size will therefore not be known before training, preventing the use of heuristics for setting the model size. We provide a method that automatically adjusts this, while maintaining near-optimal performance, and show that a single hyperparameter setting for our method performs well across datasets with a wide range of properties.

Main paper: 
 introduction: 
 Continual learning aims to train models when the data arrives in a stream of batches, without storing data after it has been processed, and while obtaining predictive performance that is as high as possible at each point in time [34] . Selecting the size of the model is challenging in this setting, since typical non-continual training procedures do this by trial-and-error (cross-validation) using repeated training runs, which is not possible under our requirement of not storing any data. Selecting model size is crucial, since if the model is too small, predictive performance will suffer. One solution could be to simply make all continual learning models so large, that they will always have enough capacity, regardless of what dataset and what amount of data they will be given. However, this "worst-case" strategy is wasteful of computational resources.A more elegant solution would be to grow the size of the model adaptively as data arrives, according to the needs of the problem (see Figure 1 for an illustration). For example, if data were only ever gathered from the same region, there would be diminishing novelty in every new batch, leading to a possible halt in growth, with growth resuming once data arrives from new regions. In this paper, we investigate a principle for determining how to select the size of a model so that it is sufficient to obtain near-optimal performance, while otherwise wasting a minimal amount of computation. In other words, we seek to answer the question of "how big is big enough?" for setting the size of models throughout continual learning.We investigate this question for Gaussian processes where excellent continual learning methods exist but assume a fixed model capacity that is large enough. We introduce a criterion for determining the necessary number of inducing variables as new data arrives, ensuring the model remains neither too small to perform well nor too large to waste resources. This method, requiring minimal hyperparameter tuning, works effectively across various datasets. For related work, see Appendix C. Submitted to Workshop on Bayesian Decision-making and Uncertainty, 38th Conference on Neural Information Processing Systems (BDU at NeurIPS 2024). Do not distribute. Top: Three consecutive batches of data for 1) growing input space 2) consistent distribution and 3) concentrated input space with occasional outliers. Bottom: Number of inducing points selected using the VIPS algorithm at each batch. We observed: 1) a linear increase, 2) after initial training, we see a halt in growth, and 3) low model size until it encounters outliers.2 Background 
sparse variational gaussian processes: 
 We consider the typical regression setting, with training data consisting of N input/output pairs {x n , y n } N n=1 , x n ∈ R D , y n ∈ R. We model y n by passing x n through a function followed by additive Gaussian noise y n = f (x n ) + ϵ n , ϵ n ∼ N (0, σ 2 ), and take a Gaussian process prior on f ∼ GP(0, k θ (•, •)) with zero mean, and a kernel k with hyperparameters θ. While the posterior (for prediction) and marginal likelihood (for finding θ) can be computed in closed form [33] , they have a computational cost of O(N 3 ) that is too high, and require all training data (or statistics greater in size) to be stored, both of which are prohibitive for continual learning. Variational inference can provide an approximation at a lower O(N M 2 ) computational and O(N M ) memory costs by selecting an approximation from a set of tractable posteriors q(f (•)) = p(f (•)|u, θ)q(u)du (1)EQUATIONwith[K uu ] ij = k(z i , z j ), [k •u ] i = [k T u• ] i = k(•, z i ), Z = {z m } M m=1, and q(u) = N (u; m, S). The variational parameters m, S, Z and hyperparameters θ are selected by maximising the Evidence Lower Bound (ELBO). This simultaneously minimises KL gap KL[q(f ) || p(f |y, θ)] between the approximate and true GP posteriors [24, 23] , and maximises an approximation to the marginal likelihood of the hyperparameters:EQUATIONThe variational approximation has the desirable properties [42] of 1) providing a measure of discrepancy between the finite capacity approximation, and the true infinite capacity model, 2) resulting in arbitrarily accurate approximations if enough capacity is added [4] , and 3) retaining the uncertainty quantification over the infinite number of basis functions. In this work, we will particularly rely on being able to measure the quality of the approximation to help determine how large M should be.We use the extension of the sparse variational GP approximation to the continual learning case developed by Bui et al. [3] . In this online setting, we aim to update our posterior and hyperparameter approximations after each batch of new data {X n , y n }. While we do not have access to data from older batches {X o , y o }, the parameters specifying the approximate posteriorq o (f ) = p(f ̸ =a |a, θ o )q o (a)are passed on. This approximate posterior is constructed as in eq. ( 1) but with a = f (Z o ) and the old hyperparameters θ o . Given this information, online sparse GP approaches aim to obtain anapproximation q n (f ) = p(f ̸ =b |b, θ n )q n (b), where b = f (Z n ) and θ n is the new hyperparameter which can differ from θ o , of the posterior distribution for all observed data p(f |y o , y n , θ n ). This is done by maximising the following the training objective:EQUATION)which we refer to as the "online ELBO". We provide technical details of this quantity in Appendix D.1, where we modify the typical derivation to 1) clarify how the online ELBO provides an estimate to the full-batch ELBO, and 2) clarify when this approximation is accurate.To achieve a fully black-box solution, we must specify how to choose the hyperparameters θ n , the number of inducing variables M b , and the inducing inputs Z n . We select θ n by maximising L using L-BFGS and determine Z n using the "greedy variance" criterion [10, 11, 4] . This leaves only the number of inducing variables M b to be chosen. 
automatically adapting approximation capacity: 
 We propose a method for adjusting the capacity of the approximation M b to maintain accuracy. We keep inducing points from old batches fixed, and select new inducing points from each incoming batch, with their locations set using the "greedy variance" criterion [4, 10, 11] . While optimising all inducing points leads to a strictly better approximation, we avoid this for simplicity. The question remains: To achieve a certain level of accuracy, "how big is big enough?" To answer this, we consider the online ELBO as a function of the capacity L(M b ), and propose a threshold after which to stop adding new inducing variables.The problem of selecting enough inducing variables remains open in the full-batch setting. One possible strategy is to derive an upper bound on the marginal likelihood (U) and stop adding inducing variables the difference U -L (which upper bounds KL[q(f )||p(f |y)]) falls below a tolerance α [40] .Similarly, we consider the maximum possible value of our lower bound, which in the online setting is obtained by retaining previous inducing inputs and adding each new datapoint to the inducing set:EQUATIONUsing properties of positive semi-definite matrices, we derive an upper bound U(M ) to eq. ( 5):L * ≤ - (N n + M a ) 2 log(2π) - 1 2 log |Q ff + Σ ŷ| - 1 2 ŷT Q ff + tI + Σ ŷ -1 ŷ + ∆ a := U(M ),where t = tr(K ff -Q ff ) andQ ff = K f b K -1 bb K b fand M is the number of inducing points used to calculate the bound (which can be unequal to M b ). Adding inducing points will eventually increase 23, 4 ]. If we add inducing points until U(M ) -L(M b ) ≤ α we can give an approximation guarantee, which details can be found in Appendix E.L until it reaches L * [2,In this final step, we specify a heuristic for selecting α that works uniformly across datasets without prior knowledge. A constant α is ineffective due to varying LML scales, influenced by dataset size and observation noise. This means that a tolerance of 1 nat [7] may be appropriate for a small dataset, but not for a large one. We take inspiration from compression and MDL [16] and view the ELBO as being proportional to negative the code length that the model requires to encode the dataset. Instead of choosing a fixed α, we select α such that we compress to within a small fraction of the optimal variational code. To address the issue of undefined quantisation tolerance, we take an independent random noise code as our baseline, and select α to be within some small fraction of the optimal variational code, relative to the random noise code, i.e. we define: where μ and σ2 are the average and variance of the observations for up to the current task and δ is a user-defined hyperparameter. The interpretability of δ allows it to be set in advance without needing prior data knowledge. For instance, to capture 95% of the information at each batch, δ should be set to 0.05. In order to stay computationally feasible for large batch sizes N n , we can replace L * in eq. 6EQUATIONwith U(M ). The number of inducing points M can be defined as the maximum between a predefined value and the size of X n , for instance, M = min(5000, N n + M a ).The algorithm for our inducing point selection method can be found in Appendix B. We name our approach Vegas Inducing Point Selection (VIPS), drawing an analogy to Las Vegas Algorithms [28] .These methods guarantee the accuracy of the output, however, their computational time fluctuates for every run [28] . 
experiments: 
 In this section, we briefly show how our algorithm performs in different data settings and compare it to alternative inducing point selection approaches. We provide additional experiments in Appendix F.For the experiment, we aim to demonstrate how data distribution affects the model's size requirement during continual learning. In Figure 1 , we use three different datasets, each divided into 10 batches with varying capacity requirements. We record the total number of inducing points selected at each batch using our inducing point method, VIPS. We observed VIPS' ability to handle model size across different data distributions during continual learning. See Appendix F.1 for details.We compare our inducing point selection method, VIPS, to two other adaptive approaches that iteratively choose the number of inducing points with different stopping criteria: Conditional Variance [4] and OIPS [12] (see Appendix D for details). We evaluate five datasets from the UCI repository [8] , simulating a streaming scenario by sorting the data based on the first dimension and dividing it into batches. Results are reported over five 80%-training and 20%-testing random splits, including root mean square error (RMSE) and negative log predictive density (NLPD). Table 1 summarises the last batch results for each dataset in terms of inducing points and RMSE, with our method achieving a balanced selection of inducing points and maintaining comparable accuracy levels to other methods using a single parameter setting across all datasets. See Appendix F.3 for further details. 
discussion: 
 In this work, we introduce a method to automatically adjust the number of inducing variables in streaming GP regression, maintaining performance close to full-batch methods while optimising computational efficiency. Unlike conventional approaches with a fixed number of inducing points, our method adapts dynamically, requiring only a single, user-controlled parameter that works well across datasets. Our hope is that these perspectives will be useful for developing adaptive deep neural networks, allowing models to be trained on large amounts of data beyond storage limitations, or learning from data streams from direct environmental interaction.All experiments were implemented in Python using libraries TensorFlow v.2.12 [1] and GPflow v.2.9 [25] .The implementation of the online lower bound (see eq. ( 11)) by Bui et al. [3] is available at https://github. com/thangbui/streaming_sparse_gp. For the "greedy variance" criterion, we use the implementation provided in Burt et al. [4] . For OIPS [12] , we adapt the Julia implementation to Python.Algorithm 1 Vegas Inducing Point Selection (VIPS)Input: X n = {x i } Nn i=1 , Z o = {z m } Ma m=1 , μ, σ, θ n , kernel k(•, •|θ n ), threshold parameter δ. Output: Updated set Z n = Z o ∪ {x m ′ } M b m ′ =1 . Initialise Z n = Z o . while U(M ) -L(M b ) ≥ δ| U(M ) -L noise (μ, σ)| do Select x = argmax x∈Xn k(x, x) -k b (x) ⊤ K -1 bb k b (x). Add x to the set of inducing points: Z n = Z n ∪ {x}.Algorithm 1 presents an inducing point selection method using our stopping criterion combined with the location selection strategy "greedy variance" [4, 10, 11] . At each batch, we keep the old inducing point locations fixed and choose the new set of inducing points from among the locations in Xn. The method takes as input a value for the hyperparameter θn. In practice, we will set θn = θo to select the number of inducing points; the hyperparameter θn is inferred by optimising L(M b ) once the inducing point locations Zn have been chosen.The most widely discussed problem in continual learning is that of catastrophic forgetting, where previously acquired knowledge is lost in favour of recent information [27, 15] . Many solutions have been proposed in the literature, such as encouraging weights to be close to values that were well-determined by past data [19, 37] , storing subsets or statistics of past data to continue to train the neural network in the future [20, 21] , and approximate Bayesian methods that balance uncertainty estimates of parameters with the strength of the data [31, 35, 6] . Within continual learning, many different settings have been investigated, which vary in difficulty [9] . Across these tasks, the gap in performance to a full-batch training procedure therefore also varies, but despite progress, some gap in performance remains.Bayesian continual learning methods have been developed because the posterior given past data becomes the prior for future data, making the posterior a sufficient quantity to estimate [29] . For the special case of linear-in-the-parameters regression models, the posterior and updates can be calculated in closed form, leading to continual learning giving exactly the same result as full-batch training. In most cases (e.g. for neural networks), the posterior cannot be found exactly, leading to the aforementioned methods [31, 35, 6] that focus on finding an approximation to the posterior and using this as the sufficient quantity.Even with a perfect solution to catastrophic forgetting (e.g. in the case of linear-in-the-parameters regression models), continual learning methods face the additional difficulty of ensuring that models have sufficient capacity to accommodate the continuously arriving information. In continual learning, it is particularly difficult to determine a fixed size for the model, since the number of data or tasks are not yet known, and selecting a model that is too small can significantly hurt performance. To improve over a fixed model size, methods can be made to grow with the data size. For example, Rusu et al. [36] extend hidden representations by a fixed amount for each new batch of data that arrives, and allows the weights of the extended representation to depend on the representation of all previous tasks. Yoon et al. [43] argue that extension by a fixed amount is wasteful and should instead be data dependent, specifically by copying neurons if their value changes too much, and adding new neurons if the training loss doesn't reach a particular threshold. Kessler et al. [18] propose to use the Indian Buffet Process as a more principled way to regularise how fast new weights are added with tasks. While the data dependence that both these methods introduce is necessary to prevent computational waste, both methods have hyperparameters that need to be tuned to dataset characteristics, which is difficult when the dataset characteristics are not known at the start of training.Growing model capacity with dataset size was one of the main justifications for research into (Bayesian) non-parametric models [13, 14] . This approach defines models with infinite capacity, with Bayesian inference naturally using an appropriate finite capacity to make predictions, with finite compute budgets. Gaussian processes (GPs) [33] are the most common Bayesian non-parametric model for supervised learning, and are equivalent to infinitely-wide deep neural networks [30, 26] and linear-in-the-parameters models with an infinite feature space. Their infinite capacity allows them to recover functions perfectly in the limit of infinite data [41] , and their posterior can be computed in closed form. These two mathematical properties provide strong principles for providing high-quality solutions to both catastrophic forgetting and ensuring appropriate capacity, and therefore make GPs an excellent model for studying continual learning.However, developing a practical continual learning in GPs is not as straightforward as it is in finite dimensional linear models, because (for N datapoints) the posterior requires 1) O(N 3 ) operations to compute it exactly, which becomes intractable for large datasets, and 2) storing the full training dataset, which breaks the requirements of continual learning. Sparse variational inducing variable methods have been proposed to solve these problems [39] , by introducing a small number of M inducing points that control the capacity of the posterior approximation.In certain settings, this approximation is near-exact even when M ≪ N [4] . This property has allowed continual learning methods to be developed for GPs that perform very closely to full-batch methods [3, 22, 5] , provided M is large enough.As in neural network models, selecting the capacity M is an open problem, with several proposed solutions.Kapoor et al. [17] acknowledge the need for scaling the capacity with data size, and propose VAR-GP (Variational Autoregressive GP) which adds a fixed number of inducing points for every batch. However, this number may be too small, leading to poor performance, or too large, leading to wasted computation. Galy-Fajou and Opper [12] propose OIPS (online inducing point selection), which determines M through a threshold on the correlation with other inducing points, which needs to be tuned based on dataset properties.In this work, we propose to instead select the capacity of the variational approximation by selecting an appropriate tolerance in the KL gap to the true posterior. This criterion works within the same computational constraints as existing GP continual learning methods, adapts the capacity to the dataset to minimise computational waste while retaining near-optimal performance. Our method has a single hyperparameter that we keep fixed to a single value, and that produces similar trade-offs across benchmark datasets with significantly different characteristics.The effectiveness of sparse approximations is determined by two key factors: the number of inducing points (M ) and their locations. These parameters are responsible for a trade-off between the scalability and accuracy of the model, thus representing the captured information about the function f [4, 22] . There exist multiple approaches to inducing point selection (see Burt et al. [4] for a review), broadly falling into two categories.The first category includes choosing a subset of the input data through sampling or clustering and refining their location during gradient optimisation of the ELBO. For instance, Bui et al. [3] decided to sample a fixed number of inducing points from the set {Zo, Xn} and optimise their locations. However, this approach uses a fixed memory, which requires a manual setting of the number of inducing points. Determining the memory size is inherently challenging as it will depend on the dataset's properties and the estimated number of data points, not possible to know in advance.In the second approach, points are iteratively selected from a set based on a specific criterion. Burt et al.[4] introduced an iterative "greedy variance" strategy that greedily chooses the location of the next inducing point to maximize the marginal variance in the conditional prior p(f ̸ =u |u). This is equivalent to maximisingdiag[K ff -Q ff ]. The quantity tr (K ff -Q ff ) is a bound for the norm of K ff -Q ff ,and can therefore be used as a stopping criterion to determine the number of inducing points. We refer to the method that uses "greedy variance" with this stopping criterion as Conditional Variance. In this algorithm, new inducing points are no longer added once tr (K ff -Q ff ) falls below a chosen tolerance value η. Although, an equivalent approach was studied in Fine and Scheinberg [10] and Foster et al. [11] , this stopping criterion has not yet been tested in the literature. In the online setting, Chang et al. [5] use the "greedy variance" criterion by simply defining {Zo, Xn} as the selection pool from which inducing point locations are selected, however, they maintain a fixed number of inducing points. Similarly, Maddox et al. [22] extends the "greedy variance" criterion to heteroskedastic Gaussian likelihoods and also uses a fixed memory approach. Galy-Fajou and Opper [12] introduced the Online Inducing Points Selection (OIPS) algorithm, a method that iteratively adds points from Xn to the existing set of inducing points: a point x ∈ Xn is added to the set of inducing points if the maximum value of the covariance vector ku(x) is below a set threshold ρ. This method measures the potential potential impact of each new point with the existing inducing points set given the covariance threshold, this allows for an automatic selection of the number of inducing points unless a maximum is specified.In our work, we are not interested in constraining the capacity of the model. Therefore, we will compare our method VIPS, to Conditional Variance with the selection pool ({Zo, Xn}) and OIPS without specifying a maximum number of inducing points. Table 2 shows a summary of the properties of these methods as well as the fixed memory approach used in Bui et al. [3] . In this work, we use the extension of the sparse variational GP approximation to the continual learning case developed by Bui et al. [3] . We modified the typical derivation to 1) clarify how the online ELBO provides an estimate to the full-batch ELBO, and 2) clarify when this approximation is accurate.In this online setting, we aim to update our posterior and hyperparameter approximations after each batch of new data {Xn, yn}. While we do not have access to data from older batches {Xo, yo}, the parameters specifying the approximate posterior qo(f ) = p(f ̸ =a |a, θo)qo(a) are passed on. This approximate posterior is constructed as in eq. ( 1) but with a = f (Zo) and the old hyperparameters θo. Given this information, online sparse GP approaches aim to obtain an approximation qn(f ) of the posterior distribution for all observed data p(f |yo, yn, θn), which we rewrite as:p(f |yo, yn, θn) = p(f |θn)p(yn|f )p(yo|f ) p(yn, yo|θn) = p(f |θn)p(yn|f ) p(yn, yo|θn) p(f |yo, θo)p(yo|θo) p(f |θo) .We denote the new variational distribution as qn(f ) = p(f ̸ =b |b, θn)qn(b) where b = f (Zn) and θn is the new hyperparameter which can differ from θo. The KL divergence between the exact and approximate posterior at the current batch is given by:KL[qn(f ) || p(f |yo, yn, θn)] = log p(yn, yo|θn) p(yo|θo) -qn(f ) log p(f |θn)p(yn|f )p(f |yo, θo) qn(f )p(f |θo) df .The posterior distribution p(f |yo, θo) is not available, however by multiplying its approximation qo(f ) in both sides of the fraction inside the log, we obtain:KL[qn(f ) || p(f |yo, yn, θn)] = log p(yn, yo|θn) p(yo|θo) -qn(f ) log p(f |θn)p(yn|f )qo(f ) qn(f )p(f |θo) df + Φ (7)where Φ = -qn(f ) log p(f |yo,θo)qo(f )df . We cannot compute Φ due to its dependence on the exact posterior, so we drop it and use the following "online ELBO" as our training objective:EQUATIONMaximising L will accurately minimise the KL to the true posterior when Φ is small, which is the case when the old approximation is accurate, i.e. qo(f ) ≈ p(f |yo, θo) for all values of f (with Φ = 0 in the case of equality).In our continual learning procedure, we will keep our sequence of approximations accurate by ensuring they all have enough inducing points.To get our final bound, we perform a change of variables for the variational distribution qo(a) = N (a; ma, Sa)to use the likelihood parametrisation [32] :EQUATIONwhere Da = S -1 a -K ′-1 aa -1 and ma = K ′-1 aa ma are the variational parameters, K ′ aa is the covariance for the prior distribution p(a | θo) and l(a) := N (a; ma, Da). In this formulation, the variational parameters ma, Da effectively form a dataset that produce the same posterior as the original dataset, but which we have chosen to be smaller in size, M < N . This makes our online ELBO from eq. ( 8)EQUATION) which has the nice interpretation of being the normal ELBO, but with an additional term that includes the approximate likelihood l(a) which summarises the effect of all previous data.While L is all that is needed to train the online approximation, it differs from the true marginal likelihood by the term log p(yo|θo). To approximate it, we could drop the term log N (a; 0, K ′ aa + Da) from L, since this term also approximates log p(yo|θo), with equality when the posterior is exact, but with no guarantee of being a lower bound.Although L is a useful training objective for general likelihoods, the regression case we consider allows us to analytically find q(b) (refer to Bui et al. [3] for derivations) resulting in the lower boundEQUATIONEQUATIONEQUATIONwith Q ff = K fb K -1 bb K bf and Qaa = K ab K -1 bb K ba .Adding inducing points will eventually increase L until it reaches L * [2, 23, 4 ]. If we add inducing points until U(M ) -L(M b ) ≤ α we can guarantee the following:Guarantee. Let M be a fixed integer and M b be the number of selected inducing points such that U(M ) -L(M b ) ≤ α. Assuming that θn = θo, we have two equivalent bounds:EQUATIONwhere Ψ = qn(f ) logq * n (f ) p(f |yn,yo) df and q * n (f ) = Z -1 qo(f )p(yn|f )represents the variational distribution associated with the optimal lower bound L * = L(Nn + Ma), with Z denoting the marginal likelihood that normalises q * n (f ).Proof. We cease the addition of points whenU(M ) -L(M b ) < α. Since U(M ) ≥ L * , then -L(M b ) < α -U(M ) < α -L * .Assuming θn = θo, eq. ( 7) can be bounded as:EQUATION)where Φ = -qn(f ) log p(f |yo,θo)qo(f ) df . Let q * n (f ) = Z -1 qo(f )p(yn|f )be the variational distribution associated with L * = L(Nn + Ma), then by multiplying this value in both sides of the fraction inside the log, we obtain:EQUATIONThen we can rewrite,EQUATION)Again by multiplying by qn(f ) both sides of the fraction inside the log, we obtain:EQUATIONOur experiments aim to answer the following questions: How big should the size of a model be to effectively learn nearly all information within a dataset? How should the size of a model be dynamically adjusted in the context of continual learning? We assume a streaming scenario where the total number of points is unknown. In all experiments, the variational distribution and kernel hyperparameters are learnt by optimising the online lower bound in eq. ( 11). For all experiments and methods, we use the L-BFGS optimiser.In the experiment, we aimed to illustrate the impact of data distribution on the model's size requirement during continual learning. In Figure 1 , we considered three different datasets, each divided into 10 batches, and recorded the total number of inducing points selected at each batch by our inducing point method, VIPS. In the first scenario, the model encounters a growing input space, leading to a linear increase in the model's capacity with each batch. In the second, the data distribution remains consistent across batches. After initial learning, the number of inducing points asymptotes to the constant value M = 30. In the third scenario, the data is concentrated in a part of the input space with occasional outliers in the last batches. Using a large model from the beginning would be a waste of resources. In contrast, our method maintains a low model size for most batches and only increases dynamically when encountering outliers. These scenarios show VIPS' ability to manage model size across different data distributions during continual learning.For the synthetic dataset, we generated random noisy observations from the test function f (x) = sin(2x) + cos(5x). We used a Squared Exponential kernel initialised with lengthscale 0.5 and variance 1. The noise variance was initialised to 0.5. For VIPS, we use δ = 0.05.We use N = 500 observations uniformly distributed from 0 to 10. The data is ordered and divided into ten random batches.We simulate a scenario where small batches of data are received but the data is distributed across the input space. We use N = 150 observations uniformly distributed from 0 to 10. The data is shuffled and divided into ten random batches.We simulate a scenario where only outliers are encountered from time to time and the rest of the data is concentrated around a small part of the input space. We use two sets of data: the first set is sampled from a uniform distribution from 4 to 6, with N = 1000 and the second set is sampled from a Cauchy distribution with a mean of µ = 5, with N = 300. The data is divided into ten random batches, where the Cauchy observations are placed in the latter batches. In continual learning scenarios, manually setting the appropriate number of inducing points beforehand is not possible since this depends on dataset properties and total data points. Fixed-memory approaches can lead to suboptimal performance if the inducing point set is too small, or computational waste if the model is too large. The following two experiments demonstrate these limitations. In contrast, our proposed method VIPS only increases the number of inducing points as needed, highlighting the importance of an adaptive number of inducing points in continual learning.We compare adaptive and fixed memory methods using a synthetic dataset divided into four equal batches.We evaluate the "greedy variance" location selection method [4] under two scenarios: with a fixed number of inducing points and using our stopping criterion to determine the number of inducing points. We consider three fixed settings (M = 10, 20, 30 total number of inducing points per batch) and record the root mean square error (RMSE) on a test set, using an exact GP model with access to all current training data as a benchmark.Figure 3a shows that fixed memory methods with 10 and 20 inducing points become less accurate as more data is observed, while our method VIPS maintains accuracy at each batch. Both the fixed memory method with M = 30 and VIPS match the performance of an exact GP. This demonstrates the importance of model size in continual learning and the need for adaptive methods to determine the number of inducing points.For the synthetic dataset, we generate 1000 random noisy observations from the test function f (x) = sin(2x) + cos(5x). We used a Squared Exponential kernel initialised with lengthscale 0.5 and variance 1.5. The noise variance was initialised to 0.5. The performance was measured on a test grid of 500 points. For VIPS, δ = 0.05.Naval dataset. We compare the time complexity of the location selection method "greedy variance" [4] under three scenarios. In the first scenario, we assume we have access to an oracle to determine the required number of inducing points for near-exact performance, and we set M = 100 based on the accuracy obtained on a sparse model compared to an exact GP. In the second scenario, we take a heuristic with a large memory (M = 1000, around 1/10th of the total data points) to ensure sufficient capacity. In the third scenario, we use our method VIPS to determine the number of inducing points. We use the "naval" UCI dataset divided into 20 batches and record the training time per batch. Figure 3b shows that both fixed-memory approaches have higher computational time. In contrast, our VIPS method adaptively adjusts M as needed, leading to a reduction in training time. This advantage is expected to be even more pronounced for larger and more complex datasets.This experiment was performed on an Nvidia RTX 6000's GPU on a high-performance computing cluster. We used a Squared Exponential kernel with hyperparameters initialised to 1. The noise variance was initialised to 0.1. The dataset was divided into 20 batches, and we recorded the time in training per batch.Overall these simple experiments aim to demonstrate the advantages of adaptive strategies over fixed approaches, achieving high-quality approximation without wasting computational resources.These experiments were performed on an Nvidia RTX 6000's GPU on a high-performance computing cluster.We used a Squared Exponential kernel with hyperparameters initialised to 1 for all datasets. The noise variance was initialised to 0.1. Table 3 shows the parameters used for each dataset and method. The smaller datasets (< 12000) were divided into 20 batches, and the larger (> 12000) into 50 batches. Figure 4 shows the results for all considered datasets. As mentioned in the main text, the method Conditional Variance often selected the The data used in this experiment is obtained from Solin et al. [38] and can be accessed at https://github. com/AaltoML/magnetic-data. The objective of this task is to detect local anomalies in the Earth's magnetic field online, caused by the presence of bedrock and magnetic materials in indoor building structures. For this purpose, a small robot with a 3-axis magnetometer moves around an indoor space of approximately 6 meters by 6 meters and measures the magnetic field strength. Out of the 9 available trajectories, we use trajectories 1, 2, 3, 4, and 5 (with n = 8875, 9105, 9404, 7332, 8313, respectively) for the experiments. Specifically, we use trajectories 1, 2, 4 and 5 for the first experiment and trajectory 3 for the second.In this study, we use the setup proposed in Chang et al. [5] . The proposed model applies a GP prior to magnetic field strength, given by GP 0, σ 2 0 + κ Mat σ 2 ,ℓ (x, x ′ ) (in µT ), where the kernel consist of a constant kernel and a Matérn-ν/2 kernel. The model assumes the spatial domain is affected by Gaussian noise with a variance σ 2 n .The initial variance for the constant kernel is set to 500, and the Gaussian likelihood is initialised with a noise variance of 0.1. Different to their setup, we do not restrict the number of inducing points, as in the rest of the experiments we set δ = 0.05 for VIPS.In the first experiment, we aim to sequentially learn the paths taken by the robot using trajectories 1, 2, 4 and 5,i.e. an entire path will correspond to one batch. We investigate whether our method can adapt to changes in the environment and adjust the number of inducing points accordingly. During this process, we concurrently learn the hyperparameters σ 2 0 , σ 2 , ℓ, and σ 2 n . Our method enables us to reconstruct a representation of the magnetic field without the need to specify the number of inducing points. Figure 7 shows the temporally updating field estimate over batches alongside the corresponding path travelled in each batch.In the second experiment, we focus on the streaming learning of trajectory 3. The trajectory is split into 20 batches. We compare our method VIPS to Conditional Variance. Since the setting in this experiment mimics an ever-expanding domain, different from Chang et al. [5] we do not restrict the number of inducing points for Conditional Variance and set η = 2.2 for the whole task. This tolerance was chosen for Conditional Variance to achieve the same performance as VIPS with a prudent number of inducing points. Conditional Variance can expand its capacity as it encounters more data, however setting a fixed value for η is a challenge, since this will depend on dataset properties. In contrast, a single parameter setting works well for our method VIPS. Note that we were able to reconstruct a similar local estimate of the first experiment, showing the versatility of our methods to different continual learning scenarios and data distributions. The comparison with OIPS is omitted since the method was not able to add sufficient inducing points to learn the path. Detailed learning of the path for each method is shown in figures 5 and 6. Recall thatEQUATIONwith K ff = K ff K fa K af Kaa .The first term can be lower bounded using Jensen's inequality as, log N (ŷ; 0 , K ff + Σ ŷ )EQUATIONwhere K f b K -1 bb K b f is the Nyström approximation of K ff . The trace will be small when b = {f (xnew, a)} and can be simplified as follows:EQUATIONwhich recovers the expression for L -∆a .In this section, we provide efficient forms for practical implementation of the online upper bound U. As the second term is constant we focus on the first term,EQUATION)This term is an upper bound for the first term of L * = log N (ŷ; 0 , K ff + Σ ŷ ) + ∆a.Letting K bb = L b L T b and using the matrix determinant lemma, we can rewrite the determinant term asEQUATIONLetEQUATIONTherefore,EQUATIONGiven the quadratic term,- 1 2 ŷT K f b K -1 bb K b f + tI + Σ ŷ -1 ŷ.Letting Σ ŷ = tI + Σ ŷ and by Woodbury's formula, we obtain:K f b K -1 bb K b f + Σ ŷ -1 = Σ -1 ŷ -Σ -1 ŷ K f b L -T b I + L -1 b K b f Σ -1 ŷ K f b L -T b -1 L -1 b K b f Σ -1 ŷ .We have, ŷT Σ -1 ŷ ŷ = y +t K bf y + K ba (Da + tI) -1 DaS -1 a ma , we obtain,EQUATIONPutting this back into the upper bound:EQUATION)The upper bound for L * is thereforeU = - (N + Ma) 2 log(2π) - 1 2 log |K f b K -1 bb K b f + Σ ŷ | - 1 2 ŷT K f b K -1 bb K b f + tI + Σ ŷ -1 ŷ + ∆a = - N 2 log(2πσ 2 y ) - 1 2 log |D| - 1 2 1 σ 2 y + t y T y - 1 2DaS -1 a ma T (Da + tI) -1 DaS -1 a maEQUATION 
 
 Appendix:  



Reviews: review: title: title: Official Review of Submission43 by Reviewer 3F3r
main_review: main_review: This paper proposes an algorithm that automatically adjusts the number of data points (inducing points) to use while maintaining near-optimal performance for Gaussian Processes (single-layer neural networks) in continual learning. The algorithm dynamically selects inducing points based on the data, ensuring efficient computation and strong predictive accuracy. Notably, it requires tuning only a single hyperparameter and performs well across various datasets.

Strengths:
1. The proposed algorithm is highly valuable in the context of streaming data. With only one hyperparameter to tune, it significantly reduces the need for fine-tuning, making the contribution noteworthy.
2. The empirical results are convincing and effectively demonstrate the algorithm's performance and efficacy across various datasets.

Weaknesses:
1. The paper could benefit from comparing the proposed algorithm with other adaptive methods in terms of prediction efficiency and additional metrics such as computational efficiency.
2. There is a lack of explanation regarding the underlying motivation for why the algorithm is effective in the continual learning and streaming data setting, which would enhance the understanding of its strengths.
title: title: review on paper43
main_review: main_review: The paper presents an innovative approach to adjusting model size adaptively for Gaussian processes in continual learning settings. This problem is significant due to the dynamic nature of data availability in many real-world applications, and the solution proposed addresses the need for efficiency and scalability without sacrificing performance.

Novelty and Relevance: The concept of adaptive model sizing based on incoming data is novel and very relevant to current trends in machine learning, particularly in areas dealing with streaming data.

Methodological Rigor: The paper is methodologically sound, detailing the criteria for model adjustment and integrating these into the Gaussian process framework effectively.