A scalable Bayesian continual learning framework for online and sequential decision making

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-04 20:19:35

Abstract: Continual learning (CL) refers to the ability to continually learn and exploit new knowledge while retaining experiences accumulated from past.  Though numerous CL methods have been proposed in recent years, it is not straightforward to deployed them directly to online or sequential decision making problems due to the computational burden and the lack of uncertainty quantification. In this paper, we focus on Instance-incremental classification problems with concept shift, and propose an online/sequential decision making model based on a novel scalable Bayesian continual learning framework that provides i) statistically principled and computationally efficient Bayesian knowledge updating scheme and ii) scalable and exact posterior inference procedure based on a Mixture of Experts model. In addition, as an exemplar-free method, our method does not require storing or modelling any previously seen instances, making it appealing to e.g. online decision making problems in biomedical applications where data privacy is of concern.

Main paper: 
 introduction: 
 Continual learning (CL) enables an intelligent system to develop and refine itself adaptively in accordance with real-world dynamics by incrementally accumulating and exploiting knowledge gained from past experience without the need to train any new model from scratch [Hassabis et al., 2017] . The main challenge CL has to tackle is known as catastrophic forgetting, which refers to previously learned knowledge being drastically interfered by new information [McClelland et al., 1995, McCloskey and Cohen, 1989] . In order to deliver accurate and trustworthy predictions, a CL model in practice should, on one hand, be able to integrate new knowledge and update existing knowledge efficiently based on the stream of new inputs from dynamic data distributions (learning plasticity) and, on the other hand, maximally retain relevant information from the past and prevent catastrophic forgetting (memory stability). The competition between these two conflicting objectives is known as the stability-plasticity dilemma, which has been widely studied from both biological and computational perspectives [Ditzler et al., 2015 , Parisi et al., 2019] . See also Wang et al. [2024] for a comprehensive review. The adaptive and sequential nature of CL makes it appealing for online and sequential decision making problems, where the decision boundary may vary temporally or spatially, and one needs to accurately identify and efficiently adapt to such changes. In this paper, we propose a sequential decision making model based on a novel scalable Bayesian CL framework. is the data labels. Here a task refers to a distinct learning problem. In this paper, we focus on the Instance-incremental classification problem under a concept shift [Kurle et al., 2019] Existing state-of-the-art CL approaches such as Chaudhry et al. [2019] , Kumari et al. [2022] and Wu et al. [2018] are computationally intensive, and require either estimating a full generative model of the input data or storing past examples as part of their memory states, which are infeasible in many applications due to privacy or storage concerns. Exemplar-free CL [He and Zhu, 2022] and regularization-based methods such as Kirkpatrick et al. [2017] and Li and Hoiem [2017] avoid the need of storing or modelling input data, but they are also computational costly and lack a statistically principled updating scheme or uncerntainty quantification. These undesirable traits make them unsuitable for problems such as online or sequential decision making. To address these issues, we propose a sequential decision making model based on a conceptually simple and scalable Bayesian continual learning framework that combines statistically principled modelling, and flexible machine learning methods. Our proposed approach utilizes scalable online exact Bayesian update and ensembles learning. The exact Bayesian updating framework ensures resilience to catastrophic forgetting, improves robustness of the model, and gives predictions with easy-to-interpret uncertainty quantification, which is crucial in decision making process. On the other hand, the ensemble framework provides extra flexibility to cope with non-stationarity: by aggregating individual base models that are aware of the potential distributional difference in training samples, the model can adapt to it more efficiently by choosing the best combination of past experiences using both evidence from data and user's domain knowledge. 
method: 
 Our proposed framework consists of two components: a fixed pre-trained feature-extractor and an ensemble of base models. We start by introducing the base model.Each base model in our proposed framework is an online Gaussian process classifier. To achieve exact and computationally efficient prediction, we use the Dirichlet-based GP classification [Milios et al., 2018] , a fast and accurate GP-based model that interprets categorical labels as samples from a Dirichlet distribution, and combine it with WISKI-GP [Stanton et al., 2021] , a scalable online Gaussian process model based on structured kernel interpolation [Wilson and Nickisch, 2015] , to ensure that the base model can be updated tractably and efficiently. This modelling choice ensures both efficient and exact posterior inference, and scalable sequential update without storing or reusing any previous samples: Information extracted from past data is represented by and updated through a interpolating kernel [Wilson and Nickisch, 2015] whose size only depends on the number of inducing points. Let c be the Dirichlet distribution hyperparameter, ν be the WISKI-GP kernel hyperparameter, p c,ν (•|X new , X prev , Y prev ) be the predictive distribution of label associated with a new data pointX new conditioned on historical observations {X prev , Y prev }. Let p c,ν (X prev , Y prev ) be the marginal likelihood of {X prev , Y prev }.In this work, we follow the suggestion in Milios et al. [2018] , set the Dirichlet hyperparameter c = 0.1.Similar to Petit et al. [2023] , we incorporate a feature extractor into our learning framework. We train a feature extractor (e.g. parameteric UMAP [Sainburg et al., 2021] or VAE [Kingma and Welling, 2013] ) based on a collection of samples {X init , Y init } prior to the continual learning process. This {X init , Y init } can either be a historical dataset or simply the first batch {X 1 , Y 1 }. Once it has been trained, the feature extractor is fixed throughout the learning process as in Prabhu et al. [2023] , and the inference and update steps are based on the embedded features generated by the feature extractor. A schematic illustration of the ensemble of three base models given three batches using a binary classification example.Here we discuss the ensemble learning procedure. We omit the task index subscript t for brevity.Suppose the feature extractor P θ parameterized by θ and the WISKI-GP hyperparameter ν have been chosen, and we denote Z θ,1:b the transformed feature obtained by applying P θ to X 1:b , and Z θ,p ens,b (•|Z θ,new ) = b b ′ =1 W b ′ p c,ν b ′ (•|Z θ,new , Z θ,b ′ :b , Y b ′ :b ) (1)where the weightsW b ′ ∝ p c,ν b ′ (Z θ,b , Y b |Z θ,b ′ :b-1 , Y b ′ :b-1 ) for b ′ = 1, ..., b -1, W b ∝ p c,ν b (Z θ,b , Y b ) and b b ′ =1 W b ′ = 1. The marginal likelihoods p c,ν b ′ (Z θ,b , Y b |Z θ,b ′ :b-1 , Y b ′ :b-1 )can also be computed efficiently using the caching procedure given by Stanton et al. [2021] . Under the assumption that test samples and the most recently seen data D t follow the same distribution, this weighting strategy allows the model to make predictions on test samples by utilising previous knowledge in an automatic and adaptive fashion: when D t aligns well with the previous knowledge, the ensemble would favour base models that contain more historical experience as those models provide more informative (i.e. concentrated) priors that are inline with the likelihood of D t . On the other hand, when there is a prior-data conflict between the historical experience and the newly kernel matrix where M is the number of inducing points. As a result, even though the ensemble needs to store an increasing number of models as new batches come in, it is manageable in most of the scenarios. 
numerical examples: 
 We demonstrate our method using a modified version of the rotating-boundary example in Kurle Let data D t = {X t , Y t } for t = 1, ..., T where X t = {X t,i } B i=1 , X t,i ∼ Uniform([-1, 1] 2 ) and We compare the prediction accuracy of our proposed ensemble method with 1) a single oracle GP that has access to all previously seen data and gives prediction based on the full history and 2) a single oracle GP that has access to at most 5 most recently observed batches (i.e. a memory window of length 5) and gives prediction conditioned on the observed batches within the memory window.Y t = {Y t,i } B i=1 , Y t,i ∼ Bernoulli(σ(w T t X t,i )) for i =We stress that unlike the oracle GPs, our ensemble does not require storing or reusing any previous data. We compare the one-batch-ahead prediction accuracy given by different models and report the results in Fig 2 . We find that our proposed method is able adapt to change points faster than both the single and windowed oracle GPs without accessing any previously seen data, indicating the efficacy of our proposed approach. 
conclusion: 
 We propose an online/sequential decision making model based on a scalable and conceptually simple Bayesian continual learning framework under the concept drift setup. Our method combines scalable online exact Bayesian update and ensembles learning. Comparing with existing CL methods, our approach is computationally efficient and statistically more principled thanks to the Bayesian framework, and does not require storing or reusing previously seen samples, making it appealing in applications where data privacy is of concern. In this work, we only consider the scenario where the conditional distribution of label changes over batches (temporally varying decision boundary). We also plan to extend our approach to the scenario where both the conditional distribution of labels and the feature distribution change over batches using techniques such as indirect discriminant alignment [Liu et al., 2020] . 
 
 Appendix:  



Reviews: review: title: title: Review of Bayesian CL
main_review: main_review: **Summary:**
The paper proposes a Bayesian continual learning model for classification with concept shift. The authors show that the proposal is computationally more scalable to existing work, and does not need to store samples.

**Strength:**
- The model is well motivated and tackles an important problem.
- The intuition of the proposed model is clearly described.
- The numerical results (though limited) illustrates the advantage of the proposed method in terms of robustness and adaptivity to data shift.

**Weakness:**
- Some of the model details are missing or not well explained. For example, the choice of the ensemble weight is not justified properly
- The Bayesian update of the predictors is not clearly stated. The readers could benefit more if Section 2.3 can provide more technical details.
- The numerical example seems overly simplified without the feature extraction stage (since it's a 2-dimensional problem). It is unclear how the proposed method perform in more complex scenarios.
- The numerical study is also somewhat limited. For example, oracles of different window size should at least be compared instead of only comparing the full history and a somewhat arbitrary case of 5-batch window.
title: title: Elegant and simple approach to continual learning via ensemble / Bayesian model averaging
main_review: main_review: # Summary

This paper provides a simple and elegant solution to continual learning under the assumption that data are provided in batches.
The core idea is that a separate, efficient probabilistic model is built for each batch (in this case, the model is a scalable and efficient Gaussian process classification representation). In particular, it is important that the (predictive, marginal) likelihood can be evaluated efficiently for each model. Upon arrival of a new batch, the weights for each previous batch (=model) are updated using Bayes rule (à la Bayesian model averaging). Prediction for a new batch / data point is then performed using these weights and the previous models.

# Evaluation

To my knowledge, while the Bayesian approach to continual learning is very common, the paper's idea of considering each batch predictor as a separate model and then effectively applying Bayesian model averaging is novel and original. This approach is significant in that this is a well-known open problem and the proposed method can seemingly apply to any domain in which we can efficiently build and update a probabilistic model that summarizes each batch. The quality is acceptable for a paper workshop -- the idea is good but the empirical evaluation is extremely limited to a small toy model. The paper is generally clear, but it seems it has been written hastily (several typos, a few sentences whose English could be improved).

# Comments

A few comments and perhaps limitations worth exploring or stating better in a future version of the paper:

1. It seems the method as proposed will critically depend on the quality of the feature extractor. Basing the extractor entirely on the first batch, as suggested, seems dubious? Regardless of how the extractor is trained, since it is fixed, there are concerns whether the extractor will still deal successfully with concept drift. I understand that the authors assume $p(X)$ is stationary, but my intuition is that the feature extractor might struggle if it sees $(x, y)$ combinations never encountered before, especially in higher dimension; it'd be good to see some empirical exploration. The $2D$ example in the paper does not use a feature extractor.
2. While it is true that the method does not store individual data points, it still stores a summary of each batch since it needs to keep one model per batch (in this case, the summary is represented by the scalable GP posterior). So the overall storage is still $O(N)$, under the reasonable assumption that the number of batches is proportional to the number of data points. The "summary" models might be quite large. Still, I see that storing summaries is better than storing individual data points.
3. The paper hinges on the idea that data arrives in batches, which may or may not be the case. Of course, any sequence of streaming data can be arbitrarily divided into batches, but then there is the question of how big each batch should be. The authors might want to explore this and give maybe some heuristic recommendations (if not a fully principled solution).

# Conclusions

Overall, this is a nice paper with an interesting idea that deserves further exploration and discussion, and very suitable in terms of topic for the workshop, and this is what I am mostly using as a base for my final rating. I don't give a higher score because the execution and demonstration is very limited, even for a workshop submission. I am looking forward to a full paper version with more extensive experiments.