Detecting and Adapting to Irregular Distribution Shifts in Bayesian Online Learning

1 Introduction . Deployed machine learning systems are often faced with the problem of distribution shift , where the new data that the model processes is systematically different from the data the system was trained on [ Zech et al. , 2018 , Ovadia et al. , 2019 ] . Furthermore , a shift can happen anytime after deployment , unbeknownst to the users , with dramatic consequences for systems such as self-driving cars , robots , and financial trading algorithms , among many other examples . Updating a deployed model on new , representative data can help mitigate these issues and improve general performance in most cases . This task is commonly referred to as online or incremental learning . Such online learning algorithms face a tradeoff between remembering and adapting . If they adapt too fast , their performance will suffer since adaptation usually implies that the model loses memory of previously encountered training data ( which may still be relevant to future predictions ) . On the other hand , if a model remembers too much , it typically has problems adapting to new data distributions due to its finite capacity . The tradeoff between adapting and remembering can be elegantly formalized in a Bayesian online learning framework , where a prior distribution is used to keep track of previously learned parameter estimates and their confidences . For instance , variational continual learning ( VCL ) [ Nguyen et al. , 2018 ] is a popular framework that uses a model ’ s previous posterior distribution as the prior for new data . However , the assumption of such continual learning setups is usually that the data distribution is stationary and not subject to change , in which case adaptation is not an issue . This paper proposes a new Bayesian online learning framework suitable for non-stationary data distributions . It is based on two assumptions : ( i ) distribution shifts occur irregularly and must be inferred from the data , and ( ii ) the model requires not only a good mechanism to aggregate data but 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . also the ability to partially forget information that has become obsolete . To solve both problems , we still use a Bayesian framework for online learning ( i.e. , letting a previous posterior distribution inform the next prior ) ; however , before combining the previously learned posterior with new data evidence , we introduce an intermediate step . This step allows the model to either broaden the previous posterior ’ s variance to reduce the model ’ s confidence , thus providing more “ room ” for new information , or remain in the same state ( i.e. , retain the unchanged , last posterior as the new prior ) . We propose a mechanism for enabling this decision by introducing a discrete “ change variable ” that indicates the model ’ s best estimate of whether the data in the new batch is compatible with the previous data distribution or not ; the outcome then informs the Bayesian prior at the next time step . We further augment the scheme by performing beam search on the change variable . This way , we are integrating change detection and Bayesian online learning into a common framework . We test our framework on a variety of real-world datasets that show concept drift , including basketball player trajectories , malware characteristics , sensor data , and electricity prices . We also study sequential versions of SVHN and CIFAR-10 with covariate drift , where we simulate the shifts in terms of image rotations . Finally , we study word embedding dynamics in an unsupervised learning approach . Our approach leads to a more compact and interpretable latent structure and significantly improved performance in the supervised experiments . Furthermore , it is highly scalable ; we demonstrate it on models with hundreds of thousands of parameters and tens of thousands of feature dimensions . Our paper is structured as follows : we review related work in Section 2 , introduce our methods in Section 3 , report our experiments in Section 4 , and draw conclusions in Section 6 . 2 Related Work . Our paper connects to Bayesian online learning , change detection , and switching dynamical systems . Bayesian Online and Continual Learning There is a rich existing literature on Bayesian and continual learning . The main challenge in streaming setups is to reduce the impact of old data on the model which can be done by exponentially decaying old samples [ Honkela and Valpola , 2003 , Sato , 2001 , Graepel et al. , 2010 ] or re-weighting them [ McInerney et al. , 2015 , Theis and Hoffman , 2015 ] . An alternative approach is to adapt the model posterior between time steps , such as tempering it at a constant rate to accommodate new information [ Kulhavỳ and Zarrop , 1993 , Kurle et al. , 2020 ] . In contrast , continual learning typically assumes a stationary data distribution and simply uses the old posterior as the new prior . A scalable such scheme based on variational inference was proposed by [ Nguyen et al. , 2018 ] which was extended by several authors [ Farquhar and Gal , 2018 , Schwarz et al. , 2018 ] . A related concept is elastic weight consolidation [ Kirkpatrick et al. , 2017 ] , where new model parameters are regularized towards old parameter values . All of these approaches need to make assumptions on the expected frequency and strengh of change which are hard-coded in the model parameters ( e.g. , exponential decay rates , re-weighting terms , prior strengths , or temperature choices ) . Our approach , in contrast , detects change based on a discrete variable and makes no assumption about its frequency . Other approaches assume situations where data arrive in irregular time intervals , but are still concerned with static data distributions [ Titsias et al. , 2019 , Lee et al. , 2020 , Rao et al. , 2019 ] . Change Point Models There is also a rich history of models for change detection . A popular class of change point models includes “ product partition models ” [ Barry and Hartigan , 1992 ] which assume independence of the data distribution across segments . In this regime , Fearnhead [ 2005 ] proposed change detection in the context of regression and generalized it to online inference [ Fearnhead and Liu , 2007 ] ; Adams and MacKay [ 2007 ] described a Bayesian online change point detection scheme ( BOCD ) based on conditional conjugacy assumptions for one-dimensional sequences . Other work generalized change detection algorithms to multivariate time series [ Xuan and Murphy , 2007 , Xie et al. , 2012 ] and non-conjugate Bayesian inference [ Saatçi et al. , 2010 , Knoblauch and Damoulas , 2018 , Turner et al. , 2013 , Knoblauch et al. , 2018 ] . Our approach relies on jointly inferring changes in the data distribution while carrying out Bayesian parameter updates for adaptation . To this end , we detect change in the high-dimensional space of model ( e.g. , neural network ) parameters , as opposed to directly in the data space . Furthermore , a detected change only partially resets the model parameters , as opposed to triggering a complete reset . Titsias et al . [ 2020 ] proposed change detection to detect distribution shifts in sequences based on low-dimensional summary statistics such as a loss function ; however , the proposed framework does not use an informative prior but requires complete retraining . Switching Linear Dynamical Systems Since our approach integrates a discrete change variable , it is also loosely connected to the topic of switching linear dynamical systems . Linderman et al . [ 2017 ] considered recurrent switching linear dynamical systems , relying on Bayesian conjugacy and closed-form message passing updates . Becker-Ehmck et al . [ 2019 ] proposed a variational Bayes filtering framework for switching linear dynamical systems . Murphy [ 2012 ] and Barber [ 2012 ] developed an inference method using a Gaussian sum filter . Instead , we focus on inferring the full history of discrete latent variable values instead of just the most recent one . Bracegirdle and Barber [ 2011 ] introduce a reset variable that sets the continuous latent variable to an unconditional prior . It is similar to our work , but relies on using low-dimensional , tractable models . Our tempering prior can be seen as a partial reset , augmented with beam search . We also extend the scope of switching dynamical systems by integrating them into a supervised learning framework . 3 Methods . Overview Section 3.1 introduces the setup and the novel model structure under consideration . Section 3.2 introduces an exact inference scheme based on beam search . Finally , we introduce the variational inference extension for intractable likelihood models in Section 3.3 . 3.1 Problem Assumptions and Structure . We consider a stream of data that arrives in batches xt at discrete times t.1 For supervised setups , we consider pairs of features and targets ( xt , yt ) , where the task is to model p ( yt|xt ) . An example model could be a Bayesian neural network , and the parameters zt could be the network weights . For notational simplicity we focus on the unsupervised case , where the task is to model p ( xt ) using a model p ( xt|zt ) with parameters zt that we would like to tune to each new batch.2 We then measure the prediction error either on one-step-ahead samples or using a held-out test set . Furthermore , we assume that while the xt are i.i.d . within batches , they are not necessarily i.i.d . across batches as they come from a time-varying distribution pt ( xt ) ( or pt ( xt , yt ) in the supervised cases ) which is subject to distribution shifts . We do not assume whether these distribution shifts occur instantaneously or gradually . The challenge is to optimally adapt the parameters zt to each new batch while borrowing statistical strength from previous batches . As follows , we will construct a Bayesian online learning scheme that accounts for changes in the data distribution . For every new batch of data , our scheme tests whether the new batch is compatible with the old data distribution , or more plausible under the assumption of a change . To this end , we employ a binary “ change variable ” st , with st = 0 for no detected change and st = 1 for a detected change . Our model ’ s joint distribution factorizes as follows : p ( x1 : T , z1 : T , s1 : T ) = T∏ t=1 p ( xt|zt ) p ( zt|st ; τ t ) p ( st ) . ( 1 ) We assumed a factorized Bernoulli prior ∏ t p ( st ) over the change variable : an assumption that will simplify the inference , but which can be relaxed . As a result , our model is fully-factorized over time , however , the model can still capture temporal dependencies through the informative prior p ( zt|st ; τ t ) . Temporal information enters this prior through certain sufficient statistics τ t that depend on properties of the previous time-step ’ s approximate posterior . In more detail , τ t is a functional on the previous time step ’ s approximate posterior , τ t = F [ p ( zt−1|x1 : t−1 , s1 : t−1 ) ] .3 Throughout this paper , we will use a specific form of τ t , namely 1In an extreme case , it is possible for a batch to include only a single data point . 2In supervised setups , we consider a conditional model p ( yt|zt , xt ) with features xt and targets yt . 3Subscripts 1 : t− 1 indicates the integers from 1 to t− 1 inclusively . capturing the previous posterior ’ s mean and variance.4 More formally , τ t ≡ { µt−1 , Σt−1 } ≡ { Mean , Var } [ zt−1|x1 : t−1 , s1 : t−1 ] . ( 2 ) Based on this choice , we define the conditional prior as follows : p ( zt|st ; τ t ) = { N ( zt ; µt−1 , Σt−1 ) for st = 0 N ( zt ; µt−1 , β−1Σt−1 ) for st = 1 ( 3 ) Above , 0 < β < 1 is a hyperparameter referred to as inverse temperature5 . If no change is detected ( i.e. , st = 0 ) , our prior becomes a Gaussian distribution centered around the previous posterior ’ s mean and variance . In particular , if the previous posterior was already Gaussian , it becomes the new prior . In contrast , if a change was detected , the broadened posterior becomes the new prior . For as long as no changes are detected ( st = 0 ) , the procedure results in a simple Bayesian online learning procedure , where the posterior uncertainty shrinks with every new observation . In contrast , if a change is detected ( st = 1 ) , an overconfident prior would be harmful for learning as the model needs to adapt to the new data distribution . We therefore weaken the prior through tempering . Given a temperature β , we raise the previous posterior ’ s Gaussian approximation to the power β , renormalize it , and then use it as a prior for the current time step . The process of tempering the Gaussian posterior approximation can be understood as removing equal amounts of information in any direction in the latent space . To see this , let z be a multivariate Gaussian with covariance Σ and u be a unit direction vector . Then tempering removes an equal amount of information regardless of u , Hu = 12 log ( 2πeu > Σu ) − 12 log β , erasing learned information to free-up model capacity to adjust to the new data distribution . See Supplement B for more details . Connection to Sequence Modeling Our model assumptions have a resemblance to time series modeling : if we replaced τ t with zt−1 , we would condition on previous latent states rather than posterior summary statistics . In contrast , our model still factorizes over time and therefore makes weaker assumptions on temporal continuity . Rather than imposing temporal continuity on a data instance level , we instead assume temporal continuity at a distribution level . Connection to Changepoint Modeling . We also share similar assumptions with the changepoint literature [ Barry and Hartigan , 1992 ] . However , in most cases , these models don ’ t assume an informative prior , effectively not taking into account any sufficient statistics τ t. This forces these models to re-learn model parameters from scratch after a detected change , whereas our approach allows for some transfer of information before and after the distribution shift .

Summary: This submission relates a method for adapting to distribution shifts online in a Bayesian fashion. A distribution shift is represented as a binary latent variable indicating that a shift has, or has not occurred. If a shift is inferred to have occurred, then the variance of the posterior is increased before the next step of the Bayesian recursion, weakening the influence of previous observations. Computational constraints associated with maintaining many possible hypotheses of the incidence of distribution shifts are remedied by the use of beam search.   This paper is timely and well-written. The topic of online adaptation to distribution shift is very relevant to the NeurIPS community, and the submission has developed through several rounds of peer review into a mature contribution to the literature. There are some substantial limitations that would be important to address in future work, but I am happy to recommend acceptance and willing to defend my evaluation.  UPDATE 08/17/2021 The author response has addressed all my concerns. In particular it is very good to know that the full changepoint history need not be maintained, and that the method will automatically adapt, in a sense, to a poor choice of \beta. I think this is a very good paper, and have increased my score to 8.