Kalman Filter for Online Classification of Non-Stationary Data
Authors: Michalis Titsias
Alexandre Galashov
Amal Rannen-Triki
Razvan Pascanu
Yee Whye Teh
Jorg Bornschein
Venue: ICLR 2024 poster (2024)
Keywords: online learning
non-stationarity
Kalman filter
continual learning
probabilistic modellling
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Kalman Filter based method applied to non-stationary classification problems with strong empirical performance

Abstract: In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps. Key challenges in OCL include automatic adaptation to the specific non-stationary structure of the data and maintaining appropriate  predictive uncertainty.  To address these challenges we introduce a probabilistic Bayesian online learning approach that utilizes a (possibly pretrained) neural representation and a state space model over the linear predictor weights. Non-stationarity in the linear predictor weights is modelled using a “parameter drift” transition density, parametrized by a coefficient that quantifies forgetting. Inference in the model is implemented with efficient Kalman filter recursions which track the posterior distribution over the linear weights, while online SGD updates over the transition dynamics coefficient allow for adaptation to the non-stationarity observed in the data. While the framework is developed assuming a linear Gaussian model, we extend it to deal with classification problems and for fine-tuning the deep learning representation. In a set of experiments in multi-class classification using data sets such as CIFAR-100 and CLOC we demonstrate the model's predictive ability and its flexibility in capturing non-stationarity.

Published as a conference paper at ICLR 2024
KALMAN FILTER FOR ONLINE CLASSIFICATION OF NON-STATIONARY DATA
Michalis K. Titsias' Google DeepMind
Alexandre Galashov* Google DeepMind a aga wwwassoooooolee...oo com
Amal Rannen-Triki
Google DeepMind
mtitsi asê google. com
arannen@googles com
Razvan Pascanu
Yee Whye Teh Google DeepMind yutehhggoogle.cm
Jörg Bornschein
Google DeepMind
Google DeepMind
razpegoogle.com
tornschhein@googee.com
ABSTRACT
In Online Continual Learning (OCL) learning system receives a stream of data and sequentially performs prediction and training steps. Key challenges in OCL include automatic adaptation to the specific non-stationary structure of the data and maintaining appropriate predictive uncertainty. To address these challenges we introduce probabilistic Bayesian online learning approach that utilizes (possibly pretrained) neural representation and state space model over the linear predictor weights. Non-stationarity in the linear predictor weights is modelled using a "parameter drift" transition density, parametrized by a coefficient that quantifies forgetting. Inference in the model is implemented with efficient Kalman filter recursions which track the posterior distribution over the linear weights, while online SGD updates over the transition dynamics coefficient allow for adaptation to the non-stationarity observed in the data. While the framework is developed assuming linear Gaussian model, we extend it deal with classification problems and for fine-tuning the deep learning representation. In set of experiments in multi- class classification using data sets such as CIFAR-100 and CLOOC we demonstrate the model's predictive ability and its flexibility in capturing non-stationarity.
INTRODUCTION
Continual Learning (e.g. Hadsell et al.) 2020 Parisi et al.) 2019 is an open problem that has been receiving increasing attention in recent years. It aims to provide answers on how train and use models in non-stationary scenarios. A multitude of different and sometimes conflicting desiderata have been considered for continual learning, including forward transfer, backward transfer, avoiding forgetting and maintaining plasticity. Training and evaluation protocols highlight different constraints such limited memory to store examples, limited model size or computational constraints.
In this work we focus on the Online Learning (OL) (Shalev-Shwartz, 2012 Hazan 2017) scenario, where learner receives a sequence of inputs Hn and targets ya At each step, the model observes In. makes a prediction and then receives the associated loss and ground truth target for learning. Within the deep learning community this scenario has been referred to as Online Continual Learning (OCL) (Cai et al. 2021 Ghunaim et al. 2023), with the focus shifted more towards obtaining the best empirical performance on some given data instead of bounding the worst-case regret. Note that in OL/OCL each observation (z,,, Mn)) is first used for evaluation before it is used for training. No separate evaluation-sets are required and this objective naturally supports task-agnostic and non-stationary scenarios. Depending sample-efficiency all play a crucial role during learning. When considering the cumulative next-step log-loss under this protocol, it directly corresponds the prequential description length and is thus a theoretically well motivated evaluation metric for non-stationary scenarios under the Minimum Description Length principle (Grunwald, 2004 Blier and Ollivier 2018 Bornschein et al. 2022b)
the nature of the data stream, plasticity, not-forgetting and
"Joint first authorship

---

Published as a conference paper at ICLR 2024
In this paper, we propose new method based on Kalman filters which explicitly takes into account non-stationaries in the data stream. It assumes a prior Markov model over the linear predictor weights that uses a "parameter drift" transition density parametrized by coefficient that quantifies forgetting. The prior model is then combined with observations using online Bayesian updates, implemented by computationally fast Kalman filter recursions, that track the posterior distribution over the weights of the linear predictor as the data distribution changes over time. These Bayesian updates are also combined with online SGD updates over the forgetting coefficient, thus allowing for more flexible adaptation to non- stationarity. While the method is developed assuming the tractable linear Gaussian case, we also extend to classification and for updating deep learning representations.
We follow the trend introduced by large scale pretrained or foundation models (Bommasani et al. 202T) of separating the representation learning from the readout classifier. Foundation models provide generic representations applicable across multitude of tasks and have been argued to offer stable representations in continual learning, shifting the focus to handling non-stationarity with the convex readout layer. This separation of concerns facilitates a more natural integration of non-neural solutions to address continual or online learning challenges. Following this perspective, we investigate the performance of Kalman filter on top of frozen pretrained representation. We demonstrate that simultaneously online learning of the representation while performing Kalman filter updates leads to stable learning and provide strong results on online continual learning benchmarks.
Our contributions can be summarised as follows: (i) We propose a Kalman filter based approach which models the parameters drift in the last layer of the Neural Network (NN) and derive efficient Bayesian inference updates. (ii) We introduce an online learning algorithm to learn the drift parameters online therefore adjusting the last layer linear dynamics to the non-stationarity of the data. (iii) We demonstrate that this approach can work with pre-trained features as well as when it is trained from scratch, allowing for training NN representations online. (iv) We demonstrate that the approach achieves superior performance on large-scale benchmarks against existing baselines.
2 ONLINE LEARNING WITH KALMAN FILTER
We first describe the method in univariate regression and discuss in Section 2.1 how we adapt it to classification. We assume a stream of data arriving sequentially so that step n we receive (Zn: Yn) where 2,, E RR is the input vector and yn. E R is the target. Our objective is to introduce an online learning model that captures sonsstationariiiss and learns good predictor for future data, Our model consists of two parts. The fist part is neural network that outputs a representation ((x;0) E Th' so that Φn == (r,,; 0) is the feature vector of the n-th data point. 0 is a set of parameters that could be fixed, if the feature extractor is pretrained network, learnable, when it is either fine-tuned or learnt from scratch. Given on the output yn is modelled by a Gaussian likelihood p(y,, p(,,|w,,) N [w, o²), where the me-dimensional vector of regression coefficients Wn depends on time index n, i.e. Wi, can change with time in order to model distributional changes.
The second part of the model is Gaussian Markov prior process on how W.,, can change over time, which models parameter drift based on the following dynamics,
Initial parameter pw (w,-1) NNw,, wn (1-12)oa1) n 1. Parameter drift
p(w)) (Nooll I.
(2)
The time-dependent parameter Yh takes values in [0, 1] and quantifies the memory or forgetting of the process. For example, when 7n 1 then wn, wn-1, which means that at time step n the model re-uses (or copies forward) the parameter -n-1 from the previous step. Such extreme case is suitable when there is no distributional change at time n. In the other extreme case, when 7, 0 0 the parameter w,, is fully refreshed, i.e. reset to the prior (00 a I), which implies a sharp change in the distribution. Similarly, intermediate values of 7n E (0, 1) can model more smooth or gradual changes. Flexible learning of 7n through time will be a key element of our method that we describe in Section 1 for the classification problem. Furthermore, it is worth noting that the transition model in equation 2 is invariant to the prior, that when Wh-1 (O, o the next state Wn = "/ "n-1" + v The (e ~ (O0 7)) also has marginal distribution A NO(0, o
7). Having specified the observation model and the transition model over the parameters Wn, we can write the full joint density up to the n-th observation as
p(w)) II p(ya (us )p (aspw///u (w;-1) = (woo00, o II Nuslu Φz, (o:oo"([[ha (1 (1-22)02) (o
2

---

Published as a conference paper at ICLR 2024
This is linear Gaussian state space model where exact online Bayesian inference over wn can be solved with standard Kalman filter prediction and update steps to compute online the Bayesian posterior over w, (Särkkä 2013). The prediction step requires computing the posterior over Wn given all past data up to time n (and excluding the current n-th observation), which is a Gaussian density p(ten 11)m (w,, mm A,r) with parameters
A, = An 1
m
The update step finds the updated Gaussian posterior p(w,, V1n) = NWw, mn, An) by modifying the mean vector my and the covariance matrix An to incorporate the information coming from the most recent observation (Ino Un) according
A on
A A A (Un - m,); An = A 0² TT On
m2 = m, +
(4)
x² An On
The initial conditions for the recursions are ma 0 and A = 12.1. The complexity is dominated by the operations 8n1 = A on and Sn which are (mm²) making the Kalman recursions very efficient. Further, if we remove the stochasticity from the transition dynamics by setting Yn for any n, the above Kalman recursions reduce to online Bayesian linear regression as detailed in Appendix A
Prediction. As the online model sequentially receives observations and updates its Bayesian posterior distribution over Wnz it can also perform next step predictions. Suppose that after n 1 steps the model has observed the data (2,, y:) and computed the posterior pl Wn (1m-1).. Then for the next input data nn the model can predict its output Yn based on the Bayesian predictive density
p(yn 181
p(yn (on (ww) (pwp://(wwm/mmmmee 11mm1) = (Vmll,, mn 0, A, on + +o²),
(5)
which analytic for this Gaussian regression case, while for the classification case will require approximate inference and Monte Carlo sampling as detailed in Section 2.1.1
2.1 APPLICATION TO CLASSIFICATION
We now apply the above online learning model to multi-class classification problems. Suppose classification problem with K classes, where the label yn encoded as K K-dimensional one-hot vector, i.c. yn E {0, 1] Ek Mn.k = 1. suitable observation model for classification is the standard softmax likelihood p(yn, 1(W,)
where W,, (Wa,l..... Wh,k) is 2n
m X K matrix storing the parameters Wn, which play similar role previous regression coefficients. These parameters follow K independent Markov processes so that each k-th column w,,, of W,, independent from the other columns and obeys the transition dynamics from equation and equation 2 However, unlike the regression case, exact online inference over Wn, using Kalman recursions intractable due to the non- -on-Sassiian form of the softmax likelihood. Thus, we need to rely on approximate inference. Next we derive a fast and very easy to implement inference technique that still uses the exact Kalman recursions. This consists of two main components described next.
(i) Fast and analytic Kalman recursion. To achieve this we introduce Gaussian likelihood that (as means to approximate inference) replaces the softmax likelihood. It has the form ( = T N(nn, w Pn. o²), which explains the elements of the one-hot vector by Gaussian density, trick that has been used successfully in the literature e.g. for Gaussian process classification [Rasmussen and Williams| 2006) and meta learning (Patacchiola et al., 2020). With this approximate likelihood the Kalman recursions remain tractable and propagate forward approximate predictive
posterior (W,, II N(wn,slm, A,)) with mean parameters given by the m K matrix M, (m.,,..... . k and covariance parameters given by the m x m matrix AÇ:
m,,k)
M = M1-1-1 A, = ? An + I
(6)
The corresponding updated posterior (Ww//1i. -I NWwn, Minkk An) has parameters
on
A O A
M2 = M + 0² + TT An
x (v - 07 M,,,)
An = A
(7)
0² T An On
3

---

Published as a conference paper at ICLR 2024
The recursion is initialized Mo = A, o where 0 denotes the x K matrix of zeros. A full iteration costs O(Km + m²) where the (mm²) operations are the same with the regression case, while the additional ((m) operations are needed in the update of Mn, such as for 0, M, If the feature vector size m is larger than the number of classes K, the term (m²)) dominates and the complexity is the same as in univariate regression. The crucial factor obtain such efficiency is that the covariances matrices overhead of the method is ((Km) for storing M,, and (mm²) for storing the covariance An.
are shared among all K classes| |
The additional memory
(ii) Posteriors combine with softmax for prediction or parameter updating. We view the above Kalman recursion as an online approximate inference procedure that provides an approximation to the exact posterior distribution. If p( W,, PW,,.lyinn-- -1...n.m) is the exact intractable predictive posterior (obtained by Bayes' rule with the exact softmax likelihood) then (W..1b1sn-1) computed from Kalman recursion an approximation p 1.) Subsequently, by following standard approximate Bayesian inference practices, when we wish to predict class probabilities or compute a cross entropy-like loss to optimize parameters, the approximate posterior is combined with the exact softmax through Bayesian averaging and Monte Carlo estimation. In Section I we make use of this to learn online the forgetting coefficient "'n Further in Appendix C we use the same principle to compute accurate predictive probabilities and fine-tune the representation parameters 6. For this second case also find it useful to introduce a Bayesian calibration procedure, which optimizes online a calibration parameter and improves the predictive probability estimates; see Appendix [[ full details.
(iii) Online backbone finetuning. We also allow for finetuning the backbone o( 8) online; Appendix G In this case, the Kalman recursion will not be exact in the sense that it combines features ((an, 0) of different 0 values. Nevertheless, we found that such approach works well in practice.
2.1.1 ONLINE UPDATING THE FORGETTIGG COEFFICIENT "/n
An important aspect of our method is the online updating of the forgetting coefficient Yno which is indexed by n to indicate that it can change over time to reflect the data distributional changes. Intuitively, if the predictive probability of the next data point becomes low, this suggests that the Bayesian posterior does not explain this point well and needs refreshment from the prior. Adjusting Ihe will move the posterior either closer or further away from the prior as can be seen from equation 6 Instead of using full Bayesian approach (that will require extra hierarchical prior) over 7/n will follow a simple empirical Bayes method where we update "/n by online point estimation. For that, we first initialize to and then for any subsequent time step 2 1, where observe (Ii., Un). we first copy the previous 7n-1 value to the new step, i.e. Tn 7n- and then we apply an SGD update to change "/n by maximizing the log predictive probability. The SGD update is written as
7/ f Yo + Pr V Tm log logn(yn,k 1 11)mm-1),
where we further parametrize Th = 0.56,, with 8n 2 0, so that the update applied to 6n However, since the log predictive probability is not tractable we consider the approximation
log Plynk = 11v1m-1) ss log f plyn, = 1IW,)a (wwW:/WWWWnW)eeeee 111l 1))
where p(y, ((y,,W,)) is the softmax and ([W.lyim-1)) = - ((wn,k/m m An) the Kalman analytic approximate posterior distribution. To estimate this we can use the standard procedure to reparametrize the integral in terms of the K-ddimensional vector of logits fn = W ón which follows the factorized Gaussian distribution qf (n,, 12 with mean H., (M,) on and isotropic variance s Φ A Φn. After this reparametrization, equation 9
leads to the estimate
log p(yn,k 1 1 1lyia--1) = log / efaa Σj efa, aSm Imm
2 Dds Rs log
E tene) 107 (10)
where c(*) ~ (0, I), and to obtain the final expression we first reparametrize the integral to be an expectation under the standard normal and then apply Monte Carlo. To update 7n2 using SGD,
'This is hecause the hyperparameter x² in the approximate Gaussian likelihood is shared among all K dimensions. If we choose of per class the time and storage cost grows to O(Km²) which is too expensive. 'The hard constraint 6,, 2 0 is imposed through elipping.
4

---

Published as a conference paper at ICLR 2024
Algorithm 1 Kalman filter online learning
Input: Data stream (In, n representation d(x, 0); parameters 6: hyperparameters (o², o Initialise Mo 0, Ag = aid I and ón (e.g. to value 0.0 so that " = 1)
for data point n 1, 2,2,3, ..... do
Observe Zn. compute di, <(Tn, 0) make prediction for the label vā via equation 13 Observe true class label 4/+
(Optional) Fine-tune 0 together with the calibration parameter a, described in Appendix Update 6n as described in Section 2. 1.1 and set Yn
= -0.56.
Mn M- An An- + (1 T
Update Kalman statistics: Mn- - Mj -2+0 An On
A. On
x (VT M,,, An - An 02+0, An on
An ono A.
end for
wrt the parameter 6n in Th e -0.56., where "n appears in (Mm, si))) through the computation M, = 7n M1 A = An-1 + o (1 - I, while the parameters (Mn-1, An-1) 1) of the posterior up time n 1 taken as constants. Monte Carlo (MC) approximation can run efficiently by pre-computing Ko.kk and 8n which overall takes O(Km + m²) time. Since these are scalar values, MC approximation complexity takes only O(KS) time. Moreover, we found that significantly increasing S did not make much difference in practice in terms of the variance of this approximation and we found that the moderate value S 50 worked well in practice for the settings we considered.
differentiate this final Monte Carlo estimate of the loss plv. - 11yim-
RELATED WORK
Online learning with Bayesian forgetting. An alternative framework for non-stationary online learning can be based the concept of Bayesian forgetting (BF) (Kulhavý and Zarrop 1993 Honkela and Valpola 2003 Graepel et al. 2010). as employed recently by Moens (2018) and Kurle et al. (2020); see also al. (2021) for a related tempering based approach. The main idea allow some forgetting of the current posterior p( w (B)...) by creating a weighted geometric average with the prior, i.e. as Par(w] (pr(w)y)m) x p w (g
) "n po(w)) 1-7a (and re-normalize to get a new posterior for the next time step) where 7/ E [0, 1] is the forgetting factor. Most methods use non-learnable exponentially decaying values for the sequence "h. while Mooles (2018) considers online variational inference. The BF approach differs significantly from the proposed Kalman filter (KF) algorithm where forgetting is achieved through convolving with the transition dynamics p(w [/i=n) = J p(w [w)p((u w 8/.:n ddw. In Appendix(D) [ we derive new algorithm following the BF principle that learns "/n online with SGD and we include this BF method in our experiments as an additional baseline compared to KF. An important computational difference between BF and KF is that, as shown in Appendix [ BF has cubic cost (mm²) per iteration as opposed to low quadratic O(m²) cost of the Kalman filter method.
Related continual learning methods. Classical approaches for continual learning have focused on reducing forgetting (De Lange al., 202 Parisi et al.) 2019 Mai et al., 2022). More recently.
the trend moved the focus towards forward transfer and efficient adaptation (Hadsell et al.) 2020
Bornschein al. 2022a Ghunaim et al. 2023 Hayes and Kanan 2022). While significant progress has been achieved through this line of research, one major drawback, highlighted by different recent works (Ghunaim al. 2023 Cai et al. [202]] Caccia et al. 2022), is the focus on small-scale artificial benchmarks with abrupt and unnatural distribution shift, and on metrics that fail to capture the capability of the models to efficiently adapt the non-stationarity in the input data. In this work, we focus on fast adaptation and the next-step prediction problem from OL/OCL.
Different recent works considered the problem of fast adaptation and the set of constraints and objectives that are realistic in an online scenario. Different recent benchmarks, c.g. CLOC (Cai et al. 202Tb and CLEAR (Lin et al. 2021), ,. propose a sequence of temporally sorted images with naturally shifting visual concepts. CLOC leverages a subset of 39M images from YFCC100M (Thmeee et al.) 2016) along with their timestamps and geolocalisation tags, spanning a period of 9 years. The paper also highlights the limits of the classical continual learning approaches, and proposes simple baselines to overcome them, based mostly on adapting the online learning rate and the replay buffer size. They focus on an online evaluation protocol where each mini-batch is used for testing before adding it to the training dataset. In this work, we largely follow the CLOC setting. We evaluate our approach on the CLOC data and on modified CIFAR 100 versions that are inspired by it.
5

---

Published as a conference paper at ICLR 2024
Bornschein et al. (202bb) highlight the theoretical and practical benefits of using the online next-step performance for model evaluation. They emphasize the connection to compression based inference and to the prequential Minimum Description Length principle (Dawid and Vovk, 1999| Poland and Hutter 2005). Empirically, they evaluate SGD based techniques on the pareto-front of prediction loss vs. computational requirements, and propose forward-calibration and specific rehearsal approaches to improve results. We compare against their methods in our experiments on CLOC.
The authors Ghunaim al. (2023) base their work on the observation that CL approaches have been developed under unrealistic constraints, allowing for offline learning on data with multiple passes without any limitation of computational cost. In realistic settings where we aim at updating a model on continuously incoming data, approaches that are slower than the stream would be impractical, or reach suboptimal performance in the limited time and computational budget. The authors propose an evaluation protocol that puts the computational cost at the center. They propose adding a delay between model update and evaluation. This delay is related the computational cost of the updating method. Under the same budget, a twice more expensive approach would update the model half as often. With this realistic evaluation, the authors show that a simple baseline based on experience replay outperforms state-of-the- art CL methods, due their complexity. In this work, we follow the protocol described in Ghunaim et al. (2023). Our approach has a relative complexity that requires no delay between training and evaluation. We therefore compare against the baselines reported in Ghunaim et al. (2023) that have the same properties, namely, ER (Chaudhry et al. 2019). ER++ GGhunaim et al. 2023) and ACE (Caccia al. 2022).
Finally, our method of learning Y online could be seen as a way to perform change-point detection. However, unlike usual change-point detection methods (Pagel 1957 Hawkins et al. 2003 Adams and MacKay| 22077] Fearnhead] (2066), learning 7 does not perform the hard change of the posterior parameter statistics, but rather operates in soft way, gradually making the parameters to reset.
Related Kalman filter methods. The idea of relying on Kalman filters to estimate the posterior distribution over the weights of a neural networks dates back to the 90s, with work such as (c.g Singhal and Wu 1988. Feldkamp et al. 1998. Puskorius and Feldkamp 1994). The focus in these works is to accelerate learning by incorporating second order information in the step size, and the setting typically considered is that of stationary learning. Kalman filter for dealing with online learning has been explored in the linear case for example in Tsiamis and Pappas (202)); Kozdoba al. (201)). In contrast, in our work we focus on the ability of Kalman filter to help with the online continual learning problem by relying on pretrained representations. In the concurrent work Chang al. (2023), the authors proposed use extended Kalman filter (EKF) directly on top of the neural network parameters assuming a similar drift in parameters as we did. One main difference with this work is that we only apply Kalman filter on the last layer and we assume that the covariance matrix is shared across all classes for the classification problems. This makes our method more scalable since it scales linearly with the number of classes and is only quadratic with respect to the dimensionality of the last layer. The method in Chang et al. (2023) scales as O(P(L + K K)² to update statistics, where P is the number of parameters in the neural network, is at the order of 10 and K the number of classes, and requires O(PL2 + L²) memory, which makes it less scalable variant. Moreover, unlike Chang et al. (023), in our work we also learn the forgetting parameter online.
4
EXPERIMENTS
4.1
ILLUSTRATIVE TIME SERIES EXAMPLE
We first apply our method to artificial time series data. The task is to track non-stationary data stream of scalar noisy observations yn, without any conditioning input I,, We further assume very simple model where the feature is just an univariate constant value equal to unity, i.e. Φ,, so that the observation likelihood simplifies as p(yn [w.) = NN(yn inferred through time, models the unknown expected value of yn-
[ym/ [w,, o²) and the parameter w.., to be
Figure I shows the results of the Kalman model that was initialized with To = 1 and learns it online, as described in Appendix B The non-stationary nature of the this series is such that the signal is piece-wise (noisy) constant with seven change-points. As shown by the second row in the left panel in Figure T the learned value of 7n is able to adjust to this non-stationarity by dropping the value of Tn quite below the value one (in order refresh the Bayesian statisties over w,, any time there is change-point Occasionally, Th drops outside the task boundaries which could be explained by the outliers in the data. The drops of 7/2 at the task boundaries, however, are more durable giving the
6

---

Published as a conference paper at ICLR 2024
Lag prodiettve density
0
22
0.8
Learning
Fixed
0.6
500 1000
1500
2000
2500
3000
1000 2000
3000
Time
Time
Figure 1: Artificial time series example of 3058 observations. Top in left panel shows the data (black dots) and predicted mean and uncertainty (orange lines) over ya (as arrive sequentially from to right and we perform online next step prediction), while the bottom row shows the optimized values of 7o = é Right panel shows the accumulated average log predictive density, i.c. >> log p(ys (1..-1). computed across time for the model that learns "/n and the model that ignores non non statiioaao by setting " = 1 for all
posterior time to re-adjust to the distribution changes. In contrast, if we remove the ability to capture non-stationarity, ie. by setting n-2 for all n, the performance gets much worse as shown by the accumulated log predictive density scores in Figure and by Figure T in the Appendix
4.2 ONLINE CLASSIFICATION
In this section, apply method online classification and consider the OCL scenario where data arrive in small batches (chunks), see Cai et al. (202T); Bornschein et al. (2022b); Ghunaim al. (2023): The learning algorithm is exposed to data stream S such that every time step n, chunk of data S, = {(In,b+s [ of size b is revealed. The algorithm first predicts the labels + (nb 1), ........ (n 1)6, and then updates its parameters (see Algorithm
D based on this chunk data. As metric, we use Average Online Accuracy, (n)) nb Eg acc(ya, Vs), where y, is the ground truth label and ÿ. is the prediction of the model. This metric contains accuracies computed on the fly over training and quantifies how well the algorithm ingests new knowledge.
Ya: s= (nb
Table 1: CIFAR-100 results. The numbers in bold correspond to the best performing method in the group.
Average Online Accuracy
Method
Stationary CIFAR- 100
Non-stationary CIFAR- 100
No backbone finetuning (Purely linear model)
Stationary Kalman Filter (y 1.0)
11.63% = 0.2%
13.73 0.9% 31.9% + 0.8%
Non stationary Kalman Filter (fixed 7 = 0.999)
9.1% 0.1%
Non non-stationary Kalman Filter (learnd 7)
11.24% + 0.1%
33.3 + 0.3%
Backbone finetuning
Stationary Kalman Filter (r 1.0)
17.24 + 0.2%
47.37% + 0.9%
Non-stationary Kalman Filter (fixed 7 0.999)
16.17% + 0.2%
53% + 0.9%
Non-statinn stanssaatonnry Kalman Filter (learned 7)
17.28% + 0.2%
53.47 1%
Backbone finetuning with Replay
Stationary Kalman Filter (ry = 1.0)
19.45% + 0.4%
52.6% + 1%
Non-stationary Kalman Filter (fixed y = 0.999)
18.75% 1 0.1%
56.53% 1 1%
Non-stationary Kalman Filter (learned
19.82% 1 0.3%
57.13% 1 0.7%
Baselines
Online SGD Online SGD+ Replay
16.25% + 0.1%
49.5% + 0.7%
16.88% + 0.1% 18.45%
54.5% + 1%
ER+ (Ghunaim et al. 2023
4.2.1 ONLINE CLASSIFICATION ON CIFAR-100
We evaluate the performance of the Kalman Filter two variants of online classification CIFAR- 100 (Krizhevsky and Hinton 2009): stationary online classifcation on CIFAR- /00 and non-statinn -tationny online classifaction
CO0AR-100. In the stationary case, follow the protocol described in Ghunaim et al. (2023) where the stream constructed by randomly shuffling CIFAR- 100 dataset and split into chunks, that the learning algorithm does pass through CIFAR-100. This is
7

---

Published as a conference paper at ICLR 2024
similar to One-Pass ImageNet (Hu et al. non-stationarity. In the non-statioonn staniiaaton case, we consider a task-agnostic class increeeealt version of Split-CIFARIA 100 as in Lee et al. (2020), where CIFAR-100 is split into 10 tasks (task identity is not communicated to the learning algorithm) each containing 10 different classes, concatenated into a stream and split into chunks. At any time, the learning algorithm solves multi-classificttion problem with 100 classes. In this setting, there is very distinct non- non-stationarity related to the task changes. This benchmark also studied by many class incremental continual learning approaches which focus on alleviating catastrophic forgetting (French [999] McCloskey and Cohen 1989) and using average incremental accuracy as metric. Dealing with catastrophic forgetting goes beyond the scope of this work and we only focus on average online accuracy. In both cases, we follow training/evallutiion protocol described in Ghunaim et al. (2023) dealing with chunks of size 10, using ResNer- 18 (He et al. 2015) as feature extractor which is randomly initialized. For more details, see Appendix
2021) benchmark. Since it is randomly shuffled, there is no
The Kalman filter variants we consider are the following: Stationary Kalman Filter (7= 1), which could be seen form of Bayesian Logistic Regression, Non-stationary Kalman Filter with fixed y= 0.999 and Non-stationary Kalman Filter with learned y. On top of that, we study performance of Kalman filter in three regimes: no backbone finetuning, regime with fixed randomly initialized features 6 which is a linear model, backbone finetuning, a regime where features ø are learned online and backbone finetuning with replay, regime where on top of learning features Φ, also replay previously data. As first baseline we consider online SGD with and without replay, which consists in sequential SGD algorithm. Also, for the stationary CIFAR-100, we consider the external baseline ER++ from Chunaim et al. (2023), which is a version of Online SGD + Replay which does more update steps on the replay memory. This was the method with the highest performance as reported by Ghunaim et al.] 20233 The hyperparameters for methods are selected by choosing the variant with highest cumulative log probabilities following MDL principle from Bornschein et al. (2022b). Replay setting is similar to the one considered in Ghunaim et al. (0023. See Appendix for more details.
The results are given in Table T In the stationary CIFAR- 100 case, we observe that stationary Kalman filter provides a reasonable performance and is generally better than non-stationary Kalman filter with fixed 7. This is consistent with our intuition that there is not much non-stationarity to model and therefore we would not expect Kalman filter to help. We see that Kalman Filter with learning y leads to slightly better results than stationary case. Intuitively, it makes sense since in the worst case, this variant could revert back to y 1. Moreover, replay- Kalman filter leads to very competitive results against external baselines. Adding replay to Kalman improves results even further, beating ER+ baseline which uses much more replay than ER; see Ghunaim et al. (202)). Kalman filter performs consistently better than Online SGD in this setting. In the non-stationary 110
Evolution ofy
Averzge Online Accuracy
Task Accuracy
1.01
0.6
0.8
1.00
0.4
0.6
0.99
0.4
0.90
0.2
10000 30000 40000 50000
10000 30000 40000 30000 Data
10000 20000 30000 40000 50000
Online SGD
Non stationary Kalman Filter, fixed
Non-stationary Kalman Filter, learned
Stationary Kalman Filter,
Figure 2: Non-stationary CIFAR-100. Left- evolution of 7, center average online aecuracy, right- - smoothed task accuracy (5% of data). Dashed lines indicate task boundaries.
case, stationary Kalman filter performs consistently worse than its non-stationary variants. This is consistent with our intuition since in this case, there is non-stationarity which Kalman filter can capture. Moreover, again learning 7 generally leads to better performance. Figure 2 left, visualizes the dynamics of learning for the Backhone Finetuning setting, with red dashed lines indicating task boundaries. In many cases, 7 drops at task boundaries, pushing down probabilities of classes from previous classes and focusing more on future data. This is the desired behaviour of the method since it essentially captures non-stationarity. Furthermore, in some cases 7 could drop slightly outside of task boundaries which could be explained by the presence of outliers. This, however, does not make the performance of the method worse. On top of that, Online SGD also achieves strong performance despite being simple method. Kalman filter, however, achieves superior performance to Online
8

---

Published as a conference paper at ICLR 2024
From Scratch
Rom Pretrained
Online SGD
Online
Online SGD: With Replay
Kalman filter: Foed backbone
Kalman fitter: Finetuned backbone
ER
0.
0.0
Data seen
Figure 3: resutss. Left- learning from scratch, right- learning from pretrained model. External baselines are taken from Ghunaim et al.] (2023): ER and ACE (Caccia et al. [2022). On the left, Online SGD with replay and Kalman filter (KF) with finetuned backbone are overlapping, while KF performs much better on the right.
SGD. We see that replay improves performance for both methods and Kalman filter performance can be further improved if replay allowed. Figure right, we compare performance the Kalman filter (learned and finetuned backhone) against Online SGD. We see that overall Kalman filter achieves high accuracy on each task faster than Online SGD making this method more data efficient.
4.2.2 ONLINE LARGE-SCALE CLASSIFICATION CLOC
In CLOC (Cai et al. 202T), each image in a chronological data-sequence is associated with the geographical location where it was taken, discretized to 713 (balanced) classes. It is a highly non- stationary task on multiple overlapping time-scales because, e.g., major sports events lead to busts of photos from certain locations; seasonal changes effect the appearance of landmarks; locations become more or less popular over time; etc. We use the version of CLOC described in Bornschein al. p022bb; around 5% of the images could not be downloaded or decoded which leaves with a sequence of 37, 0,3,,0997999 images. This version of the dataset is similar to the one considered in Ghunaim et al. 2023)) ) but we are mindful of potential small differences due to the downloading errors. We follow the same protocol as in Ghunaim et al. (023) and in Bornschein et al. (2022b).
We use a ResNet-50 backbone and receive the data in chunks of 128 examples. For the Kalman filter, we consider the variant with learned y, which performed always better than any fixed one, including y 1. We either keep the backbone fixed, or finetune it. The case of fixed backbone corresponds to linear model only. For more details and hyperparameters selection, see Appendix F We run experiments where we either start learning on CLOC from scratch, or start with a LimaeNet--pretraine backbone via supervised loss. As baselines, we consider Online SGD with and without replay from Borncchein et al. (2022b), and compare to the results from Ghunaim et al. (023): ER Chaudhry et al. (2019) and ACE CCcccia et al. (2022), which were the best performing baselines on this setting as reported by Ghunaim et al. (2023). ). To produce the plots, we asked the authors in (Ghunaim et al. 2023) to provide the data from their experiments. The results are shown in Figure where Kalman filter provides very strong performance compared to the baselines. When learning from scratch, replay-free Kalman filter matches the performance of Online SGD with replay. This a strong result since Kalman filter does not need to store additional data in memory. Even having Kalman filter with fixed backbone performs much better than Online SGD. When starting from pretrained model, Kalman filter manages to learn more efficiently than any of the baselines. Moreover, finetuning the backbone provides a large boost in performance compared to using a fixed backbone. We see that in case of learning from scratch, using online SGD with Replay performs similarly to finetuning backbone together with a Kalman filter. We believe this is due to the fact that the random backbone parameters are too noisy and when combined with Kalman filter updates, the whole process still remains too noisy. The method, however, still performs significantly better than online gradient descent without replay. For more details about backbone finetuning, see Appendix G
CONCLUSION
We presented a probabilistic online learning method that combines efficient Kalman filter inference with online learning of deep learning representations. We have demonstrated that this method is able to adapt to non-stationarity of the data and it can give competitive next time-step data predictions. Some directions for future research are: Firstly, it would be useful to investigate other approximate inference methods for the classification case where we may construct a more accurate, but still computationally efficient, online Gaussian approximation to the softmax. Secondly, it would be interesting to extend the transition dynamics to include higher order Markov terms, which could increase the ficxibility of the algorithm to model more complex forms of non-stationarity and distribution drift.

---

Published as a conference paper at ICLR 2024
REFERENCES
Adams, R. P. and MacKay, J. (2007). Bayesian online changepoint detection. stat, 1050:19.
Blier, L. and Ollivier, Y. (2018). The description length of deep learning models.
Bommasani, R., Hudson, A., Adeli, E.,, Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch. S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K. A., Davis, J., Demszky, D., Donahue, C., Doumbouya, M.. Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L.., Finn, C., Gale, T., Gillespie, L. E., Goel, K., Goodman, N. D., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho. D. E., Hong. J., Hsu, K., Huang, J., Icard, T. F., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling. G., Khani, F., Khattab, O., Koh, P. W. Krass, M. S., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, C., Nilforoshan, H., Nyarko, J. F., Ogut, G., Orr, L. J., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Pous, C.,, Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y. H.., Ruiz, C., Ryan, J., R'e, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K. P., Tamkin, A., Taori, R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J., Zaharia, M. A., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., and Liang, P. (2021). On the opportunities and risks of foundation models. ArXiv, fbs//200.007258.
D., Newman, B., Nie, A., Niebles,
Bornschein, J., Galashov, A., Hemsley, R., Rannen-Triki, A., Chen, Y., Chaudhry, A., He, X. O., Douillard, A., Caccia, M., Feng, Q. et al. (2022a). Nevis 22: A stream of 100 tasks sampled from 30 years of computer vision research. arXiv preprint arXiv:2211.11747.
Borncchein, J., Li, Y., and Hutter, M. (2022b). Sequential learning of neural networks for prequential mdl. In The Kleventh International Conference on Learning Representations.
Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., and Belilovsky, E. (2022). New insights on reducing abrupt representation change in online continual learning.
Cai, Z., Sener, O., and Koltun, V. (2021). Online continual learning with natural distribution shifts: An empirical study with visual data.
Chang, P. G., Martin, G., Shestopaloff, A. Y., Jones, M., and Murphy, K. (2023). Low-rank extended kalman filtering for online learning of neural networks from streaming data.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. (2019), Continual learning with tiny episodic memories.
Dawid, A. P. and Vovk, V. G. (1999). Prequential probability: principles and properties, Bernoulli, pages 125-162.
De Lange M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. (2021). A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3 -37835566-3385
Fearnhead, P. (2006). Exact and efficient Bayesian inference for multiple changepoint problems. Statistics and Computing, 16(2):203-213
Feldkamp, L. A., Prokhorov, V., Eagen, C. F., and Yuan, F. (1998). Enhanced Multi-Stream Kalman Filter Training for Recurrent Networks, pages 29-53. Springer US, Boston, MA.
French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128-133
Ghunaim, Y., Bibi, A., Alhamoud, K., Alfarra, M., Hammoud, H. A. A. K., Prabhu, A., Torr, P. H., and Ghanem, B. (2023). Real-time evaluation in online continual learning: A new paradigm. arXiv preprint arXiv:2302.01047
Graepel, T., Candela, J. Q., Borchert, T., and Herbrich, R. (2010). Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft's 's bing search engine. In International Conference Machine Learning.
10

---

Published as a conference paper at ICLR 2024
Grunwald, P. (2004). A tutorial introduction the minimum description length principle.
Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. (2020). Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-1040
Hawkins, D. M., Qiu, P., and Kang. C. W. (2003). The changepoint model for statistical process control. Journal of Quality Technology, 35(4):355-366,
Hayes, T. L. and Kanan, C. (2022). Online continual learning for embedded devices. arXiv preprint wrXiv:2203.1068
Hazan, E. (2017). Introduction Online Convex Optimization. Foundations and Trends in Optimiza- tion. Now, Boston.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep residual learning for image recognition.
Honkela, and Valpola, H. (2003). On-line variational bayesian learning.
Hu, H., Li, A., Calandriello, D., and Gorur, D. (2021). One pass imagenet.
Kozdoba, M., Marecek, J., Tchrakian, T., and Mannor, S. (2019). On-line learning of linear dynami- cal systems: Exponential forgetting in kalman filters. In Proceedings of the Thirty-Thire AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelli- gence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'19AATT 11A11///A/////AAATTTT AAAI Press.
Krizhevsky. A. and Hinton, G. (2009). Learning multiple layers of features from tiny images. Technical Report University of Toronto, Toronto, Ontario.
Kulhavy, R. and Zarrop, M. B. (1993). On a general concept of forgetting. International Journal of Control, 58:905-924.
Kurle, R., Cseke, B., Klushyn, A., van der Smagt, P., and Günnemann, S. (2020). Continual learning with bayesian neural networks for non-stationary data. In International Conference on Learning Representations.
Lee, S., Ha, J., Zhang, D., and Kim, G. (2020). A neural dirichlet process mixture model for task-free continual learning.
Li, A., Boyd, A. J., Smyth, P., and Mandt, S. (2021). Detecting and adapting to irregular distribution shifts in bayesian online learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W., editors, Advances in Neural Information Processing Systems.
Lin, Z., Shi, J., Pathak, D., and Ramanan, D. (2021). The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Dalasets and Benchmarks Track (Round 2).
Mai, Z., Li, R., Jeong. J., Quispe, D., Kim, H., and Samner, S. (2022). Online continual learning in image classification: An empirical survey. Neurocompuring, 469:28-51.
McCloskey, M. and Cohen, N. J. (1989), Catastrophic interference in comnectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pages 109-165. Academic Press.
Moens, V. (2018). The hierarchical adaptive forgetting variational filter. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3606-3615. PMLR.
Page, F. S. (1957). On problems in which a change in a parameter occurs at an unknown point. Biometrika, 44(1-2):248-252.
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71.
11

---

Published as a conference paper at ICLR 2024
Patacchiola, M., Tumnrr, J., Crowley, E. J., Boyle, M., and Storkey, A. J. (2020). Bayesian meta- learning for the few-shot setting via deep kernels. In Advances in Neural Information Processing Systems.
Poland, J. and Hutter, M. (2005). Asymptotics of discrete mdl for online prediction. IEEE Transac- tions on Information Theory, 51(11):37003375.
Puskorius, G. and Feldkamp. L. (1994). Neurocontrol of nonlinear dynamical systems with kalman filter trained recurrent networks. IEEE Transactions on Neural Networks, (2):279-2977.
Rasmussen, E. and Williams, C. K. (2006). Gaussian Processes for Machine Learning. MIT Press.
Shalev-Shwartz, S. (2012). Online learning and online convex optimization. Found. Trends Mach. Learn., 4(2):107-194.
Singhal, S. and Wu, L. (1988). Training multilayer perceptrons with the extended kalman algorithm. In Touretzky, D., editor, Advances in Neural Information Processing Systems, volume 1. Morgan- Kaufmann.
Särkkä, S. (2013). Bayesian Filtering and Smoothing. Institute Mathematical Statistics Textbooks. Cambridge University Press.
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. (2016). Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73.
Tsiamis, A. and Pappas, G. (2020). Online learning of the kalman filter with logarithmic regret.
12

---

Published as a conference paper at ICLR 2024
A CONNECTION WITH ONLINE BAYESIAN LINEAR REGRESSION
When the forgetting coefficient is Th. = for all n, the stochasticity in the transitions removed since the parameter transition density becomes point mass, i.e. p( ((w /ww/w,-1111 (0,1)) 6(w,, Wp-1). Then, the
Kalman filter reduces to updating the Gaussian posterior density p(wlyimm) NN An) where
An- Poaln An-1 σ² + 6, A An (yn 04 mn-1), A, = An σ² + 0 An A1
An P8n
m, = mm-1+ +
(11)
with the initial conditions me = 0 and Ao = L.L. We see that these recursions compute exactly an efficient (mm² time the standard Bayesian linear regression posterior given by p(w (yml)).) and where the connection
N ow 2 Σ == 0.3i: An). with A,, (o ( Σニ= O + = with the online updates can be seen by applying the Woodbury matrix identity. Obtaining the online Bayesian linear regression recursion as a special of a Kalman filter is well known in the literature; see
for example Section 3.2 in the book of Särkkä (2013).
B ONLINE UPDATING THE FORGETTING COEFFICIENT Yn FOR REGRESSION
Learning online 7/n for the regression case is simpler than in classification since now the predictive density is tractable Gaussian given by equation 5 Thus, given that we parametrize 7n. e -0.56, with 6n 2 0, the update for ô, is written as
6n - 5n- Pro V., log.N Ngg.Nyn (ym 10 mn on A, On + o²).
(12)
The dependence of the log density on 6, is through the parameters (m AA,)) given by equation while the parameters (m Am-1) of the posterior up to time n 1 are taken as constants.
C CALIBRATION OF CLASS PROBABILITIES AND FINE-NUNING THE REPRESENTATION
Given that in classification the predictive posterior (W,, NWW,B/1ss 31:n-1) 1 is an (possibly crude) approximation to the true posterior, the log predictive probability estimate in (12) can be inaccurate. We improve this estimate by fine-tuning over a calibration parameter that optimized with gradient steps. In the same step we can also fine-tune the neural network parameters 0 that determine the representation vector 6αx,0) Specifically, for the calibration procedure we introduce a parameter a >> 0 that rescales the "fnk
logits inside the softmax function, that the softmax in equation 10 is replaced by ΣR Fmi
and
the final Monte Carlo estimate becomes
S
appn A Fasne (A)
p(y, = 1, vo log
(13)
5
K j==
Then the negative log predictive probability, log = 1111m-1, 0), ; treated as loss that is optimized jointly over (a, 0) with online SGD steps, i.e. as individual data points (x Un) arrive sequentially. This resembles the forward-calibratte described in (Bornschein et al.) 2022b). Algorithm] [anerithm[[summnnn summarizess the whole online learning procedure that includes the Kalman filter recursion, update of coefficient "/n and fine-tuning of the representation parameters and calibration parameter a. An ablation study in Appendix shows that the calibration procedure can significantly improve the log predictive probability estimates.
The fully online process for fine-tuning the deep network parameters 0 in Algorithm 1 can be expen- sive since modern hardware is computationally more effective when forward and backpropagation passes in deep neural nets are applied jointly to minibatches rather to individual data points. In practice, we therefore also consider a faster version of Algorithm [ where we predict ahead a batch of data points by treating them as i.i.d. and then take gradient step over (a, 0). i.c. we use as loss
- Eb log p(yn -l:.n 1: a, f), i = 1,...,b b. This creates two options for the Markov dynamics: (i) either apply the transition "batch-wise where the transition is taken every b data forming the
13

---

Published as a conference paper at ICLR 2024
minibatch (so that time index corresponds to the number of minibatches), or (ii) the minibatch updating only affects the prediction over future data (we predict ahead b points instead of only the next one) and the fine-tuning of (a. 8). while the subsequent Kalman recursion steps and SGD update over "nn are applied online, i.e. by processing the b data in the minibatch one by one. Both schemes can be useful in practice, further discussed in the Appendix. For simplicity, Algorithm [ presents the purely online version, while pscudocode for the above minibatch-baa baseed variants is similar.
D ONLINE LEARNING WITH BAYESIAN FORGETTING
We develop alternative ota-stationary online method based on the concept of Bayesian forgetting
(BF) (Kulhavý and Zarrop, 1993 Honkela and Valpola 2003) Graepel et al.) 2010: Moens] 2018
Kurle et al., (020). We first derive this method and we contrast with our Kalman filter approach described in the main text. To simplify notation we remove the index n when referring to quantifies w, y, A, m. In some part description next we shall use a more general prior of the form Po(w) N(wppan Pw: Dw.) While both BF and Kalman filter recursively estimate some Gaussian posterior distribution p(u) = NN m, A) as the data arrive by incorporating forgetting with a factor [0, Il. they do this differently. Recall that in the Kalman filter method forgetting is applied through convolving with the transition/diffusee dynamics p(w [w) = =Nww 1,w + -1 (1)) (1 >")) so that
p(w 11:m) - f p(w (ww)/w)p(( Vi.m dw NN (ym + (1 M) 2² 22 (1 (1-g))=n)
where in the Kalman filter terminology this is referred the prediction step. In contrast, corresponding step in BF involves mixing the current posterior with the prior using a weighted geometric average and then re-normalizating, as follows
p(w (1.m) )' po(w)'
PBF(W (ymm)
J 1(w"l1l M1nm) (:po 1- N(wlpl
where
A' = (7 =' 7))=))
='=
+1 (1= (7 ()-)))=²)) + (1 - 7) uw)
The remaining steps for the BF method remain the same as in the Kalman filter method. For instance, the posterior update to incorporate the next data point In+1 involves the Bayes' rule p(u]V1m441 1) x p(un++ /w) (ar())/rrlss (w(y where Par(w/y1.n) := Par(w y1.n) while the online update for y is done by taking an SGD step to maximize the corresponding log predictive likelihood computed as in equation TO)
Despite that BF and Kalman filter differ only on how forgetting is incorporated, this still results in rather different properties. Firstly, the Kalman algorithm corresponds to precise probabilistic state space model, where forgetting is achieved through the Markov transition dynamics. In contrast, in BF the forgetting is incorporated heuristically by creating weighted geometric average between the posterior and the prior. Secondly, and more importantly in practice, the Kalman filter method scales as O(m²) per iteration but BF scales cubically as ) To clarify this, firstly observe that an efficient way to update online par is to propagate a forward (including also the step for incorporating the next observation 1n+1) recursion over the natural parameters in quadratic O(m²) time. These natural parameters are the inverse covariance, i.c. 7A- + (1 7)>=> and the inverse covariance times the mean vector, i.c. -A-mmm + (1 - 7)2' Mw. To give the exact form of this recursion for the classification case let us simplify the prior as N( used in the main text and all our experiments. Then the steps are the following:
Mww, Ew) = N N(w [0]]0.0 o I), i.e. consider the one
1. Initialize the natural parameters as A = 1 (m x m matrix) and À = 0 (m x h matrix).
2. Take the weighted geometric average between posterior and prior and re-normalize. This gives new Gaussian with natural parameters
A' = > +
(1
X =A
0
14

---

Published as a conference paper at ICLR 2024
3. Incorporate the next data point obtain posterior Gaussian with natural parameters
A = A' +2 o x= X o² xxx
where recall that y is a 1 one hot vector and ø x 1 vector and for simplicity do not denote the data index.
However, the above efficient updates over the natural parameters cannot prevent cubic cost because we also need to predict the next data output and learn online the forgetting coefficient 7, For that, we need to compute the predictive distribution (as in equation [0 in the main text) which requires the actual (and not the inverse) covariance matrix A- = (A' as well as the mean vector. Based on the above expressions this has (mm)) cost per iteration due to the inverse A' = -A')=
More precisely, we can see that the actual predictive posterior density the logits is factorised Gaussian with mean vector μ = (X²T (A') and isotropic/shared -'1 o. For these last two computations we need to perform an O(m²) Cholesky decomposition of the matrix A'
(7/ + (1 -
variance s² = < (A')-1
The (m)) cost of BF could be too high in some of the experiments where the size m of the feature vector is typically of order of thousands, e.g. m 2048 for the CLOC dataset.
We compare our method with BF on CLOC dataset. In Figure 5 we compare Bayesian Forgetting method against Kalman Filter, where we use fixed pretrained backbone (similar to the main experiment on CLOC), but we see only 1 example at a time. On top of that, we run large scale experiment on CLOC similar the one presented in Figure33room the main paper. In this case, similarly, we see chunk of 128 points at time and we allow for using fixed backbone for learning one. The results are given in Figure 66 In both cases, we see that KF consistently outperforms Bayesian Forgetting whilst being also much more efficient method.
E FURTHER DETAILS ABOUT THE TIME SERIES EXAMPLE
We generated the time series dataset consisted of 3058 observations sequentially by using 8 segments with mean values (1.3, 1.0, 1.3, 0.95, 0.6, 0.25, 0.8, 0.5) and where the change-points between these segments occurred (randomly) at the following 7 time steps {451, 709, 958, 1547, 2147, 2769, 2957). To obtain each observed yn we added Gaussian noise with variance 0.01.
For these time series data applied modification of the Kalman updates that all updates remain the same, except for the update of the parameter m., which now does not shrink to zero and it has the form
m, mm-1, (while before m, mnmmn-1)
This is appropriate in this case because shrinking the predictive posterior mean zero by multiplying it by 7n S. is a very strong prior assumption that Wn, has zero mean which does not hold, since the time series data can have arbitrary values away from zero. A more formal justification of the above is that the Markov dynamics have now the form p( (w)) = (W o I) and p(w, (n/wwnn))
N(w,,((1 - 7n 7n + 7n W1 oi (1 - 7) and each mean parameter fa-1 when we transit from time step 1 to n is found by empirical Bayes that p4- m and where m4-1 is the mean of p_w,,p11111111 11)
The hypermaramcters in the experiment were set to the following values: a = 0.01, 0.05 while do was intitialized to 0.0 (so that 7o = 1.0) and then updated at each step by performing SGD steps with learning rate equal to 1.0.
Figure in the main paper shows the results when 7n is updated online. For comparison, in Figure show the online predictions for the case of having fixed 7n 1 for any n. so that the ability to model non-stationarity is removed. Clearly, when not learning ", the model is not able to adjust to non-stationarity.
15

---

Published as a conference paper at ICLR 2024
Output
500
1000
1500 Time
2000
2500
3000
Figure 4: Online prediction on the artificial time series example by applying the Kalman filter model with fixed
7n
F EXPERIMENTAL DETAILS AND HYPERPARAMETERS SELECTION
F.1 HYPERPARAMETER SELECTION
For hyperparameters we do the grid search over ranges. The hyperparameters for methods are selected by choosing the variant with highest cumulative log probabilities following MDL principle from (Bornschein et al. 2022b). When finetuning backbone together with learning calibration parameter o, considered different relative scaling of the gradients of the backbone compared to a meaning that if we use learning rate 77 for a, the learning rate for backbone is 8 1). In case where we learned For each of the method, we optimized
y, we initialized y at Tini which is also a the hyyer-parameet separately. The eyper-parameters and the considered values are given in the 2
Table 2: Hyperr parametrrsssssesss and considered values.
Hyperparameter
Considered values
Learning rate for backhone and a. 1)
[0.1, 0.01, 0.001, 0.0005, 0.0001]
Learning rate for 7. Y/v
[0.5, 0.1, 0.01, 0.001J
Relative backbone-o gradient scaling,
[0.01, 0.1, 1., 10., 100.0]
Initial Yimit
[0.9, 0.99, 0.999, 1.0]
F.2 FINETUNING PROTOCOL
The algorithm describes Kalman filter online learning where the learning happens for each data point. We adapt this algorithm for chunk-based learning in the following way. When we receive the chunk of data of certain size, we use the available Kalman statistics in order to calculate predictive log-probabilities (see Appendix C) for each data point in the chunk independently. After that, we do the gradient update on the backbone and a, leading us new backbone parameters and new a. After that, we re-compute the features on the same chunk of data and we do Kalman recursion on this chunk going through it sequentially. This latter process only involves updating statistics and % and doesn't 't involve backbone options on how to apply Markov transition. the first option, Always Markov, we apply Markov transition for every data point during Kalman recursion. In the second option, Last Step Markov, we only do Markov transition on the last point in the chunk. The difference basically lies in what we consider a non-stationary data point. In case of Always Markov, it is each data point in the chunk. In case of Last Step Markoy, it is the whole chunk. We found that for different scenarios, different strategies led to different results.
finetuning. When we do this Kalman recursion,
consider two
F.3 ADDITIONAL PER- BENCHMARK PARAMETERS
On top of the lyper-parameten which we selected for each of the baseline separately, we also found that there was group of pypar-parameters which generally provided consistently good results for all the baselines for each benchmark. These parameters are the following:
Transition type: Always Markov Last Step Markov
16

---

Published as a conference paper at ICLR 2024
CLOCI Faed backeene,
CLOC Floed baclbare, learned
Rayesan Nepslie)
2.-
11/6
12l00
17ice
250
750e
30500
12400
Exarpins SeA1
Exargressem
Figure 5: CLOC results in comparing BF with KF methods. In this experiment the chunk size is 1, and we fixed pre-trained backbone for both methods.
From Scratch
From Pretrained
Average Online Accuracy
Average Online Accuracy
0.4
Kalman filter: Fixed backbone
Kalman filter: Finetuned backbone
BF: Finetuned backbone
BF: Fixed backbone
0.2
0.1
2
Data seen
1e6
Data seen
1e6
Figure CLOC results in comparing with KF methods. In this experiment the chunk size is 1, and fixed pre-trained backbone for both methods.
Bias in features: Whether add bias to the features
Features normalization: Whether normalize features vector, i.c., whether divide it by the square root its dimensionality.
Optimization algorithm: plain SGD or Adam with weight decay (A)
These parameters were found empirically and similarly chosen via MDL principle. The table 3 gives the summary.
Table 3: Per benchmark parameters.
Benchmark
Transition type
Use Bias
Use normalization
Optimization alg.
Stationary CIFAR- 100 Non-Stationary CIFAR-100
Last Step Markov
No
Yes
SGD
Always Markov
Yes
Yes
SGD
CLOC with pretrained backbone
Always Markov
No
No
SGD
CLOC from scratch
Last Step Markov
No
No
AdamW (A 10-1)
F.4 CIFAR-100 EXPERIMENTS
In both of CIFAR- 100 experiments, follow protocol described in Ghunaim al. (2023) We use ResNet- 18 as backbone model and SGD as learning algorithm. We split data into chunks of size 10. In some cases, we allow a replay strategy suggested in Ghunaim al. (2023), where keep memory of size 100 containing most recently seen examples. Then, for each learning iteration, we sample chunk of size 10 from memory and append it the right of the current chunk, creating chunk of size 20.
F.5 CLOC EXPERIMENTS
Similar to Ghunaim et al. (2023) and Bornschein et al. (202b)), we considered ResNet-50 as backbone and chunk size equal to 128. Empirically, found that SGD optimization worked much better
17

---

Published as a conference paper at ICLR 2024
From Scratch
From Pretrained
-6-
1
Data seen
1e7
Data seen
1e7
Predictive log
Predictive prob.
Calibrated
Figure 7: CLOC log probabilitites for Kalman filter with finetuned backbone and finetuned delta. We show the data (black dots) and the predicted mean and uncertainty (orange lines) over yo (as data arrive sequentially from left to right and we perform online next time step prediction).
Log probabilities
3.8
-42
-4.4
Predictive log prob.
Predictive log prob. Calibrated
10000 20000 30000 40000 50000
Figure 8: Non-stationary CIFAR-100 log probabilitites for Kalman filter with finetuned backbone and finetuned 5.
pre-trained model whereas Adam with weight decay (A = 10-4) was working much better for learning from scratch. Moreover, we also found that when learning from scratch, it worked much better we used Last Step Markov transition (see Appendix F2 For pre-trained model, it worked
better use Always Markov transition (see Appendix F.2).
For baselines, we considered Online SGD baseline from Bornschein et al. (2022b), with EMA parameter equal to for fair comparison. The case of online SGD with replay corresponds to using 8 replay streams on top of the learning stream which make sure that the data distribution the replay buffer is well behaved (see Bornschein et al. (2022b) for more details).
G THE EFFECT OF CALIBRATION ON ONLINE CLASSIFICATION
In Appendix C we describe procedure to finetune the model backbone well as finetune the parameter a which affects the predictive log probability. This parameter a essentially allows us to calibrate the predictive log probabilities. The effect of this calibration is shown in Figure for for CLOC dataset and in Figure for non-stationary CIFAR-100, In both cases, we use the version of Kalman filter which finetunes the backbone and learns 7. We calibration calibrated log probabilities become much higher. Related to the discussion of prequential MDL (Bornschein et al.) (002b), essentially allows have a model with lower description length. Moreover, since we use predictive log probability as learning signal for backbone, the calibration allows to tune the scaling of this term.
very drastic positive effect of
H CHUNK SIZE ABLATION
In this section we provide the ablation of the impact of the chunk size on the average online accuracy. We provide it for CIFAR- 100 well as for CLOC. We use the Kalman filter variant with learned with either fixed or finetuned backbone. The results are given Figure
18

---

Published as a conference paper at ICLR 2024
Stationary CIFAR-100
Non-stationary CIFAR-100
Average Online Accuracy
00.200
Average Online Accuracy
0.5
0.175
0.4
0.150
0.125
0.3
0.100
0.2
0.075
20
40
80
100
0
20
40
80
100
Chunk size
Chunk size
Fixed Backbone
Finetuned Backbone
Figure 9: Chunk size ablation for CIFAR-100.
19


Reviews:
[{"review_id": "DzwojnaUtK", "reviewer": "Reviewer_BEuY", "summary": "The article presents a method for adapting a neural network model to a possibly non-stationary stream of data. The method consists of applying a Kalman filter to infer the weights of the last layer of the neural network over a data stream defined by an Online Continual Learning task. In addition, the neural network providing the representation and a parameter setting the level of forgetting in the last layer are adapted online, further adapting to a new problem setting. The method was applied to a simple regression task and two large-scale classification problems in continual learning which used the CIFAR-100 and CLOC datasets.", "questions": "How does the training of the representation neural network (backbone finetuning) impact the behaviour of the Kalman filter, since the latent space is changed mid-stream? Can this impact or explain the results in Figure 3, where the Kalman filter method performs better compared to previous work when used on a pre-trained network rather than from scratch?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Impressive experimental results in terms of accuracy metrics. The classification experiments demonstrate that the method can be applied to large-scale high-dimensional tasks and provides clear improvements over previous work.\n- A novel method which clearly differentiates itself from similar recent work through improvements in efficiency and allowing for online adaptation of forgetting parameters\n- The modeling choices were well-motivated, and the discussion in the Related work section on a meaningful setting for continual learning tasks was convincing\n- Section 2.1. was well-written, as it thoroughly explains the details of online learning in the last layer, while also motivating the modeling choices", "weaknesses": "- It was not clear while reading the Related work section how using Kalman Filters in the representation space impacted the results compared to EKF on the full network as in Chang et al. I understood that the method presented here is more efficient, but how is the performance of the model in CL tasks impacted by only considering the last layer in the filter, especially when the representation network is trained from scratch? \n- A further explanation of the backbone finetuning in the main text would be beneficial to understanding the Experiments section, since the impact of finetuning the representation neural network $\\phi$ was often even greater than the impact of tuning the forgetting parameter $\\gamma$, but the former did not appear in the list of contributions and was explained very quickly in the main text.\n- A code implementation was not provided within the supplementary material\n- Claims of efficiency in terms of computation were not supported by empirical results. The comparison to Chang et al. is supported by a theoretical analysis, but a comparison to the baselines Online SGD and ER++ would further support the claims of the method achieving improvements in accuracy metrics without sacrificing computational efficiency, especially when the backbone neural network is finetuned. \n- Some visualizations could be improved for better readability. For instance, Figure 2 has very thin lines in the legend and plot which makes it difficult to see what is happening. \n\nChang, P. G., Durán-Martín, G., Shestopaloff, A. Y., Jones, M., and Murphy, K. (2023). Low-rank extended kalman filtering for online learning of neural networks from streaming data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OliXpEZMUo", "reviewer": "Reviewer_reom", "summary": "This paper proposes an approach to the online learning of regression and classification models using Kalman filtering. The key idea of the paper is to divide the learning into two parts: that of the feature extractor \"backbone\" and the \"last-layer\" representation. The backbone is assumed to be either pretrained and fixed, pretrained and fine-tuned, or trained from scratch using standard optimization approaches. When the backbone is fine-tuned or trained from scratch, the negative log-predictive probability is used as the objective and parameter updates are done using standard SGD steps. The last-layer representation is updated using efficient Kalman filter updates with standard Gaussian assumptions on the emission and transition densities. By allowing the weights on the last-layer representation to evolve as a Gaussian process, with a tuning parameter governing model plasticity (this parameter is shown to be learnable online), the model is able to adapt to \"natural\" (i.e., realistic) distribution shift. It is shown that the proposed hybrid scheme of an SGD-trained feature extractor and Kalman filter updates on the last layer allows the creation of an accurate online classifier in some large sequentially presented data sets.", "questions": "Figure 3 is perhaps the most interesting of the paper, and raises the main question that I have: why is the gap between using a pretrained extractor and Kalman filtering and SGD so much greater than between an extractor trained from scratch with Kalman filtering and SGD? Memory costs aside, why does the Kalman filter add no additional performance gain in this situation? I don't expect it to beat or match the pretrained case, of course, but perhaps be in between the SGD and the pretrained case. What is it about the interaction between learning these two parameter sets that causes this? Have the authors tried experimenting to try and improve the from scratch performance of their proposed method to be even a bit better than online SGD? An understanding of this is basically the missing part of the paper for me.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Online training of neural networks with Kalman filters is not a new idea, but one that has gotten some attention recently with developments aimed at making the procedures more scalable. The present paper makes a good potential addition to this literature by proposing a method to achieve this on practically-sized neural networks for image classification problems. The paper is well-written with comprehensive appendices motivating the various design decisions made. A major strength of the paper are the convincing experimental results and carefully done comparisons. The combination of pre-training (or SGD training) with some plasticity on the parameters on the last-layer representation is, to the best of my knowledge, a novel idea. The use of a Kalman filter on the last layer only allows for efficient updates, as the number of parameters on the last layer is small relative to the rest of the network.", "weaknesses": "The paper is primarily focused on learning in the case of having a pre-trained extractor of a representation. Training a representation extractor can require considerable offline computational expenses. Also, having a fixed representation extractor does not ensure plasticity of the weights of the extractor itself, which may be important in cases where there is sufficient heterogeneity in the data over time that this becomes an issue. It is unclear how much of an issue this can be in practice, and therefore a comment, or some justification, should be made on the implicit assumption that plasticity in the last layer is sufficient. \n\nAs presented, the methodology does not allow for a simple incorporation of additional weight blocks into the Kalman filter update, as this breaks the linear Gaussian assumption that the method relies on. This can perhaps be circumvented with a linearization of the observation model. Have the authors considered extensions of their method in this direction? This would lead to the additional question of what weights to learn with SGD and which weights to learn with the Kalman filter (intuitively, the ones requiring the most plasticity), and how to select them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pq3UnC9rP8", "reviewer": "Reviewer_NkkH", "summary": "The authors present an application of online Kalman filter inference within dealing with concept drift in deep learning. There is no theoretical analysis, but the computational results on variously batched and randomized classification datasets (CIFAR-100) and CLOC are promising.", "questions": "Have you considered extending the work of Kozdoba et al (https://doi.org/10.1609/aaai.v33i01.33014098) showing that Kalman filters can be approximated arbitrarily closely using ARMA models? This could reduce the runtime of O(m^2) to O(d) for recursion depth of d. \n\nHow exactly do you envision to use \"reparametrize the integral to be an expectation under the standard normal and then apply Monte Carlo\"? Plausibly, the scaleability of Monte Carlo could be a constraint? Could you present the statistical performance and runtime of the precomputing and the actual application of the method parametrized by S, as in the O(KS) or O(Km + m^2)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The experiments show a clear improvement over a number of methods that could be seen as the state of the art, incl:\nNo backbone finetuning (Purely linear model)\nBackbone finetuning\nBackbone finetuning with Replay\nOnline SGD\nOnline SGD + Replay\nER++ (Ghunaim et al., 2023).\n\nThe paper is well written, both interms of derivation of the method and explaining the expertiments.", "weaknesses": "The improvement over ER++ is marginal. \n\nThe experiments do not test against the best possible Kalman filter (in terms of the system matrices), or some restarted version of the best possible Kalman filter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CLdHsmXnBz", "reviewer": "Reviewer_VM8e", "summary": "The paper studies the challenging problem of online continual learning and introduces an approach based on the Kalman filter (KF).  This paper models non-stationary as parameter drift based on linear stochastic dynamics and employs KF to estimate the posterior distribution of adaptable model parameters. Additionally, the paper discusses updates to the forgetting coefficient of KF. The experimental results demonstrate the superior performance of the proposed method compared to previous online continual learning methods.", "questions": "1) How does the proposed method compare to related Kalman filter-based techniques for online continual learning and adaptation, such as KF in  [1]?\n\n2) Does the assumption concerning linear stochastic dynamics align with the outcomes presented in Table 1, given that the \"Backbone finetuning\" method outperforms the \"No backbone finetuning (Purely linear model)\"?\n\n3) What are the real-time computational costs associated with each algorithm?\n\n[1] A. Abuduweili, et al, “Robust online model adaptation by extended kalman filter with exponential moving average and dynamic multi-epoch strategy”, L4DC 2020.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper studies a practical and interesting problem: online continual learning, emphasizing non-stationary. It explores the application of  KF in classification problems by updating the forgetting coefficient. \n\nThe proposed method, which utilizes a Kalman filter-based approach for prior and posterior estimation of model parameters, presents a reasonable solution to tackle the (non-stationary)  online learning problem.\n\nOverall, the paper is well-structured and effectively communicates the details of the proposed method.\n\nFurthermore, in the experimental evaluation, the proposed method demonstrates superior performance compared to previous approaches, as reported in the paper.", "weaknesses": "1) This paper could benefit from a comparison with related works that utilize the Kalman filter for online continual learning and adaptation tasks, as seen in [1]. Such a comparison or discussion regarding the difference between the proposed method and [1] would enhance the paper's comprehensiveness.\n\n2) The paper assumes that the ground-truth model follows a linear predictor based on features,  followed by adapting the last layer of the neural network using KF. The results in Table 1 suggest that the \"Backbone finetuning\" method outperforms \"No backbone finetuning (Purely linear model).\" This observation raises questions about the validity of the assumption regarding linear stochastic dynamics.  It is essential to engage in an in-depth discussion regarding the correctness and practicality of these assumptions.\n\n3) In the evaluation of CLOC in section 4.2.2, it's crucial to consider the real-time computational cost for each algorithm, as emphasized in [2]: In the evaluation of the proposed method could be a prudent step.\n\n[1] A. Abuduweili, et al, “Robust online model adaptation by extended kalman filter with exponential moving average and dynamic multi-epoch strategy”, L4DC 2020.   \n[2] Bornschein, et al. “Sequential learning of neural networks for prequential mdl.” ICLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)