Information-theoretic Online Memory Selection for Continual Learning

1 INTRODUCTION . Continual learning ( Robins , 1995 ; Goodfellow et al. , 2013 ; Kirkpatrick et al. , 2017 ) aims at training models through a non-stationary data stream without catastrophic forgetting of past experiences . Specifically , replay-based methods ( Lopez-Paz & Ranzato , 2017 ; Rebuffi et al. , 2017 ; Rolnick et al. , 2019 ) tackle the continual learning problem by keeping a replay memory for rehearsals over the past data . Given the limited memory budget , selecting a representative memory becomes critical . The majority of existing approaches focus on task-based continual learning and update the memory based on the given task boundaries . Since the requirement for task boundaries is usually not realistic , general continual learning ( GCL ) ( Aljundi et al. , 2019a ; Delange et al. , 2021 ; Buzzega et al. , 2020 ) has received increasing attention , which assumes that the agent observes the streaming data in an online fashion without knowing task boundaries . GCL makes the online memory selection more challenging since one needs to update the memory in each iteration based only on instant observations . So , successful memory management for GCL needs to be both efficient and effective . Due to the online constraint , only a few memory selection methods are applicable to GCL . A selection method compatible with GCL is reservoir sampling ( RS ) ( Vitter , 1985 ) . In particular , RS can be used to randomly extract a uniform subset from a data stream in a single pass , and it showed competitive performances for GCL for balanced streams ( Chaudhry et al. , 2019b ; Buzzega et al. , 2020 ) . However , when the data stream is imbalanced , RS tends to miss underrepresented ( but essential ) modes of the data distribution . To improve the robustness against data imbalance , gradient-based sample selection ( GSS ) ( Aljundi et al. , 2019b ) proposes to minimize the gradient similarities in the memory , but evaluating many memory gradients in each iteration incurs high computational costs . We investigate online memory selection from an information-theoretic perspective , where the memory attempts to maintain the most information for the underlying problem . Precisely , we evaluate each data point based on the proposed surprise and learnability criteria . Surprise captures how unexpected a new point is given the memory , and allows us to include new information in the memory . In contrast , learnability captures how much of this new information can be absorbed without interference , allowing us to avoid outliers . Then we present a scalable Bayesian model that can compute surprise and learnability with a small computational footprint by exploiting rank-one matrix structures . Finally , we demonstrate the effectiveness of the proposed criteria using a greedy algorithm . While keeping a representative memory is essential , we show that the timing of the memory updates can also be crucial for continual learning performance . Concretely , we highlight that the agent should not update the memory as soon as it sees new data , otherwise it might prematurely remove historical data from the memory and weaken the replay regularization in the GCL process . This phenomenon affects the greedy algorithm much more than RS , since the memory updates in RS appear randomly over the whole data stream . To combine the merits of information-theoretic criteria and RS , we modify reservoir sampling to select informative points only . This filters out uninformative points , and thus encourages a diverse memory and improves the robustness against data imbalance . Empirically , we demonstrate that the proposed information-theoretic criteria encourage to select representative memories for learning the underlying function . We also conduct standard continual learning benchmarks and demonstrate the advantage of our proposed reservoir sampler over strong GCL baselines at various levels of data imbalance . Finally , we illustrate the efficiency of the proposed algorithms by showing their small computational overheads over the standard RS . 2 MEMORABLE INFORMATION CRITERION FOR ONLINE MEMORY SELECTION . Firstly we formalize the online memory selection problem and propose our information-theoretic criteria to detect informative points . Then we introduce a Bayesian model to compute the proposed criteria efficiently . Finally , we demonstrate the usefulness of the criteria in a greedy algorithm . 2.1 ONLINE MEMORY SELECTION . We consider a supervised learning problem , where the agent evolves a predictor f to learn the underlying mapping from inputs to targets . We assume that the predictor is in the following form , fθ ( · ) = gθg ( hθh ( · ) ) , ( 1 ) where θ : = ( θg , θh ) . The base network hθh ( x ) extracts the feature representation of the input x , and gθg maps the representation to predictions . The function gθg is usually a simple module , such as a linear layer ( Krizhevsky et al. , 2012 ) or mean-of-neighbours ( Snell et al. , 2017 ) mapping to the softmax logits . We assume the memory buffer contains the inputs , features , predictions , and targets , M = { ( xm , hm , gm , ym ) } Mm=1 . ( 2 ) In the online memory selection problem , the agent iteratively receives observations from an online stream , and updates the predictor and the memory buffer . Let B = { ( xb , yb ) } Bb=1 be the observed batch in one iteration , a general update formula can be expressed as , ( θ , M ) ←− ( θ , M , B ) . ( 3 ) We emphasize that updating the predictor parameters θ and updating the memoryM are essential but orthogonal components . For example , the agent can update the memory using RS ( Buzzega et al. , 2020 ) or GSS ( Aljundi et al. , 2019b ) , and update θ using GEM ( Lopez-Paz & Ranzato , 2017 ) or experience replay ( Chaudhry et al. , 2019b ) . This section focuses on how to maintain a representative memory M online , while how to update the predictor fθ is not our focus . Furthermore , unlike task-based continual learning ( Kirkpatrick et al. , 2017 ) , we assume the agent receives only the data stream without observing task boundaries . This scenario is more difficult for memory selection . For instance , it must deal with data imbalances , e.g. , if discrete tasks of varying lengths exist , we may want the selection to identify a more balanced memory and avoid catastrophic forgetting . 2.2 MEMORABLE INFORMATION CRITERION . Intuitively , the online memory should maintain points that carry useful information . We consider a Bayesian model for this problem since the Bayesian framework provides a principled way to quantify the informativeness of data points . Specifically , we let p ( y|w ; x ) be the likelihood of the observation y given the deterministic input x dependent on the random model parameter w,1 and we denote by p ( w ) the prior distribution . Then the posterior of w and the predictive distribution of new 1The density p and the parameter w here are left completely generic . data points can be inferred according to Bayes ’ s rule . Also worth mentioning , the Bayesian model is introduced merely for memory selection , which is different from the predictor fθ . In each iteration of online memory selection , the agent decides whether to include the new data point ( x ? , y ? ) into the memoryM . To this end , we present information-theoretic criteria based on the Bayesian model to evaluate the usefulness of the new data point . For notational simplicity , we denote by XM and yM the inputs and the targets in the memory , respectively . Surprise . As Claude Shannon says : “ Information is the resolution of uncertainty ” . If the memory M is certain of the target y ? of x ? beforehand , incorporating the data point ( x ? , y ? ) brings little information intoM . Thus we propose to measure the “ surprise ” of the new point given the memory . We define the surprise criterion of ( x ? , y ? ) givenM as the negative log conditional probability , ssurp ( ( x ? , y ? ) ; M ) = − log p ( y ? |yM ; XM , x ? ) , ( 4 ) where p ( y ? |yM ; XM , x ? ) = ∫ p ( y ? |w ; x ? ) p ( w|yM ; XM ) dw . Intuitively , if ( x ? , y ? ) comes from a new task , it will likely be less predictable according to the memory and the surprise will be large . In this way , the surprise criterion detects informative points and ensures diversity within the buffer . Learnability . The surprise detects unpredictable points in the data stream , but it is not capable of distinguishing “ helpful ” unfamiliar points and “ harmful ” outliers . To avoid the outliers , we propose the learnability criterion , which measures how much our Bayesian model can explain the new point once it has absorbed its information in the memory . Formally , we define the learnability as follows , slearn ( ( x ? , y ? ) ; M ) = log p ( y′ ? |y ? , yM ; XM , x ? ) |y′ ? =y ? , ( 5 ) where we use y′ ? and y ? to represent two realizations of the same random variable . Intuitively , the learnability measures the consensus between the new point and the memory . We illustrate the utility of learnability through a toy example in Figure 1 . We use a Gaussian process with the RBF kernel to fit memory points ( shown as black dots • ) . We visualize its pre- dictive mean and 95 % confidence interval as the green line and the shadow region . For two new points ( shown as red stars ? ) separately in the left and right figure , we also visualize the predictive distribution given the memory and the new point , shown as the blue line and the shadow region . We observe that both points are surprising to the memory since they lie outside ( or just outside ) of the green confidence regions . However , after conditioning on the new point itself , the left point still lies outside of the confidence region while the prediction on the right adjusts to include the new point . Thus the left point has low learnability , and the right point has high learnability , though both have high surprises . Thus the learnability can be used to detect outliers when combined with the surprise . Memorable Information Criterion ( MIC ) . Given η ≥ 0 , we propose the Memorable Information Criterion ( MIC ) as a combination of the surprise and learnability of the new point , MICη ( ( x ? , y ? ) ; M ) = ηslearn ( ( x ? , y ? ) ; M ) + ssurp ( ( x ? , y ? ) ; M ) . ( 6 ) Notably , the MIC is related to the information gain ( IG ) ( Cover , 1999 ) : IG ( ( x ? , y ? ) ; M ) = KL ( p ( w|y ? , yM ; XM , x ? ) ||p ( w|yM ; XM ) ) . By Appendix E , IG can be rewritten as , IG ( ( x ? , y ? ) ; M ) = Ep ( w|y ? , yM ; XM , x ? ) [ log p ( y ? |w ; x ? ) ] − log p ( y ? |yM ; XM ) , ( 7 ) where the second term is the surprise , and the first term is upper bounded by the learnability based on Jensen ’ s inequality . Thus we have that MIC1 ( ( x ? , y ? ) ; M ) ≥ IG ( ( x ? , y ? ) ; M ) ≥ 0 . Given the connection , we generalize IG to be a weighted combination of learnability and surprise as well in the Appendix . We also compare with an entropy reduction criterion . More discussions are in Sec C .

Summary: This work targets the problem of Continual Learning (CL) and In particular, the problem of populating the buffer memory in the Experience Replay (ER)-based paradigm. In ER, we are allowed to store a small subset of the incoming data stream, therefore deciding on a good policy to populate the memory buffer is important. The paper proposes MEMORABLE INFORMATION CRITERION (MIC) which is a combination of the surprisal and learnability of an incoming example and further motivates this with a  bayesian analysis. Using MIC, it introduces two algorithms, namely, InfoRS and infoGS for buffer update. The paper includes experiments on standard CL datasets with a focus on General Continual Learning (GCL) guideline, which focuses on not having dependence on task boundaries. Finally, it reports the proposed method's performance on imbalanced data streams and shows that infoRS outperformed reservoir sampling, GSS, and some other baselines.