Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams

1 INTRODUCTION . The prevalence of data streams in contemporary applications urges systems to learn in a continual fashion . Autonomous vehicles , sensory robot data , and video streaming yield never-ending streams of data , with abrupt changes in the observed environment behind every vehicle turn , robot entering a new room , or camera cut to a subsequent scene . Alas , learning from streaming data is far from trivial due to these changes , as neural networks tend to forget the knowledge they previously acquired . The data stream presented to the network is not identically and independently distributed ( iid ) , emanating a trade-off between neural stability to retain the current state of knowledge and neural plasticity to swiftly adopt the new knowledge ( Grossberg , 1982 ) . Finding the balance in this stability-plasticity dilemma addresses the catastrophic forgetting ( French , 1999 ) induced by the non-iid intrinsics of the data stream , and is considered the main hurdle for continually learning systems . Although a lot of progress has been established in the literature , often strong assumptions apply , impeding applicability for real-world systems . The static training and testing paradigms prevail , whereas a true continual learner should enable both simultaneously and independently . Therefore , we propose the two-agent learner-evaluator framework to redefine perspective on existing paradigms in the field . Within this framework , we introduce data incremental learning , enabling completely task-free learning and evaluation . Furthermore , we introduce Continual Prototype Evolution ( CoPE ) , a new online data incremental learner wherein prototypes perpetually represent the most salient features of the class population , shifting the catastrophic forgetting problem from the full network parameter space to the lowerdimensional latent space . As a first , our prototypes evolve continually with the data stream , enabling learning and evaluation at any point in time . Similar to representativeness heuristics in human cognition ( Kahneman & Tversky , 1972 ) , the class prototypes are the cornerstone for nearest neighbor classification . Additionally , the system is robust to highly imbalanced data streams by the combination of replay with a balancing memory population scheme . We find batch information in the latent space to have a significant advantage in the challenging non-stationary and online processing regime , which we incorporate in the novel pseudo-prototypical proxy loss . 2 THE LEARNER-EVALUATOR FRAMEWORK . To date , the paradigms of task , class , and domain incremental learning ( van de Ven & Tolias , 2018 ) dominate the continual learning literature . However , strong and differing assumptions often lead to confusion and overlap between implementations of these definitions . Furthermore , the concept of a static training and testing phase is still ubiquitous , whereas continual learning systems should enable both phases continually and independently . Therefore , we propose a generalizing framework which disentangles the continually learning system into two agents : the learner and the evaluator . Figure 1 presents an overview of the framework . The learning agent learns predicting function fθ : X → Y parameterized by θ , mapping the input space X to the target output space Y . The learner receives data samples ( xi , yi ) from stream S and has simultaneous access to the horizon D , i.e . the observable subset of stream S which can be processed for multiple iterations . Data sample i is constituted by input feature xi ∈ X and corresponding ( self- ) supervision signal yi for which the output space for classification is defined as a discrete set of observed classes Yi ← Yi−1 ∪ { yi } . To manage memory usage and to enable multiple updates and stochasticity in the optimization process , updates for θ are typically performed based on a small-scale processing batch B ⊆ D. The data and size of the horizon D are determined by the specific setup or application , ranging from standard offline learning with D = S to online continual learning with D = B . Furthermore , the learner might need additional resources after observing data from B ⊆ D , such as stored samples or model copies , confined by the operational memoryM . The evaluating agent acts independently from the learner by evaluating fθ with horizon Deval from the evaluation stream Seval , with small-scale processing batches Beval ⊆ Deval . This stream can contain yet unobserved concepts by the learner in S to measure zero-shot performance . The framework provides leeway for the concept distributions in Seval being either static or dynamically evolving , determining how performance of the learner is measured . On the one hand , static concept distributions can measure the degree to which the knowledge of learned concepts is preserved , as commonly used in continual learning . On the other hand , evolving concept distributions measure performance for the current distribution in horizon Deval only , where concepts might drift from their original representation , also known as concept drift ( Schlimmer & Granger , 1986 ) . Evaluation can occur asynchronously on-demand or periodically with periodicity ρ determining the resolution of the evaluation samples . Task , class , and domain incremental learning are based on the composition in the learner for the observable stream subset in horizon Dt , which is incrementally replaced by a new subset of data for the new task , set of classes , or domain , with t the identifier of the present data subset . Task incremental learning assumes both learner and evaluator to get data ( xi , yi , ti ) with ti+1 ≥ ti and the horizon spanning all data of a given task with Dt = { ( xi , yi , ti ) ∈ S | ti = t } ( De Lange et al. , 2019 ; van de Ven & Tolias , 2019 ) . Having explicit access to ti confines prediction to an isolated output space . Similarly , in class incremental learning the learner implicitly requires ti to identify the transitions of D , when observing new batches of classes ( Rebuffi et al. , 2017 ; Castro et al. , 2018 ; Shmelkov et al. , 2017 ; Wu et al. , 2018 ) . However , the evaluator considers the entire output space without the need for identifier t. Domain incremental learning holds the same assumptions as class incremental learning , with concepts drifting from one domain to the other for a typically fixed output space , exemplified by the widely used permuted-MNIST setup ( Goodfellow et al. , 2013 ) . Data incremental learning is a more general paradigm we introduce to facilitate learning from any data stream , with no assumption but to observe data incrementally . In contrast to existing paradigms , when the learner observes horizon D of data stream S , data incremental learning does not disclose an identifier t. Consequently , there is no explicit indication to which subset of the stream is being observed in the horizon D. Therefore , the learner either processes observed data directly in an online fashion with processing batch B = D , or infers an implicit identifier t from statistics in stream S. Similar to class and domain incremental learning , the evaluator operates without t on the full output space . This paradigm endows continually learning systems with increased practical use , as real-world streaming applications often lack supervision signal t. Moreover , even if t is provided , this would introduce a bias in the fixed choice of the supervisor , rather than dynamically determined based on the needs of the system . 3 PRIOR WORK . Continually learning systems are able to learn with limited resources from data streams prone to severe distribution shifts . The main body of works presumes the presence of tasks , which divide the data streams into large discrete batches , and are indicated to the learner with a task identifier ( Kirkpatrick et al. , 2017 ; Li & Hoiem , 2017 ; Zenke et al. , 2017 ; Aljundi et al. , 2018 ; De Lange et al. , 2020 ) . Replay methods retain representative data for observed data distributions , currently unavailable in the learner ’ s horizon D. The replay data is either obtained directly from operational memoryM with stored samples ( Rebuffi et al. , 2017 ; Lopez-Paz & Ranzato , 2017 ) or generated using generative models ( Shin et al. , 2017 ; Kamra et al. , 2017 ; Seff et al. , 2017 ; Wu et al. , 2018 ) . GEM ( Lopez-Paz & Ranzato , 2017 ) uses replay in a constraint optimization perspective to project gradients towards a local joint task optimum . iCaRL ( Rebuffi et al. , 2017 ) employs exemplars to distill knowledge ( Hinton et al. , 2015 ) to the learner from a previous model version , with new class exemplars stored in a queue to optimally represent the class mean in feature space . The prototypes are then used for nearest neighbor prediction by the evaluator , in the same vein as concurrent work to ours ( Han et al. , 2020 ) . Nonetheless , all three works strongly rely on task identifier t for the learner , mostly unavailable for real-world data streams . Moreover , in both prototypical approaches the prototypes remain static between the given task transitions and become outdated . Consequently , before using the evaluator they have to exhaustively recalculate the prototypes based on all exemplars in memory . In contrast , our prototypes evolve in an online fashion with the data stream and remain representative for the continual learner and evaluator at all times . Recent works focus on online data incremental learning ( Section 2 ) in which the learner operates completely task-free . Reservoir ( Vitter , 1985 ) is a replay baseline with strong potential to outperform continual learning methods ( Chaudhry et al. , 2019 ) . Samples are stored in memoryMwith probability M/n , with n the number of observed samples and buffer sizeM . MIR ( Aljundi et al. , 2019a ) extends Reservoir sampling with a loss-based retrieval strategy , with the cost of additional forward passes and a model copy to attain the losses for a subset of samples . The Reservoir buffer population approximately follows the data stream distribution , severely deteriorating the performance of underrepresented tasks in imbalanced data streams , as shown in Section 6.2 . An alternative memory population scheme is used in GSS ( Aljundi et al. , 2019b ) by extending the GEM constraint optimization perspective to an instance-based level . GSS adds samples to the buffer based on their gradients , whereas GEM requires the number of tasks and the task transitions to divide memory equally over all tasks a priori . In contrast , iCaRL ’ s memory population is incrementally subdivided over all classes after learning a task , by iteratively adding observed samples from D to optimally approximate the class mean in feature space . As this is computationally expensive , concurrent works to ours explore other balancing schemes ( Kim et al. , 2020 ; Chrysakis & Moens , 2020 ) , where we propose a simple but effective class-based Reservoir scheme with uniform retrieval . Another branch of parameter isolation methods ( De Lange et al. , 2019 ) allocates parameters to subsets of the data . Several task incremental works assign parameters based on the task identifier ( Mallya & Lazebnik , 2018 ; Serra et al. , 2018 ) . A new line of work instead focuses on task-free model expansion . CURL ( Rao et al. , 2019 ) enables task-free and unsupervised adaptation using a multi-component variational auto-encoder , with generative replay from a model copy avoiding forgetting in the current model . CN-DPM ( Lee et al. , 2020 ) allocates data subsets to expert networks following a Dirichlet process mixture . In contrast to these capacity expansion based methods , CoPE evades unbound allocation of resources , as the memory and network capacity are fixed with the replay memory dynamically subdivided over categories occurring in the data stream . Note that new categories require an additional prototype , but these are only d-dimensional and therefore insignificant in size , and the set of categories is typically limited as well . Besides the focus on continual learning in this work , our learner-evaluator framework generalizes to concept drift as well ( Schlimmer & Granger , 1986 ) , for which we refer to an overview in ( Tsymbal , 2004 ; Gama et al. , 2014 ) . Further , in deep embedding learning most commonly pairs ( Hadsell et al. , 2006 ) and triplets ( Harwood et al. , 2017 ) of samples are considered in contrastive losses , whereas other works use batch information in lifted structure embeddings ( Oh Song et al. , 2016 ) , or instancewise softmax embeddings ( Ye et al. , 2019 ) . These approaches fully depend on the batch size , whereas our pseudo-prototypical proxy loss aggregates both decoupled prototypes and the additional batch pseudo-prototypes to defy class interference in the latent space . Learning prototypical representations also shows promising results in few-shot learning ( Snell et al. , 2017 ) .

Summary: This article introduces a learner-evaluator framework that incorporates the different variations of problems related to incremental learning. It also proposes a method called Continual Prototype Evolution for dealing with the most general version of the problem, incremental learning on data streams, in which the learning task is not specified. The paper presents an extensive amount of experiments indicating that in this scenario, the proposed method improves significantly on existing approaches in terms of accuracy and memory efficiency. The article is well organized, easy to read and understand.