Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams
Authors: Matthias De Lange
Tinne Tuytelaars
Venue:  (2021)
Keywords: continual learning
prototypical learning
online learning
incremental learning
deep learning
representation learning
catastrophic forgetting
concept drift

Abstract: Attaining prototypical features to represent class distributions is well established in representation learning. However, learning prototypes online from streams of data proves a challenging endeavor as they rapidly become outdated, caused by an ever-changing parameter space in the learning process. Additionally, continual learning assumes a non-stationary nature of the data stream, typically resulting in catastrophic forgetting of previous knowledge. As a first, we introduce a system addressing both problems, where prototypes evolve continually in a shared latent space, enabling learning and prediction at any point in time. In contrast to the major body of work in continual learning, data streams are processed in an online fashion, without additional task-information, and an efficient memory scheme provides robustness to imbalanced data streams. Besides nearest neighbor based prediction, learning is facilitated by a novel objective function, encouraging cluster density about the class prototype and increased inter-class variance. Furthermore, the latent space quality is elevated by pseudo-prototypes in each batch, constituted by replay of exemplars from memory. We generalize the existing paradigms in continual learning to incorporate data incremental learning from data streams by formalizing a two-agent learner-evaluator framework, and obtain state-of-the-art performance by a significant margin on eight benchmarks, including three highly imbalanced data streams.

Under review as a conference paper at ICLR 2021
CONTINUAL PROTOTYPE EVOLUTION: LEARNING ONLINE FROM NON-STATIONAL DATA STREAMS
Anonymous authors Paper under double-blind review
ABSTRACT
Attaining prototypical features to represent class distributions is well established in representation learning. However, learning prototypes online from streams of data proves a challenging endeavor as they rapidly become outdated, caused by an ever-changing parameter space in the learning process. Additionally, continual learning does not assume the data stream to be stationary, typically resulting in catastrophic forgetting of previous knowledge. As a first, we introduce a system addressing both problems, where prototypes evolve continually in shared latent space, enabling learning and prediction at any point in time. In contrast to the major body of work in continual learning, data streams are processed in online fashion, without additional aas-information, and an efficient memory scheme provides robustness imbalanced data streams. Besides nearest neighbor based prediction, learning is facilitated by a novel objective function, encouraging cluster density about the class prototype and increased inter-class variance. Furthermore, the latent space quality is elevated by socudo-prototype in each batch, constituted by replay of exemplars from memory. We generalize the existing paradigms in continual learning to incorporate data incremental learning from data streams by formalizing a two-agent learner-evaluator framework, and obtain state-of-the-art performance by significant margin eight benchmarks, including three highly imbalanced data streams.
INTRODUCTION
The prevalence of data streams in contemporary applications urges systems to learn in a continual fashion. Autonomous vehicles, sensory robot data, and video streaming yield never-ending streams of data, with abrupt changes the observed environment behind every vehicle turn, robot entering a new room, or camera cut to subsequent scene. Alas, learning from streaming data is far from trivial due to these changes, as neural networks tend to forget the knowledge they previously acquired. The data stream presented to the network is not identically and independently distributed (tid), emanating a trade-off between neural stability to retain the current state of knowledge and neural plasticity to swiftly adopt the new knowledge (Grossberg) 1982). Finding the balance this stability- dilemma addresses the catastrophic forgetting (French 1999) induced by the non-iid intrinsics of the data stream, and is considered the main hurdle for continually learning systems.
Although lot of progress has been established in the literature, often strong assumptions apply, impeding applicability for real-world systems. The static training and testing paradigms prevail, whereas a true continual learner should enable both simultaneously and independently. Therefore, we propose the two-agent ceamutr-vvllutor framework redefine perspective on existing paradigms in the field. Within this framework, we introduce data incremental learning, enabling completely task-free learning and evaluation.
Furthermore, we introduce Continual Prototype Evolution (CoPE), new online data incremental learner wherein prototypes perpetually represent the most salient features the class population, shifting the catastrophic forgetting problem from the full network parameter space to the lower- dimensional latent space. As a first, our prototypes evolve continually with the data stream, enabling learning and evaluation at any point in time. Similar representativeness heuristics in human cognition (Kahneman & Tversky, 1972), the class prototypes are the cornerstone for nearest neighbor classification. Additionally, the system robust to highly imbalanced data streams by the combination

---

Under review as a conference paper at ICLR 2021
D
Devalty
Evaluator
Beval
Seval
lit1
evaluate
fo
D
M
4
optimize
S
f++
B
Learner
Figure 1: Overview of the learner-evaluaton framework, overcoming the static training and testing paradigms by explicitly modelling continual optimization and evaluation from data streams the learner and evaluator agents. The framework generalizes to both continual learning and concept drift with resources transparently defined as the horizon DD and operational memory M.
of replay with a balancing memory population scheme. We find batch information in the latent space have a significant advantage in the challenging non-stationary and online processing regime, which we incorporate in the novel pseudo-preee poauddo-roootypccll proxy loss.
2 THE LEARNER-EVALUANE FRAMEWORK
To date, the paradigms of task, class, and domain incremental learning (van de Ven & Tolias 2018) dominate the continual learning literature. However, strong and differing assumptions often lead to confusion and overlap between implementations of these definitions. Furthermore, the concept of static training and testing phase is still ubiquitous, whereas continual learning systems should enable both phases continually and independently. Therefore, we propose generalizing framework which disentangles the continually learning system into two agents: the learner and the evaluator. Figure presents an overview of the framework.
The learning agent learns predicting function fo : X - y parameterized by 0, mapping the input space X' to the target output space y. The learner receives data samples yz) from stream and has simultaneous access to the horizon 'D i.e. the observable subset of stream S which can be processed for multiple iterations. Data sample corresponding (spperisupeiiisiion signal y, for which the output space for classification is defined as discrete set of observed classes D; + Vi-1 U {y:}. To manage memory usage and to enable multiple updates and stochasticity in the optimization process, updates for 0 are typically performed based on a small-scale processing batch B C D. The data and size of the horizon D are determined by the specific setup or application, ranging from standard offline learning with D S online continual learning with D = B. Furthermore, the learner might need additional resources after observing data from B D. such stored samples or model copies, confined by the operational memory M.
constituted by input feature x; E X and
The evaluating agent acts independently from the learner by evaluating fe with horizon Devat from the evaluation stream Seval, with small-scale processing batches Beval C. Devai- This stream can contain yet unobserved concepts by the learner in S to measure zero-shot performance. The framework provides leeway for the concept distributions in Seval being either static or dynamically evolving, determining how performance of the learner is measured. On the one hand, static concept distributions can measure the degree to which the knowledge of learned concepts preserved, as commonly used in continual learning. On the other hand, evolving concept distributions measure performance for the current distribution in horizon Deval only, where concepts might drift from their original representation, also known concept drift (Schlimmer & Granger 1986). Evaluation occur asynchronously on-demand or periodically with periodicity p determining the resolution the evaluation samples.
Task, class, and domain incremental learning are based on the composition in the learner for the observable stream subset in horizon D,, which is incrementally replaced by subset of data
2

---

Under review as a conference paper at ICLR 2021
for the new task, set of classes, or domain, with the identifier of the present data subset. Task incremental learning assumes both learner and evaluator to get data (x,, Vt, 14) with t,41 2 f, and the horizon spanning all data of a given task with D, = {(x, Y2: f2) S t2 t} (De Lange et al.
2019; van de Ven & Tolias 2019). Having explicit access to 4 confines prediction to an isolated output space. Similarly, in class incremental learning the learner implicitly requires to identify the transitions of D, when observing new batches of classes (Rebuffi et al. 2017] Castro et al., 2018) Shmelkov et al. 2017 Wu al. 2018). However, the evaluator considers the entire output space without the need for identifier t. Domain incremental learning holds the same assumptions as class incremental learning, with concepts drifting from one domain to the other for a typically fixed output
space, exemplified by the widely used permuted-MNIST setup (Goodfellow et al. 2013).
Data incremental learning is a more general paradigm we introduce to facilitate learning from any data stream, with no assumption but to observe data incrementally. In contrast to existing paradigms, when the learner observes horizon DD of data stream S, data incremental learning does not disclose an identifier t. Consequently, there is no explicit indication to which subset of the stream is being observed in the horizon D. Therefore, the learner either processes observed data directly in an online fashion with processing batch B D, or infers an implicit identifier t from statistics in stream S. Similar to class and domain incremental learning, the evaluator operates without t on the full output space. This paradigm endows continually learning systems with increased practical use, as streaming applications often lack supervision signal I. Moreover, even if / is provided, this would introduce bias in the fixed choice of the supervisor, rather than dynamically determined based on the needs of the system.
3 PRIOR WORK
Continually learning systems are able to learn with limited resources from data streams prone to severe distribution shifts. The main body of works presumes the presence of tasks, which divide the data streams into large discrete batches, and are indicated to the learner with a task identifier (Kirkpatrick)
ctt al. 2017 Li & Hoiem 2017 Zenke et al., 2017 Aljundi et al.] 2018 De Lange et al. 2020). Replay methods retain representative data for observed data distributions, currently unavailable in the learner's 's horizon D. The replay data is either obtained directly from operational memory M with stored samples (Rebuffi al. 2017: Lopez-Paz & Ranzato 2017) generated using generative models (Shin et al.] Kamra et al. 2017: Seff et al. 2017] Wu et al. 2018). GEM (Lopez-Paz & [anzato|20177 uses replay in a constraint optimization perspective to project gradients towards a local joint task optimum. iCaRL (Rebulfi et al. (017) employs exemplars to distill knowledge (Hinton ct al. 2015) to the learner from a previous model version, with new class exemplars stored in queue to optimally represent the class mean in feature space. The prototypes are then used for nearest neighbor prediction by the evaluator, in the same vein as concurrent work to ours (Han et al.) 2020). Nonetheless, all three works strongly rely on task identifier ł for the learner, mostly unavailable for real-world data streams. Moreover, in both prototypical approaches the prototypes remain static between the given task transitions and become outdated. Consequently, before using the evaluator they have to exhaustively recalculate the prototypes based on all exemplars in memory. In contrast, our prototypes evolve in an online fashion with the data stream and remain representative for the continual learner and evaluator at all times.
Recent works focus on online data incremental learning (Section 2) in which the learner operates completely task-free. Reservoir [Vitter 1985) is replay baseline with strong potential to outperform continual learning methods (Chaudhry al.] 2019). Samples are stored in memory M with probability M/n, with n the number of observed samples and buffer size M. MIR (Aljundi et al. 2019a) extends Reservoir sampling with loss-based retrieval strategy, with the cost of additional forward passes and a model copy to attain the losses for subset of samples. The Reservoir buffer population approximately follows the data stream distribution, severely deteriorating the performance of underrepresented tasks in imbalanced data streams, as shown in Section 6.2 An alternative memory population scheme is used in GSS (Aljundi et al. 2019b) by extending the GEM constraint optimization perspective to an instance-basee level. GSS adds samples to the buffer based on their gradients, whereas GEM requires the number of tasks and the task transitions to divide memory equally over all tasks priori. In contrast, iCaRL's memory population is incrementally subdivided over all classes after learning a task, by iteratively adding observed samples from D to optimally approximate the class mean in feature space. As this is computationally expensive, concurrent works ours explore other balancing
3

---

Under review as a conference paper at ICLR 2021
Rd
S
B
fo
Mp
M2
f2
Figure 2: Main setup. The learner updates network So and prototypes p", Vy E y continually. The PPP-loss encourages inter-class variance (red arrows) and reduces intra-class variance (green arrows).
schemes (Kim al. 2020, Chrysakis Moens, 2020), where we propose a simple but effective class-based Reservoir scheme with uniform retrieval.
Another branch of parameter isolation methods (De Lange et al. 2019) allocates parameters to subsets of the data. Several task incremental works assign parameters based on the task identifier (Mallya & Lazebnik 2018; Serra et al. 2018). A new line of work instead focuses on task-free model expansion. CURL (Rao et al. 2019) enables task-free and unsupervised adaptation using multi-component variational auto-encoder, with generative replay from a model copy avoiding forgetting in the current model. CN-DPM (Lee et al. 2020) allocates data subsets expert networks following Dirichlet process mixture. In contrast to these capacity expansion based methods, CoPE evades unbound allocation of resources, as the memory and network capacity fixed with the replay memory dynamically subdivided over categories occurring in the data stream. Note that new categories require an additional prototype. but these are only d-dimensional and therefore insignificant in size, and the set of categories is typically limited as well.
Besides the focus on continual learning in this work, our learner-evaluator framework generalizes to concept drift as well (Schlimmer & Granger 1986), for which we refer to an overview in (Tsymbll 2004 Gama et al.) 2014). Further, in deep embedding learning most commonly pairs (Hadsell et al., 2006) and triplets (Harwood et al. (017)) of samples are considered in contrastive losses, whereas other works use batch information in lifted structure embeddings (Oh Song et al.) 2016), or instance- wise softmax embeddings (Ye et al. 2019). These approaches fully depend on the batch size, whereas our pseudo- eseudo-prototyy to defy class interference in the latent space. Learning prototypical representations also shows promising results in few-shot learning (Snell et al. 2017).
pseudo-protooypice. prototypical proxy loss aggregates both decoupled prototypes and the additional batch
4 CONTINUAL PROTOTYPE EVOLUTION
The online data incremental learning setup of the learner is described in Figure Embedding network fo maps processing batch B, composed of samples in horizon D from the non-iid data stream S and operational memory M, to low-dimensional I2d latent space, followed by a nearest neighbor classifier. We enforce [)fa((;)) = 1 with an L2 normalization layer. M is subdivided in replay memory M, and prototypical memory Mp. CoPE comprises three main components: continually evolving representations, balanced replay and the gseudo-prototypen
proxy (PPP) loss. In the following, we discuss these components and formalize the optimal choice of prototype, with fe denoting latent space projection fe (x{)) for an instance x, of class For the full algorithm, we refer Appendix
4.1 EVOLVING REPRESENTATIONS
Each observed class E )' is represented by slowly progressing prototype p" in operational memory Mp- The nearest neighbor classifier finds the most similar prototype for the given query x,, predicting
arg max pT Similar to (Mensink et al. 2013 Rebuffi et al. 2017). the class-prototyp approximates the center of mass in the latent space, which we formally justify in Section 4.4 The main crux with storing representations is to prevent them from becoming obsolete as the embedding network evolves. Additionally, this is further complicated by the shifting data distributions in the non-stations non-stationary regime, incurring catastrophic forgetting. Experience replay from a buffer M, is well known approach to address this forgetting. Nonetheless, in our setup the replayed exemplars gain additional information about the current state of the embedding space, enabling rehearsal rectify approximation pe the true center of mass. Concretely, the sampled batch
4

---

Under review as a conference paper at ICLR 2021
B, equals the horizon D from data stream S and joins batch B, of equal size from memory M,, constituting B as B, U BMM However, updating the prototypes by fully relying on features extracted from B incurs unstable optimization process the representative prototypes depend stochastic sampling of the class distributions. Therefore, we design the prototypes to evolve continually with a high momentum based update for each observed batch, aiming stabilize the impetuous changes in the data stream:
Dd ap + a)p", S.t. -=
Σ X=EB"
fo(x"),
B
with momentum parameter [0, 1]. the batch subset B = {(x,, IK- = E B) of class and pa the corresponding center of mass in latent space for the current batch. Due to triangle inequality p is no longer unit length and requires be 1.2-normalized after the update in Eq. We empirically validate the effectiveness of high momentum with a R in the ablation study in Appendix D
4.2 BALANCED REPLAY
Similar to Rebuffi et al. 2017); Chrysakis & Moens 2020)) the total buffer size M is equally divided over the number of observed classes [J] in a dynamic fashion. This scheme ensures consistent buffer capacity for all classes, making memory allocation independent the data stream characteristics. As S is typically highly imbalanced in real-world scenarios, this memory scheme prevents classes to be eradicated from the buffer and assumes equal importance to represent each class at all times. Consequently, random retrieval from the buffer resembles sampling iid replay batch. Furthermore, each class-specific replay memory M can simply capture a random subset of its parent class distribution to approximate its center of mass. This avoids computationally expensive herding techniques as in iCaRL (Rebuffi et al., (017)) which would require recalculation of the feature means on each change of the memory size or network parameters.
4.3 PSEUDO-PROTTTYPICCA PROXY LOSS
The learner optimizes fo to project an instance fç E Rd class c close to its corresponding prototype Df in the latent space. As the prototype acts as a surrogate for the class mean in latent space, the cluster population has a common reference point to reduce intra-class variance, and enforce inter-class variance by remaining distant from the other class prototypes. Additionally, due to the embedding architecture we can use intrinsic information of the batch samples in the latent space. Therefore, we exploit the supervision signal y; in a sample y., ) E not only to indicate which class x; belongs to, but also to make the distinction between positive and negative pairs in B. Consequently, we can define one-against-all subsets for an instance of class e, with positives from the same class in B = {(x., 1/4 c) E BJ and negatives in B* Starting from these sets, the prototypical attractor and repellor sets for an instance x; are constituted with the class prototype p and the other instances in B. First, the other instances of class act as preuto-rototyyes pa in attractor set F. = {pr} U {p} = fo(x)) YX E B", * j}. Second, the samples of other classes xf E B should instead avoid both x² in latent space and the class representative pº, defined by repellor set U = (p_, : (x())) .. The attractor set for x; facilitates a decrease in intra-class variance with pd safeguarding the absence of positive batch pairs with
[TP]| < |B% whereas the repellor exploits x and corresponding prototype as reference point to increase inter-class variance. To incorporate the attractor and repellor sets, we formulate a binary classification problem similar to Ye et al. (2019), with the joint probability that instance xx is predicted as class and instances x/ E B" not being predicted as class
P. = P(cx() II (1- P.(cx))))
(2)
x5
with the assumption of independence between xf and x² being recognized as We define the expected posterior probabilities for the attractor and repellor sets of instance x; respectively as
P(cxx) = E
(( p'),,
P((clx))
E prEt;
[P(cff p")]
(3)
D.EP:
with pf a proxy for the latent mean of class c in
exp(I2'p5/7
P(cf, exp(fTp'/T) Enje exp(fTp*/7)
(4)
5

---

Under review as a conference paper at ICLR 2021
where temperature controls the concentration level of the distribution (Hinton al. 2013 assuming a cosine similarity metric f f, with vectors normalized to unit length. We reformulate the objective in Eq.(2) loss function L by negative log- logllikelihood and summation over all the instances in B, which approximates the true joint probability with assumed independent pairs in the hatch:
L =
log P(c)x)) - ΣΣ log(1 P.(clx5))
(5)
B
4.4 OPTIMAL PROTOTYPES
We update prototypes to approximate the mean of the parent distribution in Eq. This assumption is optimal for Bregman divergences for which the cluster mean is shown to have minimal distance to its population (Banerjee al. 2005). This Bregman divergence is defined for differentiable, strietly convex function yp as
de (f. - (f.) ((f)) (f: ff) Vp()),
(6)
for which the squared Euclidean distance with (f() || is a canonical example. The squared Euclidean distance is proportional to the cosine distance with vectors normalized to unit length: fllf. f] 2 1 cos (f., f,), As the PPP-loss in Eq. requires a similarity measure instead of distance measure, we employ the complementary normalized cosine similarity cos Z(f,, f,) = f with |f; ||fy| = 1. Besides the desirable cluster-meat property of its complement, this metric is also efficient for calculating the full batch similarity matrix using matrix multiplication libraries.
EXPERIMENTS
This work examines five balanced data streams and 15 highly imbalanced variants based on Split- MNIST, Split-CIFARII 10 and Split-CIFAR 100, from which two low-capacity balanced setups are discussed Appendix F The learner presented a data stream S, constituted by sequence of tasks, each delineated by a subset of classes from the original dataset. Although the learner in CoPE is completely ignorant to the notion of task, this setup enables comparing to methods requiring task boundaries such as GEM and iCaRL. The evaluator uses a held-out dataset of static concepts in S Sevol, evaluating with the subset of seen concepts y in D Deval using the accuracy metric. The CoPE learner processes data online with B,, D in the data incremental setup, allowing per-tase task processing of 1 epoch for methods requiring task boundaries with B C DD We use vanilla stochastic gradient descent with limited processing batch size [B,| of 10 as in in (Lopez-Paz & Ranzato 2017: Aljundi et al.. 2019b
Lee et al. 2020). All results are averaged over different network initializations
Balanced data streams contain similar amount of data per task. We consider three benchmarks. First, Split-MNIST constitutes the MNIST (LeCun et al. 1998) handwritten digit recognition dataset with 60k training samples, split into 5 tasks according to pairs of incrementing digits. Second, Split-CIFARI considers the CIFAR10 (Krizhevsky et al., 2009) dataset, subdivided into 5 tasks with labels each, where each task entails 10k training samples. Third, Split-CIFAR100 of the CIFAR dataset with 100 different classes. The 50k training samples are subdivided in 20 tasks of 2.5k samples as (Lopez-Par Paa Ranzato [017 Lee et al. 2020). For all datasets the evaluator considers the entire original test subset for S Secol.
variant
Imbalanced data streams introduce a more realistic scenario without equality assumptions on the task durations in S and address the literature mostly balancing the data streams artificially. Besides the imbalanced Split-MNIST setup (Aljundi et al., 2019b), we introduce two novel and more challenging benchmarks based on Split-CIFARII and Split-CIFARI0N where data stream S comprises significantly more data in task T,, denoted by ST,,). Split-MNIST and Split-CIEF CIFARIO have respectively 2k and 4k samples in T,, whereas tasks T3 for + contain factor 10 less data for
five variants (T,)) Vi E (1,...,5) 5). Split-CIFARII 100 defines T, with 2.5k samples and 1 for the remaining tasks, with variants E {1, 5, .... 20).
Architectures. MNIST setups use an MLP with 2 hidden layers of 400 units with 2k memories for the balanced setup as in (Hsu al. 2018 Lee al. 2020 van de Ven & Tolias 2019) and 100 units
Appendix details the full setup with additional experiments. Code publicly released upon paper acceptance.

---

Under review as a conference paper at ICLR 2021
with M 0.3k for the imbalanced setup in (Aljundi al. 2019b). CIFAR setups slim
version of Resnet18 (He et al., 2016) with a 1k memory size for CIFARIO (Aljundi et al. 2019b Lee
et al. 2020), and 5k for CIFARI00 (Lopez-Paz & Ranzato 2017).
Methods compared to CoPE entail 11 baselines, with details on prior work discussed in Section 3
The upper reference point for performance when relaxing the challenging non-iid feature in continual learning is set by jid-online & iid-offline. The learner shuffles the full data stream $ to ensure the iid property, for which rid-online trains single epoch and rid-oflinee multiple epochs. In contrast, the finetune learner considers non-iid data stream S sequentially, but optimizes solely for the new batch which typically results in worst-case catastrophic forgetting. CoPE-CE is a reference point for the merits of prototypical approach by solely using the CoPE memory and sampling scheme, but with typical cross-entropy loss and softmax classifier. GEM and iCaRL. are standard replay methods considered in class incremental setup, with the learner requiring task boundaries. For online data incremental learning, we consider the reservoir, MIR and greedy GSS replay baselines, with CURL and CN-DPM instead relying on model expansion.
Table The three balanced data stream accuracies (%) with standard deviation over initializations. Expansion-based methods CURL and DN-CPM report results from their original work.
Split-MNIST
Split-CEFAR10
Split-CIFAR100
id-offtine
98.44 = 0.02 96.57 + 0.14
83.02 + 0.60 62.31 + 1.67
50.28 = 0.66 20.10 + 0.90
id-online
finetune
19.75 + U.U5
18,55 0.34
3.53 + 0.04
GEM
93:25 t 0.36
24.13 2.46
11.12 1: 2.48
iCARL
83.95 = 0.21
37.32 2.66
10.80 0.37
CURL (Rao et
2019
92.59 0.66
DN-CPM (T.ee 2020
93.23 0.09
45.21 + 0.18
20. 0.12
reservoir
92.16 0.75
42.48 3.04
19.57 + 1.79
MIR
93.20 + 0.36
42.80 1: 2.22
20.00 1 0.57
Figure 3: Balanced Split-
GSS
92.47 0.92
1.41 38.45
13.10 0.94
MNIST first seed Seval
CoPE-CE
91.77 1 0.87
39.73 1 2.26
18.33 1 1.52
SNEE MMaaten Hinton
CoPE (ours)
93.94 0.20
48.92 1.32
21.62 0.69
2008
CoPE: 89.8 18 COPE-EE:
:CaRL*: 53.3 =
MIR: 36.0 8.0
CaPF: 45 8.7 CsPE-CE:
KaRL, 3..5
MIR: 9.6
GEX' 87.0 + 5.7
GFM 22.8 + 2.8
60
80
Accuracy (%))
Arcurary
40-
60-
20
40
0.1k
0.2k
1.5k
2k
0.1.%
0.24
01.5k:
1AA
1.5A
24:
Memory size
Memory
sizee
Figure 4: Accuracies over buffer sizes [M for balanced Split- and Split-CIFARIO sequences. The legend reports averages over all observed buffer sizes. indicates learner with task information.
RESULTS AND DISCUSSION
6.1 BALANCED DATA STREAMS
The results for the three balanced data streams in Table cabbe[[oonssstentty report state-of-the-art for CoPE. The difficulty for learning online is reflected in the discrepancy performance between iid-offline and iid-online. indicating increasing difficulty for minimal 2% for Split-MNIST, raising by factor 10 for Split-CIFARI and culminating to 30% in Split-CIFAR 100. For Split-MNIST the gap with iid-online performance is closed by 0.7% compared main competitors GEM and DN-CPM, with
7

---

Under review as a conference paper at ICLR 2021
representations visualized in Figure Furthermore, in the more challenging Split- Split-CIFARIO setup we significantly increase the gained margin by 3. 3.7%. In the most challenging Split-CIFAR100. CN-DPM, Reservoir and MIR are able to perform par with the iid-online baseline, however, CoPE establishes an improvement of at least 1.5% over all four baselines.
Compared to balanced replay with standard cross-entropy (CoPE-CE), the prototypical approach (CoPE) proves effective with significant gains of 2.2%, 9.2% and 3.3% respectively over the three benchmarks. Except for GEM in Split-MNIST, class incremental learning methods GEM and iCaRL are not competing in the online setting and additionally require from the setup to reveal an identifier I the learner. From the expansion ppasse methods DN-CPM is competitive, whereas CURL is more suited for unsupervised learning and lacks behind. Although Reservoir and extension MIR perform on par with iid-online for Split-CIl Split-CIFAR 100, the imbalanced experiments in Section 6.2 show that full reservoir- based population of the buffer strongly relies on this assumption of equally sized tasks, which is unlikely occur in real-world data streams.
Buffer size ablation study in Figure 4 shows CoPE to prevail over all sizes of replay buffer M, compared to other replay methods, extending robustness to low capacity regimes. Although iCaRL shows competitive results for low capacity, CoPE scales with growing capacity leading to significantly outperforming iCaRL with 11% in Split-MNIST (2k) and Split-CIFARIOO (5k), and 17% in Split- CIFARIO (2k). We refer to Appendix for the Split- Dolit-CIFARION results.
6.2 IMBALANCED DATA STREAMS
Results for the highly imbalanced data stream benchmarks are reported in Figure CoPE significantly outperforms all baselines in the three scenarios, with low standard deviation for the 15 variants indicating robustness over wide spectrum of imbalanced sequences. Gradient-baus based sample selection (GSS) outperforms Reservoir and MIR for SpLit-MNIST, in correspondence with results in [Aljundi] et al. (20T9h), whereas loss-based retrieval in MIR has significant gains for the challenging Split- CIFARTO0 setting. However, CoPE surpasses both GSS and MIR for all three benchmarks, and on top of that operates profusely more resource efficient as discussed in Appendix [ The balancing memory scheme in COPE-CE highly improves Reservoir over imbalanced Split-MNISS and Split-CIFARI I0 variants with 10.8% and 3.4% respectively, and performs par for over 100 classes with limited batch size proves more difficult. Although CoPE and CoPE-CE share memory and retrieval schemes, the prototypical CoPF surpasses the cross-entropy based COPE-CE with 4.0%, 2.9% and 6.7% respectively on the three benchmarks, indicating the merits of the PPP-loss and continually evolving prototypes. Figure 6 compares the CoPE and CoPE-CE confusion matrices at the end of learning, showing that CoPE better preserves the recall over early learned classes. CoPE-CE exhibits high plasticity classes 8 and 9 of the last task have high recall compared to the earlier learned classes. Hence, CoPE seems to better preserve stability, effectively alleviating catastrophic forgetting.
where balancing
Colfee 84-4 +
MIlR: 70.0 4.9 Reservci:: 69.7 4: 4.5
CoPE: 37.4 1.7
MIR: 20. 233
CaPE: CaPFSCR: 11.9- 0S
04
MIR 17,8 = 0.3
CaPE CE 0.9
CAPF-CR: 94.5 1.1
Rosemor:: 31.4 + 2.7
Reservair: 11,8 13++00 0.5
GSS: 78.9 2,0
CSS: 30.6 2.8
CSS: 10.8 0.4
40
80
15
30
60
10
20
S(T)) S(T.)
S(T)) S(T)) S(T))
Ave.
S(T))
S(7))
S(T)
S(T.)
S(T.)
Avg.
S(T)) S(73))
8(7m))
S(T')) Avg.
Figure Accuracy (%) for imbalanced Split-MNIST (left), Split-CIFARI0 (center) and Split-
CIFAR100 (right) sequences. The legend reports average accuracies over all the sequence variations.
6.3 PPP-LOSS ANALYSIS
In the challenging setting for online processing of non-iid data streams, the PPP-loss exploits information in the small processing batch B, introducing This leads to questioning to what extent the pseudo-prototype actually contribute to the quality of the embedding, and how this relates to the batch size. We examine both inquiries in Table for the
p on top of the prototypes.
8

---

Under review as a conference paper at ICLR 2021
023
546
021
0.18
36L
3.00
011
0.00
8/00
923
CCP0 1X2
0/2
0:00
9.0
9/0
0.00
DX2
100
1:00
527
800
035
9.2
912
9/0
0.33
8.00
0.5
8.00
309
0.00
010
112
91h
033
3:01
8:19
0113
310
910
9:91
000
0000
6:00
000
049
0.00
8.0
0.34
8:38
043 6.0
500
042
000
0.00
8:03
034
000
613
0.03
8.40
360
0.99
1:3
324 065 6.5
8.1
031 611
0:00
0:5
0:31
0.6
0.2
0.01 9:00
043 6.5
16
035
8:0
0.02
9:02
031
100
814
000
0.03
0.00
309 03T
8:1 300 031
6:0
0.00
500
030
360
083
500
(a) CoPB
(b) CoPE-CE
(c) CoPE
(d) CoPE-CE
Figure 6:
CoPE and CoPE-CE confusion matrices the end of learning averaged over all variations S(T)) for the imbalanced Split-MNIST setup in (a) and (b), and Split-CIFARIO in (c) and (d).
three balanced data streams by comparing inclusion and exclusion of the pseudo-prototypes p in the PPP-loss, and extending the batch size [.,] First, including the peuudo-rotottpes significantly improves overall performance, and especially for the harder CIFAR-based data streams. Although both setups use batch information to update the prototypes following [.. it seems crucial to use additional pseudo-prototypes in the PPP-loss to improve latent space quality. Second, results for smaller batch sizes of 10 and 20 are very similar, and deteriorate towards increasing sizes. The PPP-loss implements the expectation over the prototype and the pseudo-protee prototypes, assuming uniform distribution in E3. Although this assumption impedes significance of the prototype for increasingly higher batch sizes, it results in ideal robustness for small online processing batches, ideally suited for data incremental learning. Small batches maintain the additional benefit of more frequent prototype updates for the same amount of processed data.
Table 2: Accuracies (%) for ablating pseudo p in the PPP-loss and varying batch size.
Pos-loss
Batch Size Bpp
incl.
excl. P
10 (Online)
20
50
100
200
Split-MNIST CDIt-CIFARIO
93.9 = 0.2
92.4
93.9 0.2
93.9 1:00::000 0.6
93.7 : 0.3
93.1
89.3
48.9 = 1.3
41.3
48.9 1.9+133 1.3
1.9
4
3.0
37.0 1.3
Spli--CIFAR100
21.6 + 0.7
0.7
21.6 0.7
21.7 0.7
16.5 0.4
0.5
1.
CONCLUSION
In this work, introduced new perspective current paradigms in continual learning with novel two-agent learner-evaluator framework. To overcome the standard paradigm of static training and testing phases, explicitly model continual optimization and evaluation in the learner and evaluator agents respectively. We formalized the required resources as the horizon D, containing the simultaneously available data of the data stream, and the operational memory M for operation of the learning algorithm. Transitions in the horizon D. + DD+1 enable a uniform differentiation between existing paradigms of task, class and domain incremental learning, and the horizon size encloses the range from online (D = B) to offline (D = S) learning.
Using the framework, we defined the task-free data incremental learning paradigm, requiring no additional information on the identifier t of the horizon for both the learner and evaluator. In this challenging setup, we proposed Continual Prototype Evolution (CoPE) as a prototypical solution to learn online from non-stationary data streams. As first, CoPE prevents the prototypes becoming obsolete in an ever evolving representation space, while using the prototypes to combat catastrophic forgetting. The three main components, continually evolving prototypes, novel Pseudo- Pseudo-Prototypice Prototypical Proxy loss (PPP-loss), and an efficient balancing replay scheme are proven remarkably effective over 11 baselines in both balanced and highly imbalanced benchmarks. We hope to encourage research in the direction of data incremental learning with online processing of data streams and applications beyond classification.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference Computer Vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 11849-1188 11860, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pp. 11816- 11825, 2019b.
Arindam Banerjee, Srujana Merugu, Inderjit S Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. Journal of machine learning research, ((Oet):1705-1)) 2005.
Francisco M Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv: 2019.
Aristotelis Chrysakis and Farie-Frannine Moens. Online continual learning from imbalanced data. Proceedings of Machine Learning Research, 2020.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. arXiv preprint arXivv1909.0880 2019.
Matthias De Lange, Xu Jia, Sarah Parisot, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Unsupervised model personalization while preserving privacy and scalability: An open problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
14463-1444 14472, 2020.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128- 135, 1999.
João Gama, Indré Žliobaité, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1-37, 2014.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation catastrophic forgetting gradient-based neural networks. arXiv preprint 13126211 2013.
Stephen Grossberg. Studies of mind and brain neural principles of learning, perception, development, cognition, and motor control. Boston studies in the philosophy of science 70. Reidel, Dordrecht, 1982. ISBN 90277132 03596.3
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0L) (06), volume 2, pp. 1735-1742. IEEE, 2006.
Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Continual relation learning via episodic memory activation and reconsolidation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6429-66666 6440, 2020.
Ben Harwood, BG Kumar, Gustavo Cameiro, Ian Reid, Tom Drummond, et al. Smart mining for deep metric learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2821-2829. 2017.
10

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255: 2015.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint arXiv:/810. 12488, 2018.
Daniel Kahneman and Amos Tversky, Subjective probability: A judgment of representativeness. Cognitive psychology, 3(3):400-444, 1972.
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv: 10368, 2017.
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partitioning reservoir sampling. arXiv preprint arXiv:2009.03632, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnskaaaariikka et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 96(11))2278-2
1998.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. arXiv preprint (ixv:200100068) 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40( (0(12))2935-22) 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems. pp. 6467-6476, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9NNov):557--2655, 2008
Arun Mallya and Svetlana Lazebnik. Packnet:
Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018
Thomas Mensink. Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes at near-zero cost. IEEE transactions on pattern analyvis and machine intelligence, 35(11):2624-263, 2013.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4004-4012, 2016.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems, pp. 7645-7655, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Jeffrey C Schlimmer and Richard Granger. Beyond incremental processing: Tracking concept drift. In AAAI, pp. 502-507, 1986.
11

---

Under review as a conference paper at ICLR 2021
Ari Seff, Alex Beatson, Daniel Suo, and Han Liu. Continual learning in generative adversarial nets. arXiv preprint 1705.. 08395, 2017.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. arXiv preprint arXiv: 0101.11001111.. 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim, Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-29999 2017.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3400-3400 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Alexey Tsymbal. The problem of concept drift: definitions and related work. Computer Science Department, Trinity College Dublin, 106(2):58, 2004.
Gido M van de Ven and Andreas S Tolias. Generative replay with feedback connections as general strategy for continual learning. arXiv preprint 18Xiv: 10635, 2018.
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07334 2019.
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(15:37-57, 1985.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Zhengyou Zhang, and Yun Fu. Incremental classifier learning with generative adversarial networks. arXiv preprint arXiv:
2018.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6210-6219, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3987- 3995. JMLR. org, 2017.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A ALGORITHM
Our proposed algorithm is fully formalized in this section, well in our code that will be made publicly available on acceptance of this paper. Algorithm [ and Algorithm describe the learner for CoPE, whereas the evaluator uses c* arg max f pº, classifying
category cº with the most similar prototype p" As for true continually progressing system, the evaluator can urge prediction at any point in time, while the learner keeps acquiring knowledge from the data stream.
Algorithm 1
The CoPE learner in the data incremental learning setup.
Require: data stream S, prototype momentum memory capacity M, learning rate 7)
Initialize operational memory M = 0, observed classes y
0. sample count per class N =
model parameters
1: for B, = (x11, B2 Y|B,, ~ S do B..))
>> Data stream batch w/o task information Randomly sample
2: BM f- RANDOMSAMPLE( 0
[,, exemplars from M,
3: B
4:
for (x, y.)) E Bn U BM do
5:
if y/s $ D then
6:
INITCLASS(M,N.).. N.
> Initialize memory and prototype
7:
end if
8:
B <
B U fo(x;)
Collect features
9:
end for
10:
2-0 C+0 0
Initialize loss
11:
for f B do
loo P(cxx)) Exy log(1 P(c|x)))
C+
Sum al instances PPP-loss
18
13:
end for
14: + 0 + VC
Optimize objective with SGD Update prototypes in Mp
15: PROTOTYPUUPDATE
16:
UEMORYUPPATEEMM N)
Update memory M, with new input samples
17:
end for
Algorithm Memory Management the replay memory and prototypes. UNIFORM samples elements in a d-dimensional vector with uniform probability in range [s1, 82] E R.
Rd (81,82)
Require:
memory capacity M
I: function PROTOTYPEUPDATEEMM, B,N,a
1: function IN N,J/,0
2:
for p M, do
2: N + N (NV 0}
b Sample counts
3:
No +- N |Bq]
3:
Observed classes
4:
pe B L f
4:
m m M/
> Capacity per class
pe < ap + a)p
5:
for M = (x1, m. {(MM M, do
6:
p < pl/lpp/11
Normalize
6:
M (Xi,..., Xm)
Keep first m
7:
end for
7:
end for
8:
end function
8:
M + MU 0)
9:
function MEMORY UPDATE,,, N,
9:
& UNIFORM (0,
10:
for x² B, do
D Class Reservoir
10: MY (p" (p"//lp"||>>
> Init prototype
11:
j == UNIFORM (1,
end function
12:
if [M then
13:
M [] + xx Replace exemplar
14:
end if
15: end for
16: end function
13

---

Under review as a conference paper at ICLR 2021
B SETUP
A gridsearch in the online continual learning setup was adopted, selecting the setup with highest performance, similar to (Lopez-Paz & Ranzato 2017). All methods are prone to learning rate gridsearch [0.05, 0.01, 0.005, 0.001 iCaRL knowledge distillation strength is set to 1, and GEM bias is set to 0.5, following (Lopez-Paz Paz & Ranzato 2017: Rebuffi et al. 2017;; Aljundi et al., 2019b). GSS and MIR follow their original setup from their codebase in (Aljundi al. 2019b) and (Aljundi et al., 2019a). with our additional learning rate gridsearch. CURL (Rao et al.
2019) and DN-CPM (Lee) et al. 2020) results, and the best imbalanced Split-MNIST results out of the greedy/IQP versions for GSS (Aljundi et al. 2019b) are reported from their original works. CoPE searched for a suitable temperature т = 0011, 0.2, 1,2] which was set to 0.1 for all balanced and imbalanced Split- MNIST and Split-CIFARII 10 experiments, similar to YYe et al. 2019). Based on the ablation study in Appendix D we set the prototypical momentum fixed to 0.99. For the challenging Split- Split-CIFARII setting methods are allowed multiple iterations per batch as in (Lopez-Paz & Ranzato 2017). from which the best results are selected (baselines, reservoir, CN-DPM perform iteration, others 5). The CIFAR100 temperature required higher concentration with r = 0.05 and prototypical momentum 0.9. For the balanced setups, the latent dimensionality d is fixed to 100 for Split-MNIST as in (Rao et al. 2019), and selected 256 in gridsearch [128, 256] and [128, 256, 512] for Split-CIFARTO and Split-CIFARI 100 respectively. The imbalanced benchmarks follow the low capacity setup in Appendix E. I with d [16, 32, 64] set to 64 for Split-MNIST and d [128, 256] set 128 for Aplit-CIFARTI and 256 for Aplit-CIFAR1ON
Results are obtained without L2 normalization of the prototypes as we found it to have insignificant effect. The CIFARIO labels in the confusion matrices from 0 to 10 stand for the indices in the following list: [airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). We will make our code publicly available upon acceptance of this paper to ensure reproducibility.
C RESOURCE ANALYSIS TASK-FREE REPLAY METHODS
In this section we compare usage of computational and memory resources for the replay methods fitted for the online data incremental learning paradigm.
Reservoir is a powerful baseline for balanced data streams (Chaudhry et al. 2019). with only minimal computational cost by keeping count n of how many samples have been observed. This count is then used relative to the buffer size M to define the probability M/n to store the new sample. As shown in the imbalanced data stream experiments, Reservoir is not fit for more real-world scenarios with typically varying frequency of occurrence per class. Improving this simple experience replay has led to research focusing on more complex strategies, discussed in the following.
MIR (Aljundi et al., 2019a) replaces the random retrieval from the buffer in Reservoir with a loss- based approach. They store a momentary update of the network optimized for the new incoming batch and calculate the change in loss for random subset of replay memories B. which is larger than the batch size (ideally five times the batch size for their experiments (Aljundi et al.) (019a)). Besides a copy of the full model, this also requires calculating the loss twice in a sequential manner for the full subset B and an extra temporary model update using only the new batch B,, both significantly increasing processing time for the learner.
GSS (Aljundi et al. 2019b) resides with Reservoir to use random retrieval of the buffer, but proposes a gradient-baser population strategy. They introduce two variants, in which the first solves an Integer Quadratic Problem (IQP) with polynomial complexity w.r.t. the replay memory. As this is not scalable, they also propose a stochastic GSS- -SSS-gredyy variant. This more efficient GSS-greedy approach requires
additional forward pass, loss calculation, and backwards pass to obtain the gradients for the full considered subset B in the memory. Additionally, it uses similarities of the gradients for stochastic sample selection in the replay memory My, straining memory requirements as batch B, requires for each sample [ + 1 gradients to be accessed simultancously to calculate [B] cosine similarities in the high- nigh-dimensional gradient-space.
CoPE (ours) resembles Reservoir's 's memory population by keeping count of the samples per class- specific replay memory subset. The PoP-loss requires calculation of a similarity matrix with all the d-dimensional representations in the batch B. Using a normalized cosine similarity, this implies
14

---

Under review as a conference paper at ICLR 2021
efficient matrix multiplication with the low-dimensional vectors. This is in high contrast to GSS, which calculates cosine similarity in the full high-dimensional gradient space for additional samples that are not present in current batch B, and therefore requires additional costly forward and backward passes. Furthermore, in our prototypical approach the prototype momentum updates also rely solely on samples that are in the current batch B, hence requiring only minimal additional computation. Comparing to both MIR and GSS, we don't tt require storing model copies or additional gradients, but merely store low-dimensional prototypes for each class, saving a significant amount required storage space. For example, Resnet 8 model requires 11.7 million parameters to enable model copies or gradients, whereas our method even for 1000-way classification with d 1024 would require only 9% of the model capacity in memory for the prototypes.
D EXTENDED ABLATION STUDY
D.1 ABLATION PROTOTYPE MOMENTUM
In all experiments, a high momentum is employed to update prototypes with the latent mean of the batch. Table illustrates the influence of higher momentum (2 0.9). Compared to low momentum of 0.1, Split-MNiSST only gains small margin of 0.45%, whereas Split- CIFit-CIFARIOO and Aplit-CIFARI significantly improve with at least 3.0% and 4.2% respectively. Using momentum prevents the prototype to rely solely on the current batch instances, and higher momentum values attain more gradual change of the prototypes by stabilizing its trajectory in the ever-evolving latent space.
Table 3: Ablation study changing momentum strength for prototype updates, reported in average accuracy (%) over 5 runs. Higher momentum values (> 0.9) obtain better performance, especially for the CIFAR sequences, compared to low momentum (0.1).
Prototype Momentum
0.1
0.9
0.95
0.99
Split-MNIST
93.49
0.70
94.11 + 0.34
93.96 + 0.30
93.94 + 0.20
Spli-CIFARR10
44.48 + 3.19
48.02 + 2.49
47.98 3.14
48.92 + 1.32
Split-CIFAR100
15.791..1
21.62 + 0.69
21.56 = 0.58
20.01 = 1.81
D.2 ABLATION INTER AND INTRA-CLASS VARIANCE TERMS PPP-LOSS
In this section, the importance scrutinized of the two loss components to enhance inter and intra- class variance in the PPP-loss. Table 4 compares using only positive pairs from the batch in the attractor (Epoe) or only negative pairs in the repellor (Lneg) to the flld-feedged PPP-loss (C). The attractor term shows competitive performance the full PPP-loss for SNlit-MNIST, but deteriorates as the data streams become harder for the CIFAR setups. The repellor term is on par with the full PPP-loss for SpLit-MNIST and Split-CIFAR1O, but collapses for Split-CIFARR/ 100. The latter is challenging due to the high number of classes with only a batch size of 10, which impedes having grcudo-prototypes of all classes in the same batch. The PPP-loss incorporates both reduction of intra-class variance with the attractor term and increases inter-class variance with the repellor term, attaining state-of ofetthe-rt performance.
Besides isolating the attractor and repellor terms of the PPP-loss in the ablation study, we further investigate the weighing of the two terms during the lifetime of the learner in Figure We average results over 5 runs for balanced Split-MNIST, finding the repellor to dominate. This trend is to be expected as the repellor term in Eq. (5) has per instance a summation over all other class instances. The attractor term has minimal influence especially for data presented for the first task. This indicates the samples in the binary latent space (having observed only two classes) majorly repelling rather than attracting samples. The embedding network is still learning the initial features, and overlap in the two latent class distributions summed over the other class samples results in prevailing repellor term.
15

---

Eneg 93.84 0.48 Under review as a conference paper at ICLR 2021
Table 4: Ablation using solely the attractor (Cpos) repellor (Cneg) compared to using both terms in the PPP-loss (L).
PPP-loss
C
Pps
Split-MNIST
93.94 + 0.20
93.25
0.22
Split-CIFARI0
48.92 + 1.32
30.96
3.58
49.30 3.57
Split-CIFAR100
21.62 0.69
15.85 = 0.34
9.43 + 0.94
1. (9)
10k
20A
30k Split MNIST
40k
50k
60k
Figure 7: Weighing (%) between the positive loss term Lyns compared to the full PPP-loss L, averaged over 5 runs of balanced Split-MNIST with standard deviation in blue.
D.3 PSECDO-PROTOTYPE ABLATION VISUALIZATION
In the main paper we find in an ablation study that using pseudo-prototypes P as proxy for the class-mean has significant improvements for the PPP-loss. Additionally, Figure F shows this in 2-dimensional t-SNE space for the first seed of the balanced Split-MNIST experiment. Including the pseudo-prototypes (incl. p) illustrates striking degree of inter-class variance in Figure Sh. whereas more interference occurs when excluding the pseudo-prototypes in Figure Sp. This reflected performance, including prototypes results in 94.52% accuracy, whereas excluding them has only 90.86% for the first seed.
(a) PPP-loss incl.
(b) PPP-loss excl.
Figure 8: Split-MNIST first seed t-SNE representation of the test data Sevat, including (a) and excluding (b) the pseudo-prototypes p in the Pos-loss.
E ADDITIONAL EXPERIMENTS
E. BALANCED DATA STREAMS WITH LOW CAPACITY
In these experiments we scrutinize performance of CoPE with less capacity in the memory and model, and with shorter data streams. All methods allowed multiple iterations (maximal 5) in (Aljundi)
16

---

Under review as a conference paper at ICLR 2021
Results averaged over 5 seeds. Similar to the setup of GSS (Aljundi al. 2019b). we adopt two data sequences with truncated data per task:
al.
2019b).
Spli--MNIST-mini is similar to the Split-MNIST data stream with 5 tasks, but each task is confined to IK training samples. Evaluation considers the full test subset. The network is an ML.P with two hidden layers 100 units, with total memory size of 0.3k exemplars. Latent dimensionality d is selected 32 from [16, 32, 64).
Split-cIFAR1Omiin is similar to the Split- OPFIR-CIFARIO data stream with 5 tasks, but each task comprises 2k training samples, with total subset of 10k samples out of the 50k available. The full test subset is used for evaluation. The network used is the same ResNet18 8 as in the main paper, with total memory size of 1k exemplars. Latent dimensionality d is selected 128 from [128, 256].
Analysis. Table shows the results for SpLit-MNIST-mini and Split-CIFARIO-me with GSS and DN-CPM results reported from their original works in corresponding setup. In Split-MNIST-minn our method approaches the iid-online baseline up to 1%, and outperforms its closest competitors GEM and MIR with least 1.45%. In CAIAAIIO-miiiiiiiiii CoPE saliently surpasses the iid-online baseline with 2.25%, hence outperforming online training over an iid datastream. Moreover, CoPE surpasses CN-DPM by 3%. Reservoir proves a strong baseline, with in this case the additional MIR loss-based retrieval decreasing performance. Similar to our findings in the main paper and Aljundi al. 22019 b), GEM encounters difficulties a CIFAR based setup, for which we find the bias hyperparameter y 2 0 in the gradient projection to have insignificant influence. These results confirm CoPE outperforming both GSS and CN-DPM this low capacity setting established in their original work.
Table 5: Split-MNIST-mini and (IFAARO-mmnn results, with respectively only 1 and 2k samples task. GSS and DN-CPM results reported from original work in these setups.
Split-MNISTTmini
Split-CIFARI0-minl
iid-offline
94.58 + 0.17 87.57 + 3.54
67.41 # 1.37 42.50 + 2.15
iid-online
finetune
21.74 + 3.38
16.65 0.24
GEM
85.09 + 0.52
22.31 + 1.37
iCaRL
83.23 + 0.92
26.54 + 2.73 41.78
DN-CPM (eee al. 2020
reservoir
82.73 + 2.39
38.21 + 3.39
MIR
84.40 + 0.91
37.20 2.74
GSS (Aljundi et al. 2019b
82.60 + 2.90
33.56 + 1.70
CoPE
86.54 + 1.41
44.75 2.68
E.2 BUFFER SIZE ANALYSIS: SPLIT-CIFARI00
The results for Split-CIFAR10 and Split-MNIST are reported in the main paper, whereas Split- due to lack of space. We observe the same trend, where CoPE prevails over other replay methods by high margin from low to high-capacity regimes. The performance of the learner in CoPE scales with the size of M,.
CIFARIO0 results are added here Figure due
E.3 UNABALANCED BENCHMARK RESULTS
The graphs the main paper visualize the numbers in Table 6 which fully report here reference for future work. Each S(T;) data stream performance is averaged over five different initial seeds. The 'Avg.' results average over all mean performances of the dataset variants S
17

---

Under review as a conference paper at ICLR 2021
CoPE: 16.8 4.3
iCaRL* 10.6 CSS: 11.1 + 2.3
MIR: 14.9 4.4
CoPE-CE: 14.0 4.1
Reservoir: 14.1 + 4.6
GEM* 9.4 + 1.6
Accuracy (%)
20
10
1k
2k
3k
4k
5k
Memory size
Figure Accuracies over buffer sizes [M| for balanced Split-CIFARI00 sequence.
Table 6: Numeric results for imbalanced Split-MNIST, Split-CIFARI0 and Split-CIFAR10 100 se- quences.
Imbalanced Sequence
Dataset
CoPE
CoPE-CE
GSS
MIR
Reservoir
Split-MNIST
S(T))
83.4 2.0
81.8 1.2
75.9
64.8 5.1
2.3
S(T,)
84.5 1 1.6
78.5 7.0
8.91
6.2
65.5 4.6
S(I₂)
85.1 + 0.6
2..
7.04
4.0
S(T.)
84.8 1.0
8.10
0.6
72.6
7.4
S(T;)
84.0 + 1.3
80.7 1.8
7.71
77.2
73.2 4.0
Avg.
84.4 + 0.7
80.4 + 0.9
78.9 2.0
70.9 + 4.9
69.7 + 4.5
Split-CIFAR10
S(Ti)
39.0 1.3
36.4 3.0
32.6 3.0
35.5 3,4
S(T,)
35.3 2.6
34.1
2..
2.8
29.3 2.8
S(T+)
2.5
34.6
29.5
29.6
2.1
S(T.)
2.41
33.5
1.3
31.0
0.6
S(Ts)
37.3
+
28.3 3.3
28.8 27.6
Avg.
37.4 + 1.7
34.5
30.6 2.8
2..6
31.4 + 2.7
Split-CIFARR10
S(T1)
18.2
11.7 0.6
10.2 0.8
18.4
11.1 + 0.6
S(Ts)
1.3
1.26
1.5
17.6
11.5 1.4
S(T1)
0.9
1..1
11.1
17.8
1.7
S(T1s)
1.67
1..2
11.1
17.8
1.81
S(T20)
18.5
12.8
11.1
17.6
12.51.1
Avg.
18.6 + 0.4
11.9 0.8
10.8 0.4
17.8 0.3
11.8 + 0.5
18


Reviews:
[{"review_id": "u5qnAVszzt5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new procedure for continual supervised learning from a non-IID data stream that assumes ability to maintain some of the stream examples in a buffer and use the buffer to improve updates of a prediction model. The proposed procedure is called Continual Prototype Evolution (CoPE), which controls evolution of prototypes to prevent catastrophic forgetting. \nStrenghts:\n- the paper presents very detailed experimental results\nWeaknesses:\n- the paper is not easy to read\n- the underlying assumptions about the data stream are not clearly defined. Thus, it remains unclear what problem the proposed algorithm is trying to solve.\n- the proposed algorithm is a collection of heuristics that are not clearly justified. There is no attempt to provide a theoretical insight about the behavior of the algorithm.\n- the experiments were performed for one particular synthetic setup on one benchmark data set and the proposed algorithm is compared to a very limited class of baselines. Thus, even the empirical evaluation is not very insightful.\n- there is no discussion about the computational cost \nOverall rating:\nThis paper has too many weaknesses and is not ready for publication\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pwCkCTYjcG8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis article introduces a learner-evaluator framework that incorporates the different variations of problems related to incremental learning. It also proposes a method called Continual Prototype Evolution for dealing with the most general version of the problem, incremental learning on data streams, in which the learning task is not specified. The paper presents an extensive amount of experiments indicating that in this scenario, the proposed method improves significantly on existing approaches in terms of accuracy and memory efficiency. The article is well organized, easy to read and understand.\n\nPositive aspects:\n\n- It presents an adequate coverage of the literature on continual learning in section 3 and an interesting framework organizing the area in section 2.\n- The method does not need information about the task being learned, being more applicable to real-world scenarios.\n- An extensive experimental evaluation of the proposed method and comparisons with related methods is provided.\n- Ablation studies indicate the most important aspects of the proposed model. \n\nPoints to discuss/improve:\n\n- Momentum parameter: It took me a while to understand why the authors use a “high momentum”. While in gradient descent the momentum creates a tendency to keep the parameter changing in the previous directions of motion, here the momentum is supposed to make it change more slowly. There is another way of formulating Eq. 1, in which alpha works as a learning rate. Defining alpha = (1-alpha), a slow learning rate instead of high momentum, we can have Eq. 1 as: p^c = p^c + alpha(p^c - \\bar{p}^c). This is easier to understand in my opinion, but authors can choose to disregard this if they prefer the current form.\n\n- The method assumes that one prototype for each class is enough. However, in certain problems, a class might need to be represented by different prototypes, indicating the different ways of being from the same class. How the model would handle these situations?\n\n- The conclusion does not discuss adequately the main findings of the article, probably due to lack of space.\n\n- Page 15 is completely black in most PDF viewers I tried. Only Chrome was able to display it correctly.\n\nConclusion:\n\nI believe this article should be accepted as a see it presents interesting findings in the area of incremental learning on non-stationary data streams.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VhGLunpaCVS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper covers an interesting topic of continual learning of the stream of data. One limitation of the existing classification algorithms is their close-set assumption. In close-set methods, a predefined set of classes are considered and a model is trained on the available data from these classes, based on the assumption that test data will be driven from a similar distribution as the training data. However, most of the real-world problems are open-set problems. Open-set models should be able to learn continuously in an online manner with minimum or zero supervision. In other words, they should be able to learn new classes or update the existing classes based on the received new data on-the-fly, without forgetting the previously learned knowledge.  \nPros: \nIn this paper, the authors provide an incremental learning approach that prevents catastrophic forgetting. \nTheir approach can work on both balanced and unbalanced data.  \ncons: \nThe authors need to improve the presentation of the manuscript by providing more explanation. It could be confusing for readers who are not familiar with the topic. \nIt would be very helpful to publically share the code. \nI highly recommend adding the confusion matrix or F1 score in addition to the accuracies. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an interesting idea for task-free incremental learning on the data stream. The reviewers have extensive discussions after reading all the reviews and the author's rebuttal. There are concerns raised about the presentation of the method and the justification for some parts of the model design choices. The reviewers believe that after addressing these weaknesses the work can be made stronger and may be accepted in a competitive venue.  
Decision: Reject