Unsupervised Progressive Learning and the STAM Architecture
Authors: James Smith
Cameron Ethan Taylor
Seth Baer
Constantine Dovrolis
Venue:  (2021)
Keywords: continual learning
unsupervised learning
representation learning
online learning

Abstract: We first pose the Unsupervised Progressive Learning (UPL) problem:  an online representation learning problem in which the learner observes a non-stationary and unlabeled data stream, and identifies a growing number of features that persist over time even though the data is not stored or replayed. To solve the UPL problem we propose the Self-Taught Associative Memory (STAM) architecture. Layered hierarchies of STAM modules learn based on a combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical features rather than specific examples. We evaluate STAM representations using classification and clustering tasks. While there are no existing learning scenarios which are directly comparable to UPL, we compare the STAM architecture with two recent continual learning works; Memory Aware Synapses (MAS), and Gradient Episodic Memories (GEM), which have been modified to be suitable for the UPL setting. 

Under review as a conference paper at ICLR 2021
UNSUPERVISED PROGRESSIVE LEARNING AND THE STAM ARCHITECTURE
Anonymous authors Paper under double-blind review
ABSTRACT
We first pose the Unsupervised Progressive Learning (UPL) problem: an online representation learning problem in which the learner observes a non-stationary and unlabeled data stream, learning a growing number of features that persist over time even though the data is not stored or replayed. To solve the UPL problem we propose the Self-Taught Associative Memory (STAM) architecture. Layered hierarchies of STAM modules learn based on combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical features rather than specific examples. We evaluate STAM representations using clustering and classification tasks. While there are no existing learning scenarios that directly comparable to UPL, compare the STAM architecture with recent continual learning models, Memory Aware Synapses (MAS) and Gradient Episodic Memories (GEM), after adapting them in the UPL setting.
INTRODUCTION
The Continual Learning (CL) problem is predominantly addressed in the supervised context with the goal being to learn a sequence of tasks without "catastrophic forgetting" (Goodfellow et al., 2013; Parisi et al., 2019; van de Ven & Tolias, 2019). There several CL variations but common formulation is that the learner observes a set of examples { 4, y.),, where x; is a feature vector, is task identifier, and y; is the target vector associated with (Izz, 1.) (Chaudhry et al., 2019a;b; Kopez-Paz & Ranzato, 2017). Other CL variations replace task identifiers with task boundaries that are either given (Hsu al.. 2018) or inferred (Zeno et al., 2018). Typically, CL requires that the learner either stores and replays some previously seen examples (Aljundi et al., 2019a;b: Gepperth & Karaoguz, 2017: Hayes al., 2019; Kemker et al., 2018; Rebuffi et al., 2017) or generates examples of earlier learned tasks (Kemker & Kanan, 2018 8: Liu et al., 2020; Shin et al., 2017).
The Unsupervised Feature (or Representation) Learning (FL) problem, on the other hand, is unsu- pervised but mostly studied in the offline context: given a set of examples {x;}. the goal is to learn a feature vector h; = f(x;)) ) of a given dimensionality that, ideally, makes easier to identify the explanatory factors of variation behind the data (Bengio al., 2013). leading to better performance in tasks such as clustering or classification. FL methods differ in the prior P(h) and the loss function. Autoencoders, for instance, aim to learn features of a lower dimensionality than the input that enable a sufficiently good reconstruction (Bengio, 2014; Kingma & Welling, 2013; Tschannen et al., 2018 8; Zhou al., 2012). A similar approach is self-supervised methods, which learn representations by optimizing an auxiliary task (Berthelot et al., 2019; Doersch et al., 2015; Gidaris et al., 2018; Kuo al., 2019; Oord et al., 2018; Sohn al., 2020).
In this work, we focus on a new and pragmatic problem that adopts some elements of CL and FL but also different than them we refer to this problem as single-pass unsupervised progressive learning or UPL for short. UPL can be described as follows:
the data observed as non-Ill stream (e-g., different portions of the stream may follow different distributions and there may be strong temporal correlations between successive examples), the features should learned exclusively from unlabeled data, 3. each example is "seen" only once and the unlabeled data are not stored for iterative processing, the number of learned features may need to increase over time, in response to new tasks and/or changes in the data distribution,
to avoid catastrophic forgetting, previously learned features need to persist over time, even when the corresponding data no longer observed in the stream.

---

Under review as a conference paper at ICLR 2021
The UPL problem is encountered in important Al applications, such as robot learning new visual features as it explores a time-varying environment. Additionally, we argue that UPL is closer to how animals learn, at least in the case of perceptual learning (Goldstone, 1998). We believe that in order to mimic that, MI methods should be able to learn in a streaming manner and in the absence of supervision. Animals do not "save off" labeled examples train in parallel with unlabeled data, they do not know how many "classes" exist in their environment, and they do not have to replay/dream periodically all their past experiences avoid forgetting them.
To address the UPL problem, we describe an architecture referred to as STAM ("Self-Taught Associa- tive Memory"). STAM learns features through online clustering at a hierarchy of increasing receptive field sizes. We choose online clustering, instead of more complex learning models, because it can be performed through single pass over the data stream. Further, despite its simplicity, clustering can generate representations that enable better classification performance than more complex FI. methods such as sparse-coding or some deep learning methods (Coates et al., 2011; Coates & Ng, 2012). STAM allows the number of clusters to increase over time, driven by novelty detection mechanism. Additionally, STAM includes a brain-inspired dual-memory hierarchy (short-term versus long-term) that enables the conservation of previously learned features (to avoid catastrophic forgetting) that have been seen multiple times at the data stream, while forgetting outliers. To the extent of our knowledge, the UPL problem has not been addressed before. The closest prior work is CURL ("Continual Unsupervised Representation Learning") (Rao et al., 2019). CURL however does not consider the single-pass, online learning requirement. We further discuss this difference with CURL in Section 6.
2 STAM ARCHITECTURE
In the following, we describe the STAM architecture as a sequence of its major components: hierarchy of increasing receptive fields, online clustering (centroid learning), novelty detection, and a dual-memory hierarchy that stores prototypical features rather than specific examples. The notation is summarized for convenience in the Supplementary Material (SM) (section SM-A).
I. Hierarchy of increasing receptive fields: An input vector X, E R" (an image in all subsequent examples) is analyzed through hierarchy of A layers. Instead of neurons or hidden-laye units, each layer consists of STAM units in its simplest form STAM unit functions an online clustering module. Each STAM unit processes one Pt X patch (e.g. 8 x 8 subvector) of the input at the I'th layer. The patches are overlapping, with a small stride (set one pixel in our experiments) to accomplish translation invariance (similar to CNNs). The patch dimension Pt increases in higher layers- - the idea is that the first layer learns the smallest and most elementary features while the top layer learns the largest and most complex features.
II. Centroid Learning: Every patch of each layer is clustered, in an online manner, to a set of centroids. These time-varying centroids form the features that the STAM architecture gradually learns at that layer. All STAM units of layer share the same set of centroids (t) at time - again for translation invariance.' Given the m 'th input patch X1,m at layer 1, the nearest centroid of Ct selected for X1. is
Cıj arg min d(x),m: c) eEC,
where d(x),m m c) is the Euclidean distance between the patch x1, and centroid c.2 The selected centroid is updated based on a learning rate parameter a, as follows:
CI2 = a a + +1 - a)eyj 0<a<1
(2)
higher a value makes the learning process faster but less predictable. A centroid is only updated by at most one patch and the update is not performed if patch is considered "novel" (defined in the next paragraph). We do not use a decreasing value of a because the goal is to keep learning in a non-stationary environment rather than convergence to stable centroid.
III. Novelty detection: When an input patch XI, at layer / is significantly different than all centroids at that layer (i.c., its distance to the nearest centroid is a statistical outlier), new centroid is created
We drop the time index t from this point on but is still implied that the centroids are dynamically learned over time.
We have also experimented with the L1 metric with only minimal differences. Different distance metrics may be more appropriate for other types of data.
2

---

policy) due the limited capacity Under review as a conference paper at ICLR 2021
in Cj based X1. We refer to this event Novelty Detection (ND). This function is necessary so that the architecture can learn novel features when the data distribution changes.
To do estimate in an online manner the distance distribution between input patches and their nearest centroid (separately for each layer). The novelty detection threshold at layer is denoted by Dj and it is defined as the 95-th percentile (3 = 0.95) of this distance distribution.
IV. Dual-memory organization: New centroids are stored temporarily in Short-Term Memory (STM) of limited capacity A, separately for each layer. Every time centroid is selected as the nearest neighbor of an input patch, it is updated based on (2). If an STM centroid Cr is selected more than times, is copied to the Long-Term Memory (LTM) for that layer. We refer to this event as memory consolidation. The LTM has (practically) unlimited capacity and the learning rate is much smaller (in our experiments the LTM learning rate is set zero).
time
(Lieited capecity)
consoliartion
LTM cortex
nuaber of controsds incresscs 2,3 time
roughly constast
centroids
Figure A hypothetical pool of STM and LTM centroids visu- alized at seven time instants. From la to las centroid is moved from STM to LTM after it has been selected 0 times. At time ta, unlabeled examples from classes '2' and '3' first appear, triggering novelty detection and new centroids are created in STM. These centroids are moved into LTM by fg- From b to fg. the pool of LTM centroids remains the same hecause no new classes are seen. The pool of STM centroids keeps changing when receive "out- lier" inputs of previously seen classes. Those centroids are later replaced Least-Recently-eee of the STM pool.
This memory organization is inspired by the Complementary Learning Sys- tems framework (Kumaran et al., 2016), where the STM role is played by the hippocampus and the LTM role by the cortex. This (ual-memory scheme is necessary to distinguish be- tween infrequently seen patterns that can be forgotten ("outliers"), and new patterns that are frequently seen after they first appear ("novelty").
When the STM pool of centroids at layer is full, the introduction of a new centroid (created through novelty detection) causes the removal of an earlier centroid. We use the deast-Recently Used (LRU) policy to remove atypical centroids that have not been recently selected by any input. Figure 1 illustrates this dual-memory organization.
V. Initialization: We initialize the pool of STM centroids at each layer using randomly sampled patches from the first few images of the unlabeled stream. The initial value of the novelty- threshold is calculated based on the distance distribution between each of these initial STM centroids and its nearest centroid.
CLUSTERING USING STAM
We can use the STAM features in unsupervised tasks, such offline clustering. For each patch of input x, we compute the nearest LTM centroid. The set of all such centroids, across all patches of x, denoted by (x)). Given two inputs and y, their pairwise distance is the Jaccard distance of ((x) and (y)) Then, given set of inputs that need to be clustered, and a target number of clusters, we apply spectral clustering algorithm the pairwise distances between the of inputs. We could also use other clustering algorithms, as long as they do not require Euclidean distances.
4 CLASSIFICATION USING STAM
Given small amount of labeled data, STAM representations also be evaluated with classification tasks. We emphasize that the labeled data is not used for representation learning it is only used to associate previously learned features with given set of classes.
3

---

Under review as a conference paper at ICLR 2021
I. Associating centroids with classes: Suppose given some labeled examples X (t) from set of classes L(t) at time t. We can use these labeled examples to associate existing LTM centroids at time t (learnd strictly from unlabeled data) with the set of classes in (tt).
Given a labeled example of class k, suppose that there is a patch x in that example for which the nearest centroid is That patch contributes the following association between centroid and class k:
(x.c(k) = -ixx,e))DD
where Dr is a normalization constant (calculated as the average distance between input patches and centroids). The class-association vector ge between centroid с and any class is computed aggregating all such associations, across all labeled examples in XL:
Exex((M) fxx(kk)
ge(k) EvERO Exex((M) fx,o(k))
k= 1=1...
(4)
where (k) refers to labeled examples belonging to class k. Note that En (o(k)=11
IL Class informative centroids: If a cen- troid is associated with only one class k (A) 1), only labeled examples of that class select that centroid. At the other ex- treme, if a centroid equally likely to be selected by examples of any labeled class, (A) R (t)), D. the selection of that centroid does not provide any significant information for the class of the correspond- ing input. We identify the centroids that are Class INformative (CIN) as those that are associated with at least one class sig- nificantly more than expected by chance. Specifically, a centroid is CIN
Votes
G
G
G
Lager
Layer
LIgeT 3
(Saepled)
An example of the classification process. Every patch (at any layer) that selects a CIN centroid votes for the single class that has the highest association with. These patch votes are first averaged al each layer. final inference the class with the highest cumulative vote across all layers.
Figure 2:
1
kEL()) 9e (k)
+y
(5)
[l(t]
where 1/|](t)| is the chance term and y is
the significance term.
III. Classification using hierarchy of centroids: At test time, we given input of class kxx) and infer its class k(x). The classification task is a "biased voting" process in which every patch of x, at any layer, votes for single class as long as that patch selects CIN centroid.
Specifically, if patch X1. of layer I selects CIN centroid c, then that patch votes 1/.m max ((t) (k) for the class & that has the highest association with c, and zero for all other classes. If c not a CIN centroid, the vote of that patch is zero for all classes.
The vote of layer for class k is the average vote across all patches in layer / (as illustrated in Figure 2):
(k) ΣmEM, Ul.
(6)
[M
where Mj is the set of patches in layer 7. The final inference for input is the class with the highest cumulative vote across all layers:
k(x)) arg max
0y (k)
1=1
EVALUATION
To evaluate the STAM architecture the UPL context, we consider a data stream in which small groups of classes appear in successive phases, referred as Incremental UPL. New classes
4

---

Under review as a conference paper at ICLR 2021
introduced two at time in each phase, and they are only seen in that phase. STAM must be able to both recognize new classes when they are first seen in the stream, and to also remember all previously learned classes without catastrophic forgetting. Another evaluation scenario is Uniform UPL, where all classes appear with equal probability throughout the stream - the results for Uniform UPL. are shown in SM-G. We include results on four datasets: MNIST (Lecun al., 1998), EMNIST (balanced split with 47 classes) (Cohen et al., 2017). SVHN (Netzer et al., 20 and CIFAR-10 (Krizhevsky et al., 2014). For each dataset we utilize the standard training and test splits. We preprocess the images by applying per- per-patch normalization (instead of image normalization), and SVHN is converted to grayseale. More information about preprocessing can be found in SM-H.
We create the training stream by randomly selecting. with equal probability, Np data examples from the classes seen during each phase. Np is set to 10000, 10000, 2000, and 10000 for MNIST, SVHN, EMNIST, and CIFAR-10 10 respectively. More information about the impact of the stream size can be found in SM-E. In each task, we average results over three different unlabeled data streams. During testing, select 100 random examples of each class from the test dataset. This process is repeated five times for each training stream (i.c., a total of fifteen results per experiment). The following plots show mean + std-dev.
For all datasets, we use a 3-layer STAM hierarchy. In the clustering task, we form the set & f(x) considering only Layer-3 patches of the input In the classification task, we select a small portion of the training dataset as the labeled examples that are available only to the classifier. The hyperparameter values tabulated in SM-A. The robustness of the results with respect these values is examined in SM-F.
Baseline Methods: We evaluate the STAM architecture comparing its performance to state- of-the-art baselines for continual learning: GEM and MAS. We emphasize that there are no prior approaches which are directly applicable to UPL. However, we have taken reasonable steps adapt these two baselines in the UPL setting. Please see SM-B for additional details about our adaptation of GEM and MAS.
Gradient Episodic Memories (GEM) is recent supervised continual learing model that expects known task boundaries (Lopez-Paz. & Ranzato, 2017). To turn GEM into an unsupervised model, we combined it with self supervised method for rotation prediction (Gidaris et al., 2018). Additionally, we allow GEM to know the boundary between successive phases in the data stream. This makes the comparison with STAM somehow unfair, because STAM does not have access to this information. The results show however that STAM performs better even without knowing the temporal boundaries of successive phases.
Memory Aware Synapse (MAS) is another supervised continual learning model that expects known task boundaries (Aljundi et al., 2018). As in GEM, combined MAS with rotation prediction self-supervised task, and provided the model with information about the start of each new phase in the data stream.
To satisfy the stream requirement of UPL, the number of training epochs for both GEM and MAS is set to one. Deep learning methods become weaker in this streaming scenario because they cannot train iteratively over several epochs on the same dataset. For all baselines, the classification task is performed using a K- 1 Nearest- -eiihhbr (KNN) classifier we have experimented with various values of K and other single-pass pass classifiers, and report only the best performing results here. We have also compared the memory requirement of STAM (storing centroids at STM and LTM) with the memory requirement of the two baselines. The results of that comparison appear in SM-C.
The results for the clustering task are given in Figure 3. Given that we have the same number of test vectors per class we utilize the purity measure for clustering accuracy. In MNIST, STAM performs consistently better than the two other models, and its accuracy stays almost constant throughout the stream, only dropping slightly in the final phase. In SVHN, STAM performs better than both deep learning baselines with the gap being much smaller in the final phase. In 10 and EMNIST, on the other hand, we similar performance between all three models. Again, emphasize that STAM is not provided task boundary information while the baselines are and is still able to perform better, significantly in some cases.
Clustering Task:
5

---

Under review as a conference paper at ICLR 2021
INIST, Incremental
SWAN, Incremental
phases, naw classes per phase
phases, new classes par phose
Cluster Accuracy 00)
.00
-20
Accurscy
3T0R
STM
-2*
180000
23300
20.00
40.00
5000
3000
1300
2000
40003
Uniabled Zunges Seen
Unlebbled Iuages Sean
CTFAR-10, Tneresental
FINIST, Incresenta]
phases, new classes par phase
23 phases, now classes per phase
Clucker accuracy (1)
Cluster Accuracy
18000
22,000
20:30
50000
20003
30,00
40000
unlabled Images Sees
Unlatbled Iuages Seen
Figure 3: Clustering accuracy for MNIST (left), SVHN (left-center), CIFAR-10 (right-center), and EMNIST (right). The task is expanding clustering for incremental UPL. The number of clusters is equal 2 times the number of classes in the data stream seen up to that point time.
MNIST, Increnental, 10 labeled examples P.c.
SWHN, Incresental, 100 labeled exasples p.c. 5 phases, new classes per phase
phases, 2 new classes per phase
ACCoracy (2)
2003
CIFAR-10, Increrental, 168 labeled exarples p.c.
DNIST Incrasental, 18 labeled exaeples 23 phases, 2 new classes per phase
phases, 2 now classes per phase
ACCUPICY (3)
Zhore
Trages
Images
Figure Classification accuracy for MNIST (left), SVHN (center), CIFAR- 10 (right- and EMNIST (right). The task is expanding classification for incremental UPL, i.c., recognize all classes seen so far. Note that the number of labeled examples is 10 per class for MNIST and EMNIST and 100 per class for SVHN and CIFAR-10.
Classification Task: We focus on an expanding classification task, meaning that in each phase we need to classify all classes seen so far. The results for the classification task are given in Figure 4. Note that we use only 10 labeled examples per class for MNIST and EMNIST, and 100 examples per class for SVHN and CI. We emphasize that the two baselines, GEM and MAS, have access to the temporal boundaries between successive phases, while STAM does not.
As we introduce new classes in the stream, the average accuracy per phase decreases for all methods in each dataset. This is expected, as the task gets more difficult after each phase. In MNIST, STAM performs consistently better than GEM and MAS, and STAM is less vulnerable to catastrophic forgetting. For SVHN, the trend is similar after the first phase but the difference between STAM and both baselines is smaller. With CIFAR-10, we observe that all models including STAM perform rather poorly- probably due to the low resolution of these images. STAM is still able to maintain comparable accuracy to the baselines with a smaller memory footprint. Finally, in EMNIST, we see a consistently higher accuracy with STAM compared to the two baselines. We would like to emphasize that these baselines are allowed extra information in the form of known tasks boundaries (a label that marks when the class distribution is changing) and STAM is still performs better both on all datasets.

---

Under review as a conference paper at ICLR 2021
As we introduce new classes to the incremental UPL stream, the architecture recognizes previously learned classes without any major degradation in classification accuracy (left column). The average accuracy per phase is decreasing, which is due to the increasingly difficult expanding classification task. For EMNIST, we only show the average accuracy because there are 47 total classes. In all datasets, we observe that layer-2 and layer-3 (corresponding the largest two receptive fields) contain the highest fraction of CIN centroids (center column). The ability to recognize new classes is perhaps best visualized in the LTM centroid count (right column). During each phase the LTM count stabilizes until a sharp spike occurs at the start of the next phase when new classes are introduced. This reinforces the claim that the LTM pool of centroids (i) is stable when there are no new classes, and (ii) is able to recognize new classes via novelty detection when they appear.
A closer look at Incremental UPL:
In the CIFAR-10 experiment, the initial spike of centroids learned is sharp, followed by a gradual and weak increase in the subsequent phases. The per-class accuracy results show that STAM effectively forgets certain classes in subsequent phases (such as classes 2 and 3), suggesting that there is room for improvement in the novelty detection algorithm because the number of created LTM centroids was not sufficiently high.
In the EMNIST experiment, as the number of classes increases towards 47, we gradually see fewer "spikes" in the LTM centroids for the lower receptive fields, which is expected given the repetition of patterns at that small patch size. However, the highly CIN layers 2 and 3 continue to recognize new classes and create centroids, even when the last few classes introduced.
mist, Iecrenental, 10 Labeled P.C. proses,
MEST, Ircremented, Labsled exempães P.c.
IEST, Iscremated, somples no classes
P.C.
phane
AIDLPNY 00
Jac
incresental, Labiee (hases, Classes per place
iscresental, Labble exapples plasess an Closses per plate
(ncresenta), He Labelad ecmples ploses, dr classes per plase
2000
1488
utlatted
C23M8-28 Iecranental, 100 lapaled exapple P.C.
CITAB-18, Iecranental, 100 Labeted esanples P.c.
CIRAR-10 Zecranental, 100 Labeled exapples p.c.
LV8
Jast
Uslai0 THEE Scan
Uslodnn DI Sco
bdada Inso Soa
RMIST,
p.r.
23 plasee, no clacces per place
33 places, aew clasees per place
31 plazes, 2 new clacoss per place
ACCEPC 59
laper
Figure 5: STAM Incremental UPL evaluation for MNIST (row-1), SVHN (row-2), EMNIST (row-3) and CIFAR-10 (row-4). Per-class and average classification accuracy (left); fraction of CIN centroids over time (center); number of LTM centroids over time (right). The task is expanding classification, i.e., recognize all classes seen so far.
Several STAM ablations are presented in Figure 6. On the left, we remove the LTM capability and only use STM centroids for classification. During the first two phases, there is little (if any) difference in classification accuracy. However, we a clear dropoff during phases 3-5. This suggests that, without the LTM mechanisms, features from classes that are no longer seen in the stream are forgotten over time, and STAM can only successfully classify classes that have been
Ablation studies:
7

---

Under review as a conference paper at ICLR 2021
LDN Abjatzon now classes
LTW Updates Ablation
Rolation comples P-
SMN, Incromental comples 9-5.
50N, Incromental LIt, Jobeled comples 9-5. phases, classes
SMA,, Incromental int,
classes
static
byres:
Ztages
Stages
Figure 6: Ablation study: A STAM architecture without LTM (left). STAM architecture in which the LTM centroids are adjusted with the same learning rate a as in STM (center), and a STAM architecture with removal of layers (right)
recently seen. We also investigate the importance of having static LTM centroids rather than dynamic centroids (Fig. 6-middle). Specifically, we replace the static LTM with dynamic LTM in which the centroids are adjusted with the same learning rate parameter a, as in STM. The accuracy suffers drastically because the introduction of new classes "takes over" LTM centroids of previously learned classes, after the latter are removed from the stream. Similar to the removal of LTM, we do not see the effects of "forgetting" until phases 3-5. Note that the degradation due to a dynamic LTM less severe than that from removing LTM completely.
Finally, we look at the effects of removing layers from the STAM hierarchy (Fig. 6-right). We see small drop in accuracy after removing layer 3, and a large drop in accuracy after also removing layer 2. The importance of having a deeper hierarchy would be more pronounced in datasets with higher-resolution images or videos, potentially showing multiple objects in the same frame. In such cases, CIN centroids can appear at any layer, starting from the lowest to the highest.
6 RELATED WORK
The UPL problem has some similarities with several recent approaches in the machine learning literature but is also different in important aspects we describe in this section. Each paragraph highlights the most relevant prior work and explains how it is different from UPI..
Continual learning: In addition to CL. models cited in the introduction, other supervised CL methods include eegulariattionbassee approaches (Aljundi et al., 2018; Golkar et al., 2019; Hayes & Kanan, 2019; Kirkpatrick et al., 2017: Yoon et al., 2018; Zenke et al., 2017), expanding ar- chitectures (Lomonaco & Maltoni, 2017: Maltoni & Lomonaco, 2019; Rusu et al., 2016), and distillation based methods (Lee et al., 2019; 2020; Li & Hoiem, 2017). Their main difference with UPL and STAM is that they are designed for supervised learning, and it is not clear how to adapt them for non-stationary and unlabeled data streams.
II. Offline unsupervised learning: Additional offline representation learning methods include clustering (Caron et al., 2018; Jiang et al., 201 Xie et al., 2016; Yang et al., 2016), generative models (Eslami et al., 2016; Jiang et al., 2017b; Kosiorek et al., 2018: 2019), information theory (Hjelm et al., 2019; Ji et al., 2019), among others. These methods require prior information about the number of classes present in given dataset (to set the number of cluster centroids or class outputs) and iterative training (i.c. data replay), and therefore cannot be directly applied in the UPL setting.
III. Semi-supervised learning (SSL): SSL methods require labeled data during the representation learning stage and so they are not compatible with UPL (Kingma et al., 2014; Lee, 2013; Miyato et al., 2018: Oliver et al., 2018; Rasmus et al., 2015: Springenberg. 2015; Tarvainen & Valpola, 2017).
IV. Few-shot learning (FSL) and Meta-learning: These methods recognize object classes not seen in the training set with only a single (or handful) of labeled examples (Fei-Fei et al., 2006; Finn et al., 2017; Ren et al., 2018; Snell et al., 2017). Similar to SSL., FSL methods require labeled data to learn representations and therefore are not applicable the UPI. context. Centroid networks (Huang et al., 2019) do not require labeled examples at inference time but require labeled examples for training. Few-Shot Incremental Learning (Ayub & Wagner, 2020) includes a similar evaluation protocol to our work, but includes pre-training and learns centroids using data labels.
V. Multi- Learning (MTL): Any MTL method that involves separate heads for different tasks is not compatible with UPL because task boundaries are not known priori in UPL (Ruder, 2017). MTL methods that require pre- pre-training on a large labeled dataset are also not applicable to UPL (Pan & Yang, 2010; Yosinski et al., 2014).
8

---

Under review as a conference paper at ICLR 2021
VL Online and Progressive Learning: Many earlier methods learn in online manner, meaning that data is processed in fixed batches and discarded afterwards. This includes progressive learn- ing (Venkatesan & Er, 2016) and streaming with limited supervision (Chiotellis al., 2018; Li et al., 2018; Loo & Marsono, 2015), both of which require labeled data in the training stream. (Caccia al., 2019) is another very recent work which focuses online continual compression using discrete auto encoder and self-replay.
VII. Continual Unsupervised Representation Learning (CURL): Similar the UPL problem, CURL (Rao et al., 2019) focuses on continual unsupervised learning from non-stationary data with unknown task boundaries. Like STAM, CURL also includes mechanism trigger dynamic capacity expansion as the data distribution changes. However, a major difference is that CURL is not streaming method it processes each training example multiple times. We have experimented with CURL but we found that its performance collapses in the UPL. setting due to mostly two reasons: the single-pass through the data requirement of UPI., and the fact that we can have more than one new classes per phase. For these reasons, choose not to compare STAM with CURL because such comparison would not be fair for the latter.
VIII. Data dimensionality and clustering mustering-based representation learning: As mentioned earlier in this section, clustering has been used successfully in the past for offline representation learning (e.g., (Coates et al., 2011; Coates & Ng, 2012)). Its effectiveness, however, gradually drops the input dimensionality increases (Beyer et al., 1999; Hinneburg et al., 2000). In the STAM architecture, avoid this issue by clustering smaller subvectors (patches) of the input data. If those subvectors are still of high dimensionality, another approach is to reduce the intrinsic dimensionality of the input data at each layer by reconstructing that input using representations (selected centroids) from the previous layer.
VIII. Related work to other STAM components: STAM relies on online clustering. This algorithm can be implemented with rather simple recurrent neural network of excitatory and inhibitory spiking neurons, as shown recently (Pehlevan et al.. 2017). The novelty detection component of STAM is related to the problem of anomaly detection in streaming data (Dasgupta et al., 2018) and the simple algorithm currently in STAM can be replaced with more sophisticated methods (e-g.. (Cui et al., 2016; Yong et al., 2012)). Finally, brain-inspired dual-memory systems have been proposed before for memory consolidation (e.g., (Kemker & Kanan, 2018; Parisi et al., 2018; 8; Shin et al., 2017)).
7 DISCUSSION
The STAM architecture aims to address the following desiderata that is often associated with Lifelong Learning (Parisi et al., 2019):
I. Online learning: STAMs update the learned features with every observed example. There is separate training stage for specific tasks, and inference can be performed in parallel with learning.
II. Transfer learning: The features learned by the STAM architecture in earlier phases can be also encountered in the data of future tasks (forward transfer). Additionally, new centroids committed to LTM can also be closer data of earlier tasks (backward transfer).
II.. Resistance to catastrophic forgetting: The STM-LTM memory hierarchy of the STAM archi- tecture mitigates catastrophic forgetting by committing to "permanent storage" (LTM) features that have been often seen in the data during any time period of the training period.
IV. Expanding learning capacity: The unlimited capacity of LTM allows the system to gradually learn more features as it encounters new classes and tasks. The relatively small size of STM, on the other hand, forces the system forget features that have not been recalled frequently enough after their creation.
V. No direct access to previous experience: STAM only needs to store data centroids in a hierarchy of increasing receptive fields there is no need to store previous exemplars or to learn generative model that can produce such examples.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 11809-1860,, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pp. 11816 11825, 2019b.
Ali Ayub and Alan R Wagner. seniitively-isspiiee model for incremental learning using few ex- amples. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition Workshops, pp. 222-223, 2020.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint 1r0vv:1407.7906, 2014.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., (5(8):1798-1888 August 2013. ISSN 0162-8828. doi: 10.1 10.1109/PAMM..013.00
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5050-5060, 2019.
Kevin S. Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When "nearest neighbor" meaningful? In Proceedings of the 7th International Conference on Database Theory, ICDT '99 pp. 217-235, London, UK, UK, 1999. Springer-Verlag. ISBN 3-540-65452-6.
Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned continual compression with adaptive quantization modules. arXiv, pp. arXiv-1911, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In The European Conference on Computer Vision (ECCV), September 2018
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations, 2019a.
Arslan Chaudhry, Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic 2019b.
memories. arXiv preprint
Ioannis Chiotellis, Franziska Zimmermann, Daniel Cremers, and Rudolph Triebel. Incremental semi-supervised learning from streams for object classification. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5743-5749. IEEE, 2018.
Adam Coates and Andrew Y Ng. Learning feature representations with k-means, pp. 561-580. Springer, 2012.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223, 2011.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters. ArXiv, abs/ ftss/1702.05373. 2017.
Yuwei Cui, Subutai Ahmad, and Jeff Hawkins. Continuous online sequence learning with an unsupervised neural network model. Neural Comput., 28(11): 2504, November 2016. ISSN 0899-7667. doi: 10.1162/NECO. 00893.
10

---

Under review as a conference paper at ICLR 2021
Sanjoy Dasgupta, Timothy C Shechan, Charles F Stevens, and Saket Navlakha. A neural data structure for novelty detection. Proceedings of the National Academy of Sciences, 115(51):13093-13333 13098, 2018.
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1422-1430, 2015.
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Proceedings of the 30th International Conference on Neural Informa- fion Processing Systems, NIPS' 16, pp. 3233-3241, USA, 2016. Curran Associates Inc. ISBN 978-1-5108-3899
Li Fei-Fei, Fergus, and P. Perona. One-shot learning of object categories. IEEE Transactions on Puttern Analysis and Machine Intelligence, 28(4):594-611, April 2006. ISSN 0162-8828. doi: 10.1 10/1009/....206.67.
Chelsea Finn, Pieter Abbeel, and Sergey Levine, Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 1126-1135, JMLR.org, 2017.
Alexander Gepperth and Cem Karaoguz. Incremental learning with self-organizing maps. 2017 12th International Workshop Self-Organizing Maps and Learning Vector Quantization, Clustering and Data Visualization (WSOM), pp. 1-8,201 2017.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Robert L Goldstone. Perceptual learning. Annual review of psychology, 49(1):585 612, 1998.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv:1903.044: 2019.
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting gradient-based neural networks. arXiv preprint arXiv: 1312.6211, 2013.
Tyler L Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. arXiv preprint arXiv:1909.01520, 2019.
Tyler L Hayes, Nathan Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9769-9776. IEEE, 2019.
Alexander Hinneburg, Charu C. Aggarwal, and Daniel A. Keim. What is the nearest neighbor in high dimensional spaces? In Proceedings of the 26th International Conference on Very Large Data Bases, VLDB '00, pp. 506-515, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1-55860-715-3.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In /CLR 2019. ICLR, April 2019.
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. In NewrIPS Continual learning Workshop, 2018.
Gabriel Huang, Hugo Larochelle, and Simon Lacoste-Julien. Centroid networks for few-shot clustering and unsupervised few-shot classification. arXiv preprint arXiv:1902.000 08605, 2019.
X. Ji, J. Henriques, and A. Vedaldi. Invariant infromation clustering for unsupervised image classi- fication and segmentation. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
11

---

Under review as a conference paper at ICLR 2021
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, LICAI'17, pp. 1965-1972. AAAI Press, 2017a. ISBN 9780999241 103.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, LJCAI' 17, pp. 1965-1972. AAAI Press, 2017b. ISBN 178-0-9992211-0-3.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. International Conference on Learning Representations (ICLR), 2018.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks. AAAI Conference on Artificial Intelligence, 2018.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes. arXiv preprint
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. Proceedings of the 27th International Conference on Neural Information Processing Systems Volume 2, NIPS 14, pp. 3581 1-3589, Cambridge, MA, USA, 2014. MIT Press.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
et al. Overcoming
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In S. Bengio, H. Wallach, H. Larochelle, Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 8606-8616. Curran Associates, Inc., 2018
Adam Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoencoders. In Advances in Neural Information Processing Systems, pp. 15486-15496, 2019.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 10 dataset. online: http://www. cs.
toronto. edu/kriz/cifar, hmm, 55, 2014.
Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20(7): 512-534, 2016.
Chia- Wen Kuo, Chih-Yaa Yao Ma, Jia-Bin Huang, and Zsolt Kira. Manifold graph with learned prototypes for semi-supervised image classification. arXiv preprini arXiv: 1906.05222, 2019.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2324 Nov 1998. ISSN 0018-9219. doi: 10.1109/
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 312-321, 2019.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for fask-free continual learning. arXiv preprint arXiv:2001.00689 2020.
Yanchao Li, Yongli Wang, Qi Liu, Cheng Bi, Xiaohui Jiang, and Shurong Sun. Incremental semi- supervised learning on streaming data. Pattern Recognition, 88, 11 2018. doi: 111100666.ppaogg 2018.1 11.06.
12

---

Under review as a conference paper at ICLR 2021
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(11):2935-22 0et1)997972277,222 2017.
Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost de Weijer. Generative feature replay for class-incremental learning. arXiv preprint (rXxv:0004.09199 2020.
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. arXiv preprint arXiv: 1705v5170,.03500, 03550, 2017.
H. R. Loo and M. N. Marsono. Online data stream classification with incremental semi-supervised learning. In Proceedings of the Second ACM IKDD Conference on Data Sciences, CoDS '15, pp. 132-133, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450334365. doi: 10.1 14.11457332888..752266
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17. pp. 6779, USA, 2017. Curran Associates Inc. ISBN 978 1-18-----0--
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single-ineremeenal--ssk scenarios. Neural Networks, 1 11::56-73, 73, 2019.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep superviee learning algorithms. Advances in Neural Information Processing Systems, pp. 5235-3246, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: arXiv:/807.033: /807.03777. 2018
S.J. Pan and Q Yang. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):335-1135, 2010.
German I Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter. Lifelong learning of spatiotemporal representations with dual-memory recurrent self-organization. Frontiers in neurorobotics, 12:78, 2018.
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. Neural Networks, 113:54 71, 2019. ISSN 0893-6080. doi: https://doi.org/ 10. 10t.://0.1....emuet200......
Cengiz Pehlevan, Alexander Genkin, and Dmitri B Chklovskii. clustering neural network model of insect olfaction. In 2017 51st Asilomar Conference on Signals, Systems, and Computers, pp. 593-600. IEEE, 2017.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Tch, and Raia Hadsell. Continual unsupervised representation learning. In Advances in Neural Information Processing Systems 32, pp- 7655-7655. Curran Associates, Inc., 2019.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 3546-3554 Curran Associates, Inc., 2015.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR' 17, pp. 5533-5542, 2017.
13

---

Under review as a conference paper at ICLR 2021
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi supervipeeesee few-shot classification. In Proceedings of 6th International Conference on Learning Representations ICLR. 2018.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05095, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 140647160..04671, 2016.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2990-2999. Curran Associates, Inc., 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying superiupeee learning with consistency and confidence. arXiv preprint arXivv2000.076655 2020.
Jost Tobias Springenberg. Unsupervised and semi-superrisee learning with categorical generative adversarial networks. arXiv preprint arXiv: 1.6990, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1 1195-1204. Curran Associates, Inc., 2017.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv: 18/2.05065 2018.
Gido M van de Ven and Andreas S Toliaa. Three scenarios for continual learning. arXiv preprint arXiv: 2019.
Rajasekar Venkatesan and Meng Joo Er. A novel progressive learning technique for multi-class classification. Neurocompul, 207(C))100-31,, September 2016. ISSN 0925-2312. doi: 10. neucom.2016.05.000
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 487, New York, New York, USA, 20-22 Jun 2016. PMLR.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5147-5156, 2016.
Suet-Peng Yong, Jeremiah D. Deng, and Martin K. Purvis. Novelty detection in wildlife scenes through semantic context modelling. Pattern Recogn., (000)3339-34510 September 2012. ISSN 0031-3203. doi: 111016/j.patcog 2at.:dg..attooo222222....00 2012.02.006.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference Learning Representations, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3320-3328. Curran Associates, Inc., 2014.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
14

---

Under review as a conference paper at ICLR 2021
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint 1rxvv11033.00122 2018.
Guanyu Zhou, Kihyuk Sohn, and Honglak Lee, Online incremental feature learning with denoising autoencoders. In Neil D. Lawrence and Mark Girolami (eds.), Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 1453-1461, La Palma, Canary Islands, 21-23 Apr 2012. PMLR.
15


Reviews:
[{"review_id": "vH3iNWnPl1", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an \"Unsupervised progressive learning\" (UPL) problem, where a model is exposed to data in an non-iid manner, and each training example is presented once. Simple to continual learning, but a little more explicit in the connections to the way biological agents learn. They present a model that uses clustering and long-term memory (buffered) and compare on a few UPL tasks with additional supervision signal (classification) or unsupervised (clustering).\n\nThe paper is interesting. I appreciate the straightforward outline of the problem and connection to biological learning and some of the motivations for the model (e.g., long term buffered memory of centroids).\n\nThe model itself though isn't very clear in a few regards that I think are rather important.\n\n1) for the architecture itself, are the cluster centroids propagated to the next layer, or is the same input used at the next layer? If so, I'm not sure what is hierarchal here, it's just ordered in terms of \"patch dimension\".\n\n2) How large does the LTM get during training? Is there a maximum size? I didn't see any restrictions on the size of the LTM, so it seems this buffer could get more and more precise and eventually resemble just a normal buffer over remembered examples (with some small permutations depending on alpha.\n\n3) During supervision the model has access to the number of classes?\n\nOther questions:\nInstead of a separate test-set, one could evaluate online, collecting the classification accuracies across learning before the learning step in the centroids, since each example is seem exactly once.\n\nMy main concerns though are on the size of the LTM and if this is really doing anything other than KNN with K=1 over a buffered memory of examples. I need to have some evidence that the LTM is doing something *general* w.r.t. those cluster centroids.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tS8DS50belD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n\nThe paper introduces a non-parametric approach, STAM, for unsupervised progressive learning (UPL), a variant of continual unsupervised learning with a single-stream requirement. STAM is developed for visual tasks. It comprises several components: (1) online clustering of hierarchical visual features (2) novelty detection (3) dual-memory for prototypical features. Experiments show STAM performs better than GEM, MAS in specific scenarios. \n\n##########################################################################\n\nReasons for score: \n\nCurrently, I vote for rejecting the paper. The paper is well written, easy to follow. My main concerns are (1) to what extent UPL is a real and practical problem, compared with unsupervised continual learning; (2) the comparisons with GEM, and MAS do not look fair; (3) each technical component is not new.\n\n##########################################################################\n\nPros: \n\nThe paper is well written, with enough details (and code) to reproduce the results;\n\nThe proposed method is technically sound;\n\nDiscussions on related work are comprehensive;\n\nThe experiments are thorough. Ablations are performed to justify the various design considerations.\n\n##########################################################################\n\nCons:\n\nI’m not sure if UPL is a real and practical problem. To me, the single-stream requirement looks somewhat unnecessary and is unfair to all parametric models (including deeply learned models), especially when the number of training iterations is small (e.g. on epoch on MNIST). On the other hand, as STAM relies on prototypical features from online clustering, essentially no parameters need to be learned. Thus, the comparisons between STAM and MAS/GEM presented in the paper are unfair as I believe the MAS/GEM models were highly undertrained in this case. What if STAM/MAS/GEM are all trained for two epochs, or trained on a larger dataset for one epoch? I guess the performance of MAS/GEM will be improved but I’m not sure if the performance of STAM can be improved.\n \nSTAM is only evaluated on small datasets (e.g. MNIST, SVHN, CIFAR10). I’m not sure if it is expressive enough to model larger datasets (e.g. CIFAR 100) with a reasonable memory footprint (e.g. comparable with a deeply learned model)\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ouJ1rdNtUeg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an approach (architecture + algorithms) to unsupervised progressive learning in a non-stationary environment (the number of classes grows gradually) by keeping centroids at several hierarchies, using a combination of techniques from online clustering, via computing and updating centroids, with novelty detection, and dropping (forgetting those deemed outliers).  A variety of experiments are performed on several image datasets (MNIST, EMNIST, SVHN, CIFAR-10) with comparisons to other adapted methods. They evaluate performance in a supervised setting where they describe how they learn centroid to label(s) mappings.\n\n\n\nThe experiments are somewhat promising,  and I liked the hierarchical aspect of the centroids.\n\nBut I have concerns about\napplicability of the approach to practice: The approach has some of\nthe shortcomings of nearest neighbors: efficiency specially at\nclassification/test time (as the number of classes goes up to many\n1000s, and required number of centroids per class goes up), choice of\ndistance, and so on.     What do you do if too many centroids?  space and\nspeed of finding closest centroids to a patch?  How is fast nearest\nneighbor done?  Also, different classes require different complexities\n(in terms of distance to use, manifold learning), and one parameter\nfor novelty detection and another for when to add a centroid to LTM,\nmay not be sufficient...\n\nRegarding evaluation: Why the particular elaborate learning/classification scheme chosen (that computes\n a mapping from centroids to labels)?    the research or\nmotivation behind it? Why not a plain classifier, such as a simple nn,  svm, etc... given the features (centroids)\navailable? Perhaps because there is no explicit feature representation.    A quick discussion would be good.\n\n\nThe paper was overall clearly written, and the supplements provide much useful detail on the experiments.\n\nSome  detailed comments:\n\nsection I, hierarchy: ruo_l x ruo_l, the subscript 'l' is not\nexplained  before introduced (later as layer). It would be good to\nquickly give examples of the patch dimensions.\n\nsection 2.II:\n\nwhat is t in C_l(t) (in footnote, they say they drop time index, so t\nis probably time)\n\nDoes the online update take place only if (patch is) not deemed\noutlier? The text seems to imply that, but it's not clear.  How do you\n(re)use centroids that have been moved to long-term memory? It's not\nclear from the text but it appears all centroids are used to compute\nthe nearest centroid (both in LTM and STM, ie C_l is the union).\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T8BWROQ0q-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, it is proposed to have a new problem setting of \"unsupervised feature learning\", which as claimed, is supposed to be different from standard continual learning settings. In this proposed setting, a data point can be seen only once by a model for training. I think, this is too strict a condition, as it is reasonable for a model to keep a data point in memory for a short while so as to use it for training across multiple epochs. In the experimental setup, neural baselines are trained only for a \"single\" epoch, in consideration of the proposed problem setting, which doesn't make sense for all practical purposes as neural models need a decent number of epochs for training. Furthermore, the final classifier used after learning the unsupervised representations is k-NN which is again unrealistic in the context of continual learning literature, especially in the context of neural baselines. All of this is justified for accommodating the  proposed problem setting which is itself not well motivated. \n\nThe proposed model for the introduced problem setting is more of an engineering approach, relying upon some basic techniques such as clustering, novelty detection. There is no clear motivation for the learning algorithm, it is not clear how the model is optimized wholistically. It is more of a heuristic driven approach. The model is claimed to be brain-inspired; for instance there is a component in the proposed model which has a \"hierarchy of increasing receptive fields\", which is nothing but CNN-like neural net.  \n\nFor experiments, I have the following suggestions.\n(1) In Fig. 3, x-label should be changed to something more appropriate, number of new classes introduced.\n(2) Results are good for MNIST dataset primarily.\n(3) Any other evaluation metrics besides accuracy? Accuracy can be misleading for multi-class scenario. \n(4) Report accuracies separately for new classes introduced and the old classes.\n(5) It needs to be clarified exactly what \"class boundary information\" the baseline methods have access to.\n(6) Only 10 new classes introduced from the 5 phases, what about the datasets with a large number of classes?", "extra_scores": {}, "extra_text": {}}, {"review_id": "HkLUa67gVEF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose and formulate a new problem setting UPL which addresses a number of limitations of current methods. I think this problem formulation is very important and relevant and the authors provide a fairly solid foundation about how to address this problem. \n\nThere are a number of strengths to this work. I found the non-deep learning approach they took refreshing and their experimentation demonstrates the effectiveness of it in this setting. I found the evaluation relatively convincing. More specifically, I find the classification and clustering results good. Again, the classification results are difficult to interpret as you do use the labels (albeit not for learning the representation), but it becomes more like semi-supervised learning at this point. On the other hand, the clustering results do not have this problem and are more impressive. I appreciate the code being included in the supplementary material.\n\nI have a number of questions/concerns about the work.\n\t1) How effective is this hierarchical receptive field processing approach? My feelings are that it may be insufficient for larger/more complex data. Evaluation involving these would be much welcomed. Further, how does it perform for non-image data types?\n\t2) The focus on classification is a little out of place, I can see why it's useful, but as a main section of the paper, preceding the purely unsupervised setting, it can be confusing to readers. It would be clearer in my opinion if the focus was on the unsupervised setup, and then the semi-supervised setup as an additional evaluation (i.e., not preceding the purely unsupervised approach as it does now). This is a criticism of the structure of the paper (and perhaps the main message), more than the work itself. \n\t3) It seems like a limitation of this approach is that clustering algorithms that can be used on the representation are limited to discrete distance measures. Can the authors speculate on potential future limitations of this for the setting?\n\t4) \"the small batch size required by UPL\" I do not follow this - can you further explain?\n\t5) Further, while it may be unfair on CURL to compare directly, but as it is a very similar approach,  I do think it would be interesting to see it how it performs, while acknowledging the difficulties and reasons why. If not a full set of experiments, \"We have experimented with CURL but we found that its performance collapses in the UPL setting\" leaves me wondering what exactly the collapse is.\n\nOverall, I think this is worthy of acceptance, acknowledging the fact  that it is early work in this new area.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the reviews, rebuttal, and looking through the paper I do feel that UPL setting is one that we need to consider. However is not clear to me that proposed approach matches the conditions described by the authors. In particular the scalability constraints seem important. I do feel that for the UPL setting makes sense particularly in the large case scenario of many examples and classes and how the system behaves under strict computational budgets for learning and inference. And I'm wondering whether in that limit parametric models would actually becomes relevant again, and whether there is a "burn-in" that one has to pay to use parametric models. 
That said I don't think current approaches (CURL, AGEM etc.) will do well even in that setting, partially because they were not necessarily thought for that. 
So in summary, I find the problem interesting, probably more so than the solution and particularly in an large scale setting.

However I think for the paper to have the impact it needs, and be ready for acceptance it needs a bit more. I think looking at a larger scale setting, and relying on that to motivate the problem will considerably help with its impact.  Also is not clear to me how the proposed solution scales (non-parametric approaches don't always do well in large scale settings), which I think is needed for it to be convincing. 
Decision: Reject