Unsupervised Progressive Learning and the STAM Architecture

Abstract: We  first pose  the  Unsupervised  Progressive  Learning  (UPL)  problem:   learning salient representations from a non-stationary stream of unlabeled data in which the number of object classes increases with time.  If some limited labeled data is also available, those representations can be associated with specific classes, thus enabling classification tasks.  To solve the UPL problem, we propose an architecture that involves an online clustering module, called Self-Taught Associative Memory (STAM). Layered hierarchies of STAM modules learn based on a combination of online clustering, novelty detection, forgetting outliers, and storing only prototypical representations rather than specific examples. The goal of this paper is to introduce the UPL problem, describe the STAM architecture, and evaluate the latter in the UPL context. 

Reviews and author responses:

The paper presents a new problem formulation in the broader area of lifelong learning.
Specifically, the Unsupervised Progressive Learning (UPL) problem, requires a learner to consume a stream of data, where each data point is associated with a class, but only very few labels are provided. Similar to continuous learning the learner can only access current and not access previous data in the stream. The paper clearly describes how this setup is different to commonly other learning setups studied, including continual learning. 

To solve this problem the paper proposes a deep-learning-free approach based on clustering and novelty detection.

Strength:
-	I think the problem formulation is very realistic and interesting and I am not aware of it being explored previously
-	The newly proposed architecture STAM is also interesting.
-	The paper evaluates STAM on UPL on three datasets, MNIST, EMNIST, SVHN
-	The paper ablates several aspects of the model evaluates the effect of hyper parameters.

Weaknesses:
1.	The paper misses to include baselines, both, w.r.t. the proposed method, as well as for the learning problem. 
1.1.	The paper argues that deep learning cannot work, I am wondering, why don’t include standard baselines, e.g. auto-encoder based with a prototype stored whenever a labeled example is seen. (For single pass continual learning, see e.g. [B], although that work is supervised)
1.2.	Hebbian learning inspired algorithms might similarly form a valuable baseline
2.	The UPL problem makes sense to me, however, the experimental setup could be improved. Specifically, the paper uses the entire dataset to find hyperparameters. This is highly concerning as this basically means the entire dataset has been seen multiple times before the final pass over the dataset (the one with the best hyperparameters) is performed. This is basically a contradiction to the setup of UPL, which assumes all data is seen only once.
3.	Related Work:
3.1.	[A] seems to study a similar setup and proposes an approach which should also be applicable in this work.







Conclusion:
The paper’s contribution is a new learning setup (UPL) and an approach (STAM). While it remains unclear if STAM can generalize to realistic images, the idea of UPL is very interesting and speaks for accepting the paper, although there are several weaknesses the authors should address.
It would also be great if the authors plan to release the exact experimental setup (i.e. which images are sampled in which order and which ones are labeled) so future work can compare to this work.



Reference:
[A] Continuous Online Sequence Learning with an Unsupervised Neural Network Model
Yuwei Cui, Subutai Ahmad and Jeff Hawkins; Neural Computation, 2016
[B] Gradient episodic memory for continual learning, D Lopez-Paz, MA Ranzato ; NeurIPS, 2017


==== Post rebuttal comment:
Thanks for the clarifications and additional experiments;
I do think the the paper should be accepted and increased my score.

Regarding 2) I was thinking more of using a small part of the dataset to do hyper parameter selection, see e.g. 
Efficient Lifelong Learning with A-GEM
A Chaudhry, MA Ranzato, M Rohrbach, M Elhoseiny
International Conference on Learning Representations (ICLR)




Thank you for this additional follow up comment. We commit to moving the memory requirement results/discussion from the appendix to the main paper in the final revision of the paper. 

We would like to clarify an important point. The memory requirement of STAM includes all parameters of this model (i.e. the centroids stored in both STM and LTM). STAM does not require stored images for replay. A deep neural net that uses replay to avoid catastrophic forgetting would need to store both its parameters and a number of images. For example, the deep learning baseline in Appendix F has a memory requirement of about 4,500 grayscale SVHN images just to store its parameters.

Many thanks for your additional experiments and comments. It is good to hear you will make this publicly available. Please do use any of my suggestions/ideas as you see fit, I will be interested to see future work.

Thank you for the intuition of Delta=500 corresponding to about 930 grayscale images in SVHN. I would actually like to see this point made in the main paper: I think it provides much-needed intuition to the memory cost of STAM. 930 grayscale images is quite a lot (and Delta=500), and this would be useful for readers to know. I do not view this to be a downside for this paper's acceptance, rather a clarification of potential negatives of STAM, which could of course be improved in the future. UPL is a hard problem and the STAM approach requires what seems like a lot of memory. This number (of 930) would be useful for other approaches, as and when they are developed, to compare to.

Thank you for the constructive comments.

1) Following your suggestion, we have also generated results using a convolutional autoencoder for feature learning, followed by a K-nearest-neighbor classifier. In the UPL context, where the unlabeled data should be seen only once, we train the autoencoder by setting the number of epochs and the batch size to one. So each unlabeled input in the data stream is used in the training process only once. The use of a KNN classifier means that we also use the labeled examples only once. The related results for SVHN appear in Appendix F (see Figures 9 and 10). Please note that STAM performs significantly better than this deep-learning baseline. We understand of course that this is only a simple baseline and we plan to develop a number of more sophisticated baselines in the near future. 

1.1) Thank you for sharing the work of [B]. We agree this is a similarly motivated work and we included that paper in the references. However, we think that their method is not applicable to UPL because they still need to know the tasks, even if the task schedule is not known. 

1.2) Regarding Hebbian learning: please note that the online clustering method that STAM is based on is effectively a Hebbian learning method: please see the reference by Pehlevan et al. 2017 in our paper. That work has shown how to do online k-means clustering based strictly on Hebbian learning. 

2) The reviewer is right that we have used the entire dataset to choose hyperparameters -- this was a mistake that we regret. However, based on our experience working with MNIST, EMNIST and SVHN, it is very unlikely that there would be any significant change if we had only used a portion of the dataset for hyperparameter tuning. Additionally, please note that with the exception of the parameter Delta, the rest of the hyperparameters do not cause a major effect on classification accuracy. In future work, we plan to choose hyperparameter values based on a different, but similar, dataset -- for example, to use examples from ImageNet to choose hyperparameters, and then applying that STAM architecture on Google’s Open Images.  

3) Thank you for sharing the work of [A]. It is relevant in that they do continuous online learning from streaming data -- we included that paper in the references. However, we do not think that their method is relevant enough to STAM that would warrant a direct comparison.

Additional comment: We have inserted an improved description of the experiment setup (see Section 3, Paragraph 3). 

We commit to release the STAM code on GitHub after publication.

Note: the changes discussed above are available in a recently uploaded revision. The red vertical lines on the right margin represent modifications from the original submission.

Thank you for this constructive review and suggestions. You gave us a lot to think about and, with your permission, we would like to investigate all these questions in follow-up work. Also. we commit to make the STAM code and evaluation framework publicly available on GitHub after publication.

We agree that the memory footprint of STAM is important and have now included some related calculations in Appendix G. We calculate (for the case of SVHN), how many pixels would be required -- and what is the equivalent number of SVHN images -- based on the number of learned centroids in both Short-Term Memory and Long-Term Memory.

Please note that in the submitted paper we did not optimize STAM in terms of memory efficiency. Following your suggestion, we performed a number of experiments in which we asked: how small can the memory footprint (controlled by the parameter Delta, which is the Short-Term Memory size) be without introducing a significant loss in accuracy. These results appear in Figures 8-d,e,f in Appendix E. Interestingly, we can reduce the memory footprint significantly (from Delta=2000 to 500 centroids) without a noticeable loss in accuracy. 

For Delta=500 centroids, the memory footprint of STAM for SVHN is equivalent to about 930 grayscale images. This is only 5% of the memory that would be required to store the parameters of the convolutional autoencoder that we use in Appendix-F. 

Regarding the idea of increasing both Delta and theta, unfortunately we did not have enough time to generate those results yet but we plan to do so next. 

We appreciate the suggestions of adopting progressive neural networks and including uncertainty estimates and robustness to adversarial examples. These are great directions to take this research forward!

Note: the changes discussed above are available in a recently uploaded revision. The red vertical lines on the right margin represent modifications from the original submission.

Thank you for the constructive comments. 

We agree that experiments with additional datasets are required and we are already in the process of doing them. We want to note however that it is common, at least in the unsupervised learning literature, that any new approaches are first studied using simpler datasets (e.g., see [1]) before experimenting with more complex datasets that include multiple objects in the same image or limited examples per class. 

Per the suggestions of the two other reviewers, we have included a comparison with a deep-learning baseline (see Appendix F) and a memory footprint comparison (see Appendix G). 

Furthermore, we have generated new results that show how STAM performs with only a limited number of LTM centroids (updated Appendix E). The results show that STAM performs equally well, in the case of SVHN, even when we reduce the number of LTM centroids from 2000 to 500. 

We would like to argue that the number of unlabeled training examples we use in STAM does indeed mimic animal learning. Animals receive a stream of sensory input (think of video instead of static images). Every object is seen by many angles, depths, perspectives, etc and, typically, for at least several seconds. 

We agree that the number of labeled examples should be limited, and this is why we consider only 10-100 such examples per class. In the future, we plan to reduce this range to 1-10 but include some degree of data augmentation for the labeled examples (e.g., looking at the same object from different angles and distances). 

Regarding your comment about the L2 distance metric and the nearest-neighbor problem in high-dimensional data: we are deeply aware of the theoretical challenges of the nearest-neighbor problem in high-dimensional data (and we cite, for instance, the 1999 paper by Beyer et al). But please note that we apply this metric in small patches of the images -- not at the level of entire images. We should mention that we have also experimented with the L1 metric, without seeing significant changes in the results.

Note: the changes discussed above are available in a recently uploaded revision. The red vertical lines on the right margin represent modifications from the original submission.

[1] A. R. Kosiorek, S. Sabour, Y. W. Teh, and G. E. Hinton. “Stacked Capsule Autoencoders”. In: CoRR. arXiv:1906.06818, 2019

This paper proposes a method called Self-Taught Associative Memory (STAM) for Unsupervised Progressive Learning (UPL) , i.e., learning salient representation from streams of mostly unlabeled data with occasional class labels, where the number of class increases over time. The motivation of this paper is quite interesting in that the authors try to mimic how animals learn. The surrounding environments of animals are considered to be unlabeled, and animals gradually learn to distinguish between objects without explicit information. The model shed light on the problem of catastrophic forgetting by introducing dual-memory organization. To be specific, Short-Term Memory contains a set of centroids associated with the unlabeled data, whereas Long-Term Memory stores the prototypical centroids, which are frequently seen patterns. In addition, the model utilizes novelty detection technique to introduce new centroids to each layer of the model, and it prepares the newly created centroids to be associated with new classes. 
Overall, the paper reads well and it is self-explanatory with clear notations and hyperparameters. Each step of the architecture is well-formulated mathematically and the necessity of the step in the model is explained clearly. The problem proposed, Unsupervised Progressive Learning (UPL) problem, is novel.
On the other hand, there are shortcomings of the paper, one being the model evaluation on dataset such as MNIST, SVHN, and EMNIST. The dimensional size of the dataset and the number of classes are small which is not convincing to evaluate the performance of the model. In addition, each layer of the model is independent from other layers (no connections between units at different layers), keeping the updated representation of patches to itself. This fact would hinder the model from understanding complex representation. Furthermore, there is a concern on the use of L2 distance metric for the similarity between the patch and patterns (centroids). L2 is not meaningful in high dimensional spaces.
This paper is a good start in tackling the Unsupervised Progressive Learning problem, but some weaknesses are present in the nature of the model architecture as mentioned above. However, the approach taken appears to be ad hoc. It is difficult to imagine the method can work for more complex situations. Even for the small datasets used in the paper, the learning require 10’s of thousands of training example. This does not mimic animal learning at all. Also, the accuracy of the model on EMNIST with 47 classes is around 65% which makes me doubt the application of the model on real world.
-	Shed some light on the problem of catastrophic forgetting and continual learning without supervision
-	The model architecture is well defined, but has some weaknesses in the methodology
-	Interesting motivation of trying to mimic how animals learn in an environment


This paper sets up a new problem based on a continuous stream of potentially partially labelled data, which the authors call the Unsupervised Progressive Learning problem. The paper also introduces a new model designed to approach this problem, called the STAM architecture, with many concepts applied in a novel way. The STAM architecture is tested on example problems from the UPL problem.

I find this paper to approach a problem that is not well studied in the literature, and it is well-written and easy to read. I am not aware of previous works that tackle this specific problem setup. Many previous works seem to slowly be converging to tackling this kind of problem, but this is the first work that directly tackles this realistic problem. It would be good if the authors made the benchmark (/how they are generated) public.

The STAM architecture is an interesting way of tackling image classification, and it is refreshing to see a technique not dependent on neural networks. The experiments in the paper are extremely detailed, with good figures and ablation studies.

I am recommending this paper be accepted. But I have one main question about this work: what is the memory cost of the STAM architecture? Many 1000s of LTM centroids are stored, what is the memory cost of this? Is this memory cost greater than the cost of just storing all the labelled inputs that have been seen? I would imagine that just replaying these stored labelled inputs (or just training a NN on these stored inputs) would provide extremely high accuracies for the datasets considered in this paper.

I understand that STAM has potential beyond just replaying memory for potentially larger datasets; the authors also make the point that they do not believe that the brain works by replaying memory. However, I would then like to see one (or both) of the following tests: a run with much fewer LTM centroids (with the corresponding memory cost detailed explicitly), and/or a dataset where it is clear that the memory cost incurred by the STAM architecture is a better use of memory than just storing previous data.

I would also argue that the brain *does* replay memory in some manner in order to learn. But that is a debate for another time!

By fixing the LTM centroids that are learnt on previous data, the method is essentially freezing previous knowledge and then learning new knowledge by increasing model capacity. It would be interesting (as future work / other papers) to see an adopted version of eg Progressive Neural Networks and how that compares on the same benchmarks.

I would also be interested to see, as future work, the uncertainty estimates of this new architecture and its robustness to adversarial examples.

Finally, a couple of minor points:
- Figure 4 is small and hard to read, particularly the left column.
- The authors find (in Appendix E) that increasing the \Delta hyperparameter leads to a growing number of LTM centroids. I wonder if that is still the case if \theta is also increased along with \Delta?
- Typos: last line of Section 3.2: 'lowst'; a few places where \citet{} and \citep{} are incorrect.

Meta-review: 


The paper presents a semi-supervised data streaming approach. The proposed architecture is made of a layer-wise k-means structure (more specifically a epsilon-means approach, where the epsilon is adaptively defined from the distortion percentile). Each layer is associated a scope (patch dimensions); each patch of the image is associated its nearest cluster center (or a new cluster is created if needed); new cluster centers are adjusted to fit the examples (Short Term Memory); clusters that have been visited sufficiently many time are frozen (Long Term Memory). Each cluster is associated a label distribution from the labelled examples. The label for each new image is obtained by a vote of the clusters and layers.

Some reviews raise some issues about the robustness of the approach, and its sensitivity w.r.t. hyper-parameters. Some claims ("the distribution associated to a class may change with time") are not experimentally confirmed; it seems that in such a case, the LTM size might grow along time; a forgetting mechanism would then be needed to enforce the tractability of classification. 

Some claims (the mechanism is related to how animal learn) are debatable, as noted by Rev#1; see hippocampal replay.

The area chair thinks that a main issue with the paper is that the Unsupervised Progressive Learning is considered to be a new setting ("none of the existing approaches in the literature are directly applicable to the UPL problem"), preventing the authors from comparing their results with baselines.

However, after a short bibliographic search, some related approaches exist under another name:
* Incremental Semi-supervised Learning on Streaming Data, Pattern Recognition 88, Li et al., 2018;
* Incremental Semi-Supervised Learning from Streams for Object Classification, Chiotellis et al., 2018;
* Online data stream classification with incremental semi-supervised learning, Loo et al., 2015.

The above approaches seem able to at least accommodate the Uniform UPL scenario. I therefore encourage the authors to consider some of the above as baselines and provide a comparative validation of STAM.

Decision: reject