Continual Learning with Delayed Feedback
Authors: THEIVENDIRAM PRANAVAN
TERENCE SIM
Venue:  (2020)

Abstract: Most of the artificial neural networks are using the benefit of labeled datasets whereas in human brain, the learning is often unsupervised. The feedback or a label for a given input or a sensory stimuli is not often available instantly. After some time when brain gets the feedback, it updates its knowledge. That's how brain learns. Moreover, there is no training or testing phase. Human learns continually. This work proposes a model-agnostic continual learning framework which can be used with neural networks as well as decision trees to incorporate continual learning. Specifically, this work investigates how delayed feedback can be handled. In addition, a way to update the Machine Learning models with unlabeled data is proposed. Promising results are received from the experiments done on neural networks and decision trees. 

Under review as a conference paper at ICLR 2020
CONTINUAL LEARNING WITH DELAYED FEEDBACK
Anonymous authors
Paper under double-blind review
ABSTRACT
Most of the artificial neural networks are using the benefit of labeled datasets whereas in human brain, the learning is often unsupervised. The feedback or label for a given input or a sensory stimuli is not often available instantly. Af- ter some time when brain gets the feedback, updates its knowledge. That's how brain learns. Moreover, there is no training or testing phase. Human learns contin- ually. This work proposes can be used with neural networks well as decision trees to incorporate continual learning. Specifically, this work investigates how delayed feedback can han- dled. In addition, way to update the Machine Learning models with unlabeled data is proposed. Promising results are received from the experiments done on neural networks and decision trees.
-agnosic continual learning framework which
INTRODUCTION
The high representational capacity of neural networks as well as the discovery of good training methods, and the access to fast processing units resulted in super human accuracy in various artificial intelligent tasks. Supervised image classification results has improved to super human levels by various convolutional neural netN) architectures such AlexNet (Krizhevsky al. 2012).
VGG(Simonyan & Zisserman 2014), Dotgleeet(Szzegeee et al.) 2015), and ResNe(He et al. 2016 Most of the CNNs are learned with supervision. The limitation of labeled dataset opens the path for unsupervised learning. Moreover, human brain often does continual learning in an unsupervised manner. As an example, take an infant. does not know most of the things in the environment. But it sees everything. Whenever somebody gives some detail or a label about the object that the baby has seen, updates its knowledge. This work names this detail or label as delayed feedback. Hence, the key difference in CNNs is that we feed everything with labels at the start which does not occur in human brain. In addition, humans learn continually. The distinction training and testing is not realized in actual learning of human.
Figure I explains a delayed feedback scenario. Suppose an infant sees different cars. But it does not know anything about the After sometime, somebody shows Now the infant learns that the previous unknowns vehicles should be cars. In this case, the baby does not know any car before seeing these objects. This scenario can be extended different types or shapes of cars as well where baby knows about one set of cars and how it can learn other different cars.
car and labels it car.
Unsupervised learning is discussed predominantly in categories. One is reconstructing the original data using auto-encoders and the latent representations are used for the downstream tasks(Hinton let al. [20IT). Especially this resulted in good dimensionality reduction techniques such as t-SNE (Maaten Hinton| 2008). The second one instead of reconstructing the given data, the model learns a contrastive task which can produce latent representations which are useful for other AI tasks (Hjelm et al.) 2018), (Schneider et al., 2019). Both categories can find better representations to capture the variability of the data in the input space. In either ways, the task specific information extraction is not possible due to learning the same data or a contrastive task. The area of updating a supervised model parameters using unlabeled data is not explored as per our knowledge. In this work, an idea is proposed to update the model to improve the accuracy over some new data. Mainly, two contributions are presented in this paper.

---

Under review as a conference paper at ICLR 2020
Somebody gives feedback with to the learner.
Car
Figure Delayed Feedback
1. A way handle delayed feedback is presented and implemented by maintaining a queue to store some recent data of samples. This concept introduces the same delayed feedback of actual brain artificial neural networks.
2. Instead of having training and testing phase, continual learning model is proposed which can update the model even with unlabeled data.
RELATED WORK
2.1 CONTINUAL LEARNING
Neuroscience literature has widely spoken about continual learning ((assaaiis al. 2017) Parisi et al. (2019)). However, the literature on continual learning in artificial neural network is not studied deeply and still at the inception level. Continual learning is mainly discussed via resolving the issue of Catastrophic Forgetting (Kirkpptrick et al. 2017)). Catastrophic forgetting occurs when neural network forgets most part of the previously learnt task while learning a new task. The distributional shift in inputs is not clearly modeled using neural networks. Hence, neural networks perform badly with the old tasks. Superposition of models into single neural network (CCeuun et al. (019)) and neural network pruning (GGolkar et al. 2019) are proposed to do continual learning.
2.2 DELAYED FEEDBACK
Delayed Feedback an important phenomenon human learning. Brain encounters different ob- jects routinely without knowing the actual identifier of what it is. When some stimulus gives the label, brain updates the knowledge. If there no signal for certain period of time, then the object seen will be erased from memory. This mechanism is named as delayed feedback. Speech and vi- sual delayed feedback is discussed previously ((Lee) 1950), (Smith 1962)). But the effect delayed feedback is vastly unexplored in artificial neural networks.
Data hungry supervised algorithms achieved state of the art results in popular AI tasks such as image classification, recognition, and segmentation, but the availability of labeled dataset is always limited than the abundant unlabeled data in the wild. The availability of abundant unlabeled dataset and delayed feedback can jointly build strong continuous learning models.
2

---

Under review as a conference paper at ICLR 2020
(x,
Bounded Queue
X
Queue full
NO
Get next
Update the model
Figure 2: Growing Phase
CONTINUAL LEARNING FRAMEWORK WHICH HANDLES DELAYED FEEDBACK
3.1
BASIC MODEL.
This work introduces 2 terms seeding and growing instead of training and testing. Seeding the phase where the initial learning occurs. This is the phase where the model takes advantage and learns from labeled data. Growing the phase where the model learns information from new unlabeled data and updates the model. The learnt neural network of classification problem can be seen function which produces confidence values(f(X) = Y). Initial function f is improved to f in the conventional training of neural network, same thing happens here with seeding, But the function is fixed in conventional neural networks in testing phase where as in growing phase of the continual learning framework the model parameters are updated with the framework rules. Hence mostly a new function(s") will be seen after every new inputs.
The figure Minuueellltttteee illustrates how the frame work works in growing phase. When new sample(X,) comes in(for an image classification problem, X; is an image), the model will predict confidence value for the input. Confidence value is the probability vector(Y,) for each sample(In neural network, Y; is the output from softmax layer). The input(X;) and the confidence value vector(Y,) are inserted to bounded queue. If the queue is not full, the sample will be processed and appended to the queue. When the queue is full, an element from queue is removed based on the following criteria. In addition, one hot encoding is randomly created give the feedback (Zo).
Queue searched for confidence value(Y,) which has the lowest distributional distance or the highest similarity with the one hot encoding( (o)). Bhattacharyya coefficient is used to measure the similarity between two distributions.
2. Now the best Y1 is selected match the given feedback. has corresponding image (Xi). Now the model is updated with the sample (X1, Zo). The element (X1, Yi) is removed from the queue.
3. Assume the learning rate used for seeding
The learning rate used in growing phase is r which is around 10 times smaller than r. When we update the model with sample (X1, Zo), we use higher learning rate(r x where > 1) if the bhattacharyya coefficient between Zo and Y, is in [0.5, E] (Here 1). If Ymazz
the model is updated with small learning rate (r' (r'/s where 1). 0.8 used in most of the experiments.

---

Under review as a conference paper at ICLR 2020
distance (Zo, K2
Distance((ref: X1) ())
r'/s
T'S
Update Scheme
No update
No update
Figure 3: Update rule with CBIR Features
4. If there samples with bhattacharrya coefficient 2 0.5, then the oldest element in the queue is removed (dequeue operation in the queue) without any update to the model.
The reason behind selecting a lower learning rate for high confidence vector is that the model is confident about the decision. Hence, updating with higher learning rate has higher probability of overfitting or fitting wrong sample with high confidence. If the confidence values are in [0.5, e]. then the model is not highly confident, but fair enough say that these decisions will be correct. Hence, we are updating these samples with higher learning rate. Having good initial model from seeding leads to improvement accuracy. But starting with bad model with respect to accuracy may result in models which are worse than initial model. Take the same infant example, if the infant taught wrong things from the start, then it will make more decisions which are wrong. Similar thing can happen here as well.
Figure illusttates the update rules. If the distance in the input space is higher (dref and X1), then no update is done. If the distance in the input space is lower, then the same update rule explained in section 3.1 is applied to the model. The update the model restricted by the input space good margin.
similarity. This update rule increased the accuracy
In the experiments, because the probability of getting inaccurate model increases with r's which greater than
5 is used update the model. Selecting higher r' than is not desirable
Seeding phase contains most of the updates to the model. It is seeded with several epochs whereas growing phase deals with single update for each sample. If the initial learnt function from seeding is the growing phase functions will fall in a small manifold around g.
Continual learning is different from online learning. In online learning, samples come with labels and the update is done instantly. But in continual learning, instant update is not done. Instead of updating the samples are kept in queue. When the queue full, updates are done as described in this section. The learning is almost fixed in online learning at least for certain period of time whereas in continual learning, it depends on the sample. Infact, machine learning models which have online version are easy to do continual learning since the updates can be done partially for the given samples.
3.2 EXTENSION WITH CONTENT BASED IMAGE RETRIEVAL(CBIR) FEATURES
The feedback is just given as label, Zg. In addition to the label, reference image (Ire)) for la- bel Zo is passed. Reference images can be collected from initial samples used for seeding. When it comes to calculate similarity, the distributional similarity is calculated between Zn, Y, More-

---

Under review as a conference paper at ICLR 2020
over the queue contains the images correspond all labels. Content based similarity is calculated between /re/ and X1. The model update decision done as shown in figure Content Based Image Retrieval Systems use various features such color, shape and texture. In this work, color histograms are used to measure the similarity between Tref and X1.
3.3 EXTENSION WITH BATCHES
Instead of searching for a single element the queue, searching for multiple elements and update the model together resulted in better model than single sample update model described in section 3.1 In an image classification problem with 10 classes, feedback Zo will be sampled with probability 1 1/10 for class. Suppose if we have samples which closer match for generated feedback Zo. then the model can be updated together. Indeed, it resulted in models with better accuracy and reduction in growing time. Not doing batch update may keep good samples longer in the queue.
EXPERIMENTS AND RESULTS
4.1 CONVOLUTIONAL NEURAL NETWORKS(CNNS) BASED EXPERIMENTS
4.1.1 EXPERIMENT SET UP
small CNN with around 66000 parameters and convolutional layers used for all the evalua- tions. The CNN used for traditional neural network functionality is noted T-CNN and the CNN used for continual learning is noted as CL-CNN. But the structure of the network same ei- ther cases. CIFAR DEFAR1O(Kriz) 1r/Kiiheyskky al. 2009) dataset is used for evaluation. During seeding phase CNN is trained over 200 epochs. Stochastic Gradient Descent is used for updating the weights of CNNs. Learning rate
0.0001 is used in seeding. CNNs are trained/seeded over different initial weights which were generated from seeding different random state values in pytorch (Paszke) al. 2017). The average accuracy 8 random states is 63. 63%19% for T-CNN. Using CNN with more parameters, a higher accuracy is achievable. The smaller network is selected to see the impact of delayed feedback.
4.1.2 T-CNN AND BASIC CL-CNN RESULTS
Table illustrates the results obtained using various queue sizes over different initial random states to initialize neural network weights. The second row in table shows the T-CNN accuracy. Other rows illustrates continual learning with delayed feedback accuracy(CL-CN) accuracy) with various queue sizes. All the accuracies are reported in percentage(%).
Table 1: Accuracy against various queue Sizes and random states
Queue Size
2
3
5
6
7
8
Mean
62.79
65.04
61.96
63.42
63.62
62.25
62.98
63.43
63.19
100
61.36
62.39
60.38
61.06
61.25
59.26
60.02
62.01
60.97
200
61.84
62.47
61.13
62.24
62.18
61.02
60.94
61.91
61.72
500
62.92
64.45
62.67
63.75
62.88
62.17
62.47
63.57
63.11
1000
64.96
66. 16
63.40
65.25
66.06
64.20
64.24
65.62
64.99
2000
68.52
70.27
67.96
69.52
69.65
68.01
68.24
70.08
69.03
The accuracy of CL-CNN is steadily increasing with queue size. Having a bigger memory to hold unknowns will help to make good decisions. With 10, 000 samples for growing in CIFAR10 increment the queue size is stopped at 2, 000. When queue sizes are 1000, 2000, the accuracy is above T-CNN accuracy.
5

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy without final queue of samples final accuracy
Queue Size (A)
Accuracy for samples (10000- A)
Delayed Feedback accuracy(10000)
100
61.27
60.97
200
62.40
61.72
500
64.91
1000
68.70
64.99
2000
77.39
69.03
The table shows an interesting result of CL-CNN. The results are averaged over random states which are used to initialize network weights. The final queue contains the hardest examples for CL- CNN. For instance, the queue size 100, then 9900 samples are processed first. Finally, the last content of the queue is processed. Infact, the samples which get high confidences will get higher chance to leave the queue. These queues can be used to assist clustering the samples into harder and simpler samples for the given model.
4.1.3 COLOR FEATURES WITH DELAYED FEEDBACK
Table shows the results when the feedback given with reference image as discussed in section
3.2 This reference image is picked from the images used for seeding. For each class 100 reference images are selected. For every feedback a random image out of the relevant class is picked. Using the reference image, the color similarity is calculated and the updates done based on figure 3 It quite evident that giving more information with feedback resulted in increase in accuracy with all different queue sizes of CL-CNN. The final accuracy has increased from 69.03% to 73.26%. The initial layers CNN function as modules which can detect color and basic shapes. But adding color T-CNN with larger margin 10% well as basic CL-CNN without color features with a margin of 4%. Color features in feedback play a different role in improving accuracy rather than initial layers of CNN which detect colors.
as feedback in basic CL-CNN improves
Table 3: Accuracy with color features accuracy without color features
Queue Size
Accuracy with color features (%)
Accuracy without Color (%)
100
02.73
60.97
200
63.62
61.72
500
65.56
63.11
1000
68.34
64.99
2000
73.26
69.03
4.1.4 BATCH UPDATE WITH DELAYED FEEDBACK
Table shows the results with batch update. Adding batch update to the feedback with color decreases the accuracy over CL-CNN with color feedback. But it ended up in a faster training time(nearly 3 time faster for the batch size of 4). A batch of is used to update. Sometimes getting meet the updating rule is not possible, in those scenarios model is updated with a smaller batch or even no update. Rather than a random feedback, color feedback with batch update improves bit as shown in the table.
4 samples which

---

Under review as a conference paper at ICLR 2020
Table 4: Accuracy with color feedback and batch update vs accuracy of basic CL-CNN
Queue Size
Accuracy with color and batch update (%)
Basic Delayed Feedback (7)
100
61.92
60.97
200
62.60
61.72
500
63.98
63.11
1000
65.67
64.99
2000
69. .3
69.03
4.2 DECISION TREE BASED EXPERIMENTS
The framework is deployed in decision trees. Small changes done to the framework. Instead of maintaining a queue, DT's are updated instantly. Unlike neural network, partial update to the model for sample is not possible. Hence, for each new sample, the whole dataset is fitted. In addition, if the confidence values are higher, no update is done. The model is updated if the confidence value is in [0.5.e].
Digits data in scikit learn (Pedregosa et al. 2011) is used to train various models. contains 1797 digit images(0-9) of size x 8). Various sizes of decision tree depth are checked. Higher depths are not used here to avoid overfitting in the seeding phase. T-DT denotes a Traditional Decision Tree and CL-DT denotes Continually learnt Decision Tree.
SOURCY
430-
103-
300-
(a) DT Depth 4
(b) DT Depth
0003
443
1..5
170
(c) DT Depth 6
(d) DT Depth- - 7
(c) DT Depth
(f) DT Depth 12
Figure 4: Accuracy vs Seeding-Learning Split
Test corresponds to the test accuracy of T-DT model whereas learning corresponds the growing accuracy of a CL-DT. Figure 4 illustrates that when DT depth is small, the model performs better than T-DT. Depth 4 and 5 of CL-DT produce better results than a T-DT in most of the different evaluating portion of the data. With increase in depth, the difference in accuracy is not significant. The reason is that model ovefits the data in training and seeding phases. This leads towards 100% seeding and training phase accuracy. Hence, for the new data sample, the DT gives high confident values for the decisions. As per the update rule, CL-DT is updated if confident values are not that strong(Y is the confidence vector such that Vy E Y.y & (0.8, 1)). Hence, in the case high depth
7

---

Under review as a conference paper at ICLR 2020
CL-DTs, actually update is really happening per our rule (The reason is that update the model the confidence values / probability values fall in a range [0.5, 0.8). If the confidence values are above 0.8, then the number of updates would go towards 0 with the height of CL-DT.). Hence, both CL-DT and T-DT accuracies are around the same.
The number of samples is small. Instead of having bigger queue, the updates done the model instantly (technically there is no queue. But this can be changed if the number of samples are higher. However, the update of the model will take longer time train). Similar to neural network, online update is not possible with sci-kit learn DT models (D3, C4.5, C5.0 and CART). The updates are done for the whole sample set.
CONCLUSION AND FUTURE WORKS
-odel-agnostic continual learning framework is proposed and deployed in CNNs and DTs. The results show improvement in classification accuracy using delayed feedback system. The frame- work works well with the machine learning models which can be learnt online. CL.-CNN is clear example where images can be learnt continually. However, continual learning differs from online learning in the way of updating weights and learning rates. The nature of DT and SVM does not match the framework since everything is updated together in these models(Batch Learning). Similar to DT similar framework version can be used to SVM, but the models have the limitations of scaling if the dataset used to learn blows up. Buffering recent data in a queue gives the model chance to select an easier sample to learn. Increase in queue size resulted better accuracy which says that having larger buffer data helps.
There are few limitations with this framework. Switching testing data of CIFAR 10 for seeding and training data for growing data resulted in accuracies which are around 55%. CL-CNN improved it by mere 1% with the queue size of 5000. There is a diminishing return with the improvement of accuracy when growing data surpassed the seeding data by bigger factor. This is limitation with this framework.
Continual Learning framework discussed in this work can be used with supervised learning. The difference from T-CNN(with Stochastic Gradient Descen((SGD)) is that the learning rate varies with the confidence values whereas in T-CNN(with SGD) the learning rate is reduced over number of epochs and usually fixed during epoch. Similar to ADAM optimizer ((Kingma & Ba, 2014)), proposed CL framework can be used as an adaptive learning technique where the learning rate is function of confidence values.
Delayed Feedback given with a label and a reference image where one an output element and other one is the input element. The update is not done if the similarity is not high. There are chances where conventional color histograms, shapes and textures differ by a good margin, but still the objects belong to the same category. More sophisticated ways needed to update with intrinsic input properties. addition, instead of a label and a reference image, more auxiliary information such text be given feedback. Feedback is any form of information which can be used to do learning.
This work creates further directions in research areas of continual learning, and delayed feedback systems.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. arXiv preprint UtXiv://02.055300 2019.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint 4rXiv:1903.04476, 2019.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245-258, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton, Alex Krizhevsky, and Sida Wang. Transforming auto-encoders. Interna- lional Conference on Artificial Neural Networks, pp. 44-51. Springer, 2011.
R Devon Hjelm, Alex Fedorov, Samuel Aavoic-Marchildoon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:/808. 1808,06670, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980. 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Bernard S Lee. Effects of delayed speech feedback. The Journal of the Acoustical Society of America, 2246))224-826, 1950.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-266) 2008
Genman Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv: 1994.09444.... 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 14X9v:1409.1556 2014.
Karl U Smith. Delayed sensory feedback and behavior. 1962.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.


Reviews:
[{"review_id": "SklhMPn0Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper claims to tackle a semi-supervised continual learning problem where the feedback or the labeled data is delayed and is provided based on the model performance. Authors do not provide standard benchmarks for comparison and no baseline is considered.\n\nThe idea of using unlabeled data for continual learning is interesting and to the best of my knowledge this is the first work that suggests using delayed feedback for continual learning but unfortunately they do not consider measuring forgetting and this work seems an online learning method using delayed feedback.\n\nI vote for rejecting this paper due to the following reasons. I have listed the issues in chronological order and not their importance.\n\n1- I start with the writing. The paper, in its current form, needs to be thoroughly proofread and reorganized. The text does not read well and is vague in most parts (for example section 3 and 4). The text is informal in some parts (ex. in Figure 1). There are also grammar errors and typos for which I have found passing my writing through the free version of Grammarly very helpful in getting rid of most such errors. \n\n2- As one of the main motivations for the paper, authors claim humans learn continuously in an unsupervised fashion (paragraph one). I disagree with this statement because we all have been constantly learning from the feedback we have been receiving the environment throughout our lives. For example we all have learned how to walk by falling on the ground multiple times and using the pain signal in our muscles as a negative feedback to correct our movements. Getting corrected while speaking or question answering in our dialogues are examples of receiving feedback from the environment letting our learning behavior receiving lots of supervision.\n\n3- The related work section misses significant number of prior work on continual learning (I have provided a short list at the end [4,5,6] but authors are strongly encouraged to read more on this literature). However, my biggest concern is that I this work should not be introduced as a continual learning algorithm. The proposed method is an online learning method with delayed feedback which has been extensively studied before. Authors should consider citing the pioneering work in this field such as Weinberger & Ordentlich (2002) [2] or Joulani et al from ICML 2013 [3]. Providing comparison to [3] is strongly encouraged. Also note that the citation for “catastrophic forgetting” is wrong and should be corrected to McCloskey & Cohen (1989). \n\n4- The figures and tables do not meet the conventional scientific standards and have to be significantly improved.\n\n5- Authors use softmax probabilities as a confidence score which are known to be uncalibrated by large as deep models are usually overconfident about their predictions. (see [1] for example). Was this investigated at all? Using a calibration technique might be able to help with this [1].\n\n6- On page 4, paragraph 5, the authors claim that “in continual learning instant update is not done”. This is vague to me and I think it is not true because there are plenty of supervised continual learning approaches where the labeled data is available when a task is learned (for example [4,5,6])\n\n7- The experimental setting is not well designed and does not use a standard continual learning setting and there is no baseline included which are very important reasons for rejecting this paper. Authors can benefit from applying their method on standard benchmark datasets commonly used in the literature to provide a fair comparison. Most importantly authors should evaluate their method against prior work. \n\n8- Exploring continual learning for decision trees is completely vague not justified in the paper.\n\n[1] Guo, Chuan, et al. \"On calibration of modern neural networks.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n[2] Joulani, Pooria, Andras Gyorgy, and Csaba Szepesvári. \"Online learning under delayed feedback.\" International Conference on Machine Learning. 2013.\n[3] Weinberger, Marcelo J., and Erik Ordentlich. \"On delayed prediction of individual sequences.\" IEEE Transactions on Information Theory 48.7 (2002): 1959-1976.\n[4] Kirkpatrick, James, et al. \"Overcoming catastrophic forgetting in neural networks.\" Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.\n[5]Lopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" Advances in Neural Information Processing Systems. 2017.\n[6] Serrà, J., Surís, D., Miron, M. & Karatzoglou, A.. (2018). Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:4548-4557", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hyx_QzKsFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method that draws inspiration from neuroscience and aims to handle delayed feedback in continual learning (ie. When labels are provided for images after a phase of unsupervised learning on the same classes). It is an interesting idea, and worth exploring.\n\nI found the paper quite hard to follow at times, and I suggest the authors go through the paper in detail to address some of the issues with grammar and clarity of explanation.\n\nIn addition, I think the paper is lacking some grounding and context in terms of what problem is being solved and what previous work exists.\nFor example, based on the experimental setup section it seems like the problem being addressed is that of unsupervised learning on CIFAR images followed by supervised learning on images with labels (IE. Delayed feedback) - is this the case? If so, this is much more a semi-supervised learning or fine-tuning problem than a continual learning problem (which typically looks at a single class at a time or some other non-stationary sequence of tasks). Either way, the recent literature in semi-supervised learning and continual learning should be referenced - see the citations below as a few examples, and consider referencing and more closely perusing some of the examples in the cited review paper by Parisi et al (2019).\n\nLastly, the experiments show the performance as a function of queue length for different features and for CNNs versus decision trees, but there is no comparison to existing methods and very simple models are used - this means that again, it's difficult to gauge the efficacy of the approach and place this in the context of prior art.\n\nUnfortunately, I think the paper in its current state does not meet the bar for ICLR - I suggest the authors consult the vast literature in semi-supervised and continual learning, and try to place their work in this context, along with external comparisons.\n\n\nNguyen, Cuong V., et al. \"Variational continual learning.\" arXiv preprint arXiv:1710.10628 (2017).\n\nLopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" Advances in Neural Information Processing Systems. 2017.\n\nMiyato, Takeru, et al. \"Virtual adversarial training: a regularization method for supervised and semi-supervised learning.\" IEEE transactions on pattern analysis and machine intelligence 41.8 (2018): 1979-1993.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BklCS-XNqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "General\n\nThe paper is quite hard to follow. The figures are very coarse and the problem definition is not given very well. The paper claims that it is about continual learning, but it does not give ANY experimental results on the continual learning benchmarks. The paper seems to be dealing with an online learning with unlabeled data. \n\nCon & Questions:\n\n- Fig 3 shows the update rule, but there is no explanation on I_ref or X_1. \n- The paper says it generates random one-hot vector when queue is full, but what does it have to do with delayed feedback?\nThe algorithm requires a completely trained model and a queue that needs to store large amount of data. Then, what is the good nature of this method?\nThere is no baseline in the experimental results. \nThe T-CNN result on Cifar-10 is too low. This makes the result dubious. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper claims to present a model-agnostic continual learning framework which uses a queue to work with delayed feedback. All reviewers agree that the paper is difficult to follow. I also have a difficult time reading the paper. 

In addition, all reviewers mentioned there is no baseline in the experiments, which makes it difficult to empirically analyze the strengths and weaknesses of the proposed model. R2 and R3 also have some concerns regarding the motivation and claim made in the paper, especially in relation to previous work in this area.

The authors did not respond to any of the concerns raised by the reviewers. It is very clear that the paper is not ready for publication at a venue such as ICLR at the current state, so I recommend rejecting the paper.
Decision: Reject