Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In contrast to previous works on lifelong machine learning (LML) that put their focus on the supervised learning settings, this paper concentrates on the scenario that only a limited amount of data is available. The proposed method MAKO is mounted on the top of supervised LML model, without introducing additional knowledge based overhead, for better leveraging the unlabeled data. Labeling new data can be realized by using the data programming method which is supervised by the labeled data. The target of this paper is to design a SSL LML framework that minimizes the performance between using partially labeled data, and the upper-bound performance using fully labeled data. Several experiments on standard image classification data sets including MNIST, CIFAR-10 and CIFAR-100 are used to evaluate the the effectiveness of MAKO. 
main_review: Pros:

[1] The lifelong learning without catastrophic forgetting is a critical and interesting issue, the method proposed in this paper is shown to be effective in addressing this issue on some small-scale benchmarks, such as MINIST, CIFAR-10 and CIFAR-100. 

[2] The authors conducted comprehensive experiments on LML and proposed several evaluation metrics to quantitatively measure the ability of MAKO in eliminating catastrophic forgetting, improving peak per-task accuracy and final accuracy. 

[3] Exploring the lifelong learning using a limited amount of data, i.e. semi-supervised LML, is a new setting and of great interest. 

[4] The paper is well-organized. 

Cons:

[1] Although authors conducted comprehensive experiments on MNIST and CIFAR, the experiments on larger benchmarks may be necessary to show the improvements obtained through using MAKO. 

[2] Most of the components of MAKO are based on the prior arts proposed in previous literatures, for example, the method for labeling training data with weak supervision is based on Snuba, therefore, the technical contribution of this paper may be limited. 

[3] If the method is resistance to catastrophic forgetting, why the forgetting ratio is always lower than baseline method? Does it mean that there is less positive knowledge transfer from the later tasks to the earlier ones?

[4] In table 1, the more unlabeled data is used during the training process, the worse the final accuracy is, which is quite confusing. The final accuracy and forget ratio are both lower than the baseline method when 40 unlabeled instances are used. If the proposed method is truly effective and is able to avoid catastrophic forgetting, why the performance is lower?

This metric is less than 1 if the LML model loses its performance on the earlier tasks, and it is greater than 1 if there is positive knowledge transfer from the later tasks to the earlier ones.

summary_of_the_review: I like the idea of semi-supervised lifelong machine learning and I agree with the authors that previous works on LML only consider the fully-supervised lifelong learning, which require the use of all labeled data and is quite expensive. However, the technical contribution of this paper may be limited. Considering that the performance of the proposed method is not that satisfying either, my current rate of this paper is: "3: reject, not good enough".
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper propose a wrapper tool that mounts on top of supervised Lifelong Machine Learning (LML) frameworks, leveraging a well-known method data programming. The contributions of this paper can be summarized in three aspects. 1)  Adapting automatic label generation by semi-supervised learning/data programming to LML in some special scenario. 2) Implementing a LML wrapper that can accomplish some tasks under some restrictions. 3) Through detailed experimental results prove the superiority of its method.
main_review: Strength:
The definition of the problem, the applicable scenarios and the description of the methods are very clear in this paper. The scenario that the author want to solve where labeled training data is expensive to obtain in some lifelong machine learning tasks is also interesting.   Under the premise of this scenario, the author built a processing framework, completing a series of functions such as hyper-parameters search, pseudo labeling and so on. Experimental results also demonstrate its effectiveness. 

Weakness:

1）	I am confused about the definition of some methods in the article. For example, What does the “weak labeler” refer to in the article?  Does the definition of this part correspond to “weak augmentation” in the semi-supervised problem?. I don’t know what is the “weak label” in classification problem. I hope the author can give some description about this.

2）	The article spends a lot of time on the definition of various issues and contexts, It's not even possible for me to figure out where the work is the original work of this article. Even in Sec.4, I found that it seemed to be integrating some of the previous working methods to form a system framework, such as data programming, confidence calibration and so on. It makes me feel that this article is more like a technical report rather than a research work. May be I’m misunderstanding about it, So I hope the author can clearly state the contribution of this work.

3）	The paper proposed a framework based on semi-supervised continual learning, the main statement of the article also lies in the application of semi-supervised learning. However, some description about semi-supervised is hard to understand. For example, the most important problem in semi-supervised learning is how to obtain accurate pseudo labels, but I don’t see how the author solves this problem, only by using a previous method data programming? Or does the article just introduce the semi supervised learning into the some scenarios of continuous learning?  If it is the latter, there are many more mature semi supervised algorithms, can you explain data programming is used?

summary_of_the_review: This paper implemented a semi-supervised continual learning framework that can be mounted on top of any existing LML tool. It integrates some methods to solve the scenario where labeled training data is expensive. The problems and application scenarios proposed in this paper are of practical significance. However, in my opinion, the engineering significance of the paper is greater than its academic significance, it lacks of novelty in the view of research.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an interesting idea of using data programming techniques to enable continual semi-supervised learning with limited labeled data. It proposes a stage-wise pipeline where probabilistic pseudo labels are first produced by a Snuba based Data Programming framework, then calibrates them by the temperature scaling, and finally inputs into the mounted Lifelong Machine Learning (LML) tools. Experiments show that the proposed framework achieves similar performance to fully supervised methods.
main_review: #### Pros
Continual learning with fewer annotated data is an important research task. The proposed framework uses data programming techniques to generate pseudo labels for unlabeled data and makes the task suitable for various fully supervised LML methods. Compared with fully supervised methods, the experimental results seem promising.

#### Cons
1. Since most benefits come from the data programing techniques (Snuba in this paper), I would like to see more analysis on the annotation process, performance of down-stream tasks is only one of the metrics, e.g.,
	a. It is unclear that how many labeling functions are generated in each task? Precision and coverage of each LF in the committed set are more meaningful than labeling accuracy in Fig.3. 
	b. One key fact of labeling functions in Snuba is diversity. Although the authors trained labeling functions on different sampled sets, the laziness of the neural network makes it prefer to output similar results. How different are the results obtained from different labeling functions? It would be nice to see a numeric comparison.  
	c. It seems a sophisticated process to annotate unlabeled data. It would be better to provide/plot average iterations and computational times for [hyperparameter search <--> data programming] for each task. 

2. I’m concerned about the module in Sec 4.1 and Sec 4.2:
	a. I am not sure whether I grasped the main points of the automatic hyperparameter search and data programming. Does the automatic hyperparameter search module first search the best hyperparameter as a fixed heuristic pattern, then the DP module generates different LF with the same hyperparameters but different sampled subsets?	
	b. How many heuristics are generated for pruning. 
c. Some important technical details are not clear in this paper, e.g., the hyperparameters in Snuba.

#### Misc
1. Just for curiosity, why do both Mako-labeled DEN and fully-labeled DEN achieve worse performance than the baselines?
2. I have a doubt about the experimental results of DEN in the last row of Table 7 and Table 10: What’s the reason for the performance booming in the last task?

#### Minor points
“purposed” in the 1st line of the second paragraph in Sec.2 should be “proposed”?

summary_of_the_review: In conclusion, despite the successful improvement of the LML task, I still have lots of concerns about the labeling results, which is the key to improvements. I would be more convinced if more analysis of labeling results could be done in this paper. For the current version, I recommend a negative score as my initial rating.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes  data programming method, named Mako,  for semi-supervised continual learning. Mako automatically generates labels for unlabeled data with a set of weak labeling functions, each of the functions is trained on subset of training set with bootstrapping. Experimental on several datasets demonstrate the effectiveness of the proposed methods. 
main_review: Strengths:

*  The paper is clearly written and the topic is relavent.
*  Experiments on  several different scenarios 


Weakness:
* The technical novelty of proposed methods is very limited. The method is simply the combination of existing methods. For example, the classifer is a collection of weak labeling functions with bootstrapping, e.g., Snuba. In addition, the data programming techniques are not novel as well. It is very similar to the self-training [1], which predicts the labels of unlabeled data during the training process.  The self-training techniques are widely adopted in semi-supervised learning. It would be better to discuss in related work for the difference between the proposed method and self-training and compare to state of art self-training methods in experiment.

* There is lack of insights on the proposed method for mitigate catastrophic forgetting, which is main focus of this paper. 

* For the evaluation metrics, the author proposed  catastrophic forgetting ratio up to task i for evaluating the forgetting. However, for most continual learning works, they commonly adopt backward transfer [2] to evaluate the forgetting issue. It would be bettter to show the commonly adopted metrics for a direct comprions to related works. 

* For the experiment,  improvement on forgetting seems quite marginal, although the peak per-task accuracy is improved. Also, seems that there are many hyperparameters in the proposed method. It would be great to show the sensitivity of the hyperparameters.


Reference:
[1] Uncertainty-aware Self-training for Text Classification with Few Labels. Subhabrata Mukherjee and Ahmed Hassan Awadallah. NeurIPS 2020
[2] On Tiny Episodic Memories in Continual Learning, Arslan Chaudhry, et al. https://arxiv.org/abs/1902.10486
summary_of_the_review: The paper is clearly written, but the technical novelty is quite limited. The analysis and experiments also need to be significantly improved.
review_title: null

Meta-review: This submission proposes "Mako", which enables continual learning when only a limited amount of labeled data is available (along with a good deal of unlabeled data). Reviewers shared concerns about difficulty in understanding which components of the proposed system were novel, especially given that the most important components seemed to be proposed in past work. Reviewers also had difficulty getting insight on which parts of the system were most useful, and further requested additional experiments on harder benchmarks. There consensus was therefore to reject the paper.
Decision: Reject