Mako: Semi-supervised continual learning with minimal labeled data via data programming
Authors: Pengyuan Lu
Seungwon Lee
Amanda Watson
David Kent
Insup Lee
ERIC EATON
James Weimer
Venue: ICLR 2022 Submitted (2022)
Keywords: lifelong machine learning
data programming
semi-supervised learning

Abstract: Lifelong machine learning (LML) is a well-known paradigm mimicking the human learning process by utilizing experiences from previous tasks. Nevertheless, an issue that has been rarely addressed is the lack of labels at the individual task level. The state-of-the-art of LML largely addresses supervised learning, with a few semi-supervised continual learning exceptions which require training additional models, which in turn impose constraints on the LML methods themselves. Therefore, we propose Mako, a wrapper tool that mounts on top of supervised LML frameworks, leveraging data programming. Mako imposes no additional knowledge base overhead and enables continual semi-supervised learning with a limited amount of labeled data. This tool achieves similar performance, in terms of per-task accuracy and resistance to catastrophic forgetting, as compared to fully labeled data. We ran extensive experiments on LML task sequences created from standard image classification data sets including MNIST, CIFAR-10 and CIFAR-100, and the results show that after utilizing Mako to leverage unlabeled data, LML tools are able to achieve $97\%$ performance of supervised learning on fully labeled data in terms of accuracy and catastrophic forgetting prevention. Moreover, when compared to baseline semi-supervised LML tools such as CNNL, ORDisCo and DistillMatch, Mako significantly outperforms them, increasing accuracy by $0.25$ on certain benchmarks.

Under review conference paper ICLR 2022
MAKO: SEMI-SUPERVISED CONTINUAL LEARNING WITH MINIMAL LABELED DATA VIA DATA PROGRAM- MING
Anonymous authors Paper under double-blind review
ABSTRACT
Lifelong machine learning (LML) is well-known paradigm mimicking the hu- man learning process by utilizing experiences from previous tasks, Nevertheless. an issue that has been rarely addressed is the lack of labels at the individual task level. The ofate-of-the-art of LML largely addresses supervised learning, with few semi -une-supeeeieed continual learning exceptions which require training addi- tional models, which in turn impose constraints on the LML methods themselves. Therefore, we propose Mako, a wrapper tool that mounts top of supervised LML frameworks, leveraging data programming. Mako imposes no additional knowledge base overhead and enables continual semi-supervised learning with a limited amount of labeled data. This tool achieves similar performance, in terms of per-task accuracy and resistance to catastrophic forgetting, as compared to fully labeled data. We ran extensive experiments on LML task sequences created from standard image classification data sets including MNIST, CIFAR-10 and CIFAR- 100, and the results show that after utilizing Mako leverage unlabeled data, LML tools are able to achieve 97% performance of supervised learning on fully la- beled data in terms of accuracy and catastrophic forgetting prevention. Moreover, when compared baseline semi-supervised LML tools such as CNNL, ORDisCo and DistillMatch, Mako significantly outperforms them, increasing accuracy by 0.25 on certain benchmarks.
INTRODUCTION
Since 1995, researchers have been actively seeking machine learning (ML) frameworks that can better mimic the human learning process by retaining memories from past experiences (Thrun & Mitchell 1995 Liu 2017). Hence, in contrast to traditional isolated ML where knowledge is never accumulated, the concept of lifelong machine learning (LML) is defined as training MI. models for continual learning over task sequences, with a knowledge base to store information that could be helpful in the future. Under this definition, LML can be seen as continual transfer learning (Pan & Yang. 2015) with memory or sequential multi-task learning (Caruana 1997).
Nevertheless, the level of individual tasks, the current state-of-the-ar of LML still remains largely supervised. A challenge that has rarely been mentioned is that labels can still be expensive and rarely available for each task, such that supervised training can hardly provide acceptable performance. For LML. performance is not only measured at task level, but also for the entire lifetime, with attributes such as resistance to catastrophic forgetting and space- epace-efficiency of knowledge base.
We take more generic approach to semi-supervised lifelong learning and develop a technique that wrap around any existing continual learning algorithm. Specifically, propose to data programming (Ratner al. (2116)) to label new data under the supervision of set of weak labeling functions trained from a limited number of labeled data points. Data programming has proven successful for isolated learning, and so this is the first work integrating with lifelong learning.
Our approach, called Mako, sits atop an existing lifelong learner turn into semi-supervised learner. For each task, Mako takes as input a small labeled dataset plus an unlabeled dataset with no restriction on size, generates labels for the unlabeled data and then updates the lifelong learner supervised by both sets. This approach builds upon the widely used Snuba algorithm (Varma & Rd

---

Under review conference paper ICLR 2022
2016), and we show that theoretical guarantees still hold in the lifelong setting. Mako's s procedure leverages automatic hyperparameteer tuning, data programming and confidence calibration, in order to adapt automatic labeling to constantly varying LML tasks. We show that Mako produces ade- quately high performance, terms of both task accuracy and prevention of catastrophic forgetting, that approaches that of training on fully labeled data without costing extra knowledge base space overhead.
We extensively evaluated Mako on various partially labeled LML task sequences generated from commonly used datasets including MNIST, CIFAR-IO and CIFAR-I00 (LeCun & Cortes) 2010; Krizhevsky 2009), while mounting on diversity of supervised LML frameworks such as Decon- volutional Factorized CNN (DF-CNN), Dynamic Expandable Network (DEN) and Tensor Factor- ization (TF) (Lee et al., 2019; Yoon et al., 2018 Bulat et al. 2020). We compared the results to supervised learning on fully labeled data, as well as the same partially labeled data with existing semi-supervised LML tools: CNNL, ORDisCo and DistillMatch (Baucum al., [017] Wang et al., 2021] Smith et al. (02T). Empirically, we show that the performance of LML. methods improves as giving more traminn data even with Mako labels, achieving at least 97% performance relative to LML methods trained ground-truth labels and being able to beat existing semi semi-supervised LML tools by approximately 0.25 higher accuracy.
Contributions Summary:
1. Adapting automatic label generation by semi-supervised learning/data programming to L.M. the scenario where labeled training data is expensive to obtain compared to unlabeled for all tasks.
2. Implementing a LML wrapper, namely Mako, for the scenario of expensive labels, such that, when given partially labeled data limited size and compared to the current supervised state-of- the-art with fully labeled data, each LML task sequence (1) requires significantly fewer labeled data to achieve high accuracy
individual tasks, (2) maintains similar resistance to catastrophic forgetting and (3) does not cost extra knowledge hase storage.
3. Extensive experiments on lifelong binary and multiclass image classification on 3 commonly used datasets, including comparison supervised and supervisd LML baselines, accomplish- ing very close performance to supervised LML on fully labeled data and significantly outperforming baseline cemi-supervised LML frameworks.
2 RELATED WORK
Lifelong machine learning (LML) is concept of continual learning among sequence of tasks. Specifically, given multiple machine learning tasks arriving continuously, the LML framework ac- cumulates the knowledge of previous tasks and utilizes it for future ones (Chen & Liu 2016; Ruvolo & Eaton 2013b). The method of retaining and adopting the knowledge base varies depending on the characteristics of tasks and learning models. For instance, a latent matrix is used to keep proto- typical weights for linear models (Kumar & Daume 2012 Ruvolo Eaton (013a)) and for deep neural networks, either weights of layers (Yoon et al. [2018] Lu et al. [017 Bulat et al. 2220 Lee et al.] (219)) extracted features of layers (Rusu al. [216;] Misra et al. 2016)] Gao al.) 2019]
Schwarz et al. 2018) are considered as the know knowledge hase to transfer. Besides research on models to transfer knowlede ledge across tasks, the LML community has also been working on task order sorting for higher overall performance (Sun al. 2018 Ruvolo & Eaton| 2013b). Nevertheless, all these works assume that each individual task training is supervised with ground-truth labels for the entire training set. In many real-life MI. tasks, obtaining labels has been and continues to he expensive (Settles (009). Only few works have addressed this issue on LML in very specific domains, such as continual sentimental analysis with words partially labeled (Wang et al. 2016) and reinforcement learning in a sequence of state spaces without explicit rewards (Finn et al. 2017). Different from previous work, Mako enables high performance in continual learning on generic ML tasks given partially labeled data, with the cardinality of labels restricted at small number.
To tackle expensive labels in ML, researchers have purposed various methods such as active learning (Settles) (2009) or semi- supervipe learning (Olivier et al. 2006), which is an ML framework that takes both labeled and unlabeled data as inputs. The unlabeled data can directly assist the training, or can serve as target data set yet to be labeled. In recent years, with the rising enthusiasm and need in deep learning, various methods have been used to assist supi-supervised learning with DNN Tarvainen & Valpolal 2017). One branch of
models (Rasmus et al. 2015
Laine & Aila 2016

---

Under review conference paper ICLR 2022
sepe-supervised learning, namely data programming or weak supervision (Ratner et al. 2016b). focuses on anto-generation of labels on an unlabeled dataset with the help of a small labeled dataset. Targeting this problem, researchers leverage weak labeling functions that each can label with an accuracy slightly higher than coin flip. Then, ensembling algorithms are able to bag these weak labelers and produce highly accurate final labels. In other words. the weak labels supervise the training of the final ensembled generative model. There exists different implementations of such generative model training such as Snorkel (Ratner et al. (2016a).
Semi- sumi-supereiiee learning is an underexplored problem the LML setting; all the works mentioned above have addressed problems in isolated ML only. Early work in semi-supervised incremental learning. in which all tasks trained simultaneously batches of unlabeled data incremen- tally observed by classifier, has identified challenges pertaining to selection of the label generating model architecture and confidence calibration of the generated labels (Baucum ct al. 2017). pro- cesses which Mako undertakes autonomously through automatic search for weak labelers before en- sembling, and confidence calibration after labels are produced. Pioneering work in supeissdd LML requires training of additional models for -ut-t--distributtion detection (Smith et al. 202T) or generative replay (Wang al. [2021). Conversely, by focusing label generation through data pro- gramming, our approach achieves similar behavior through a pre-processing workflow that does not impose additional constraints on the LML approach or require additional knowledge base storage.
For data programming tools, it has been long standing issue to obtain weak labeling functions input. One approach is manually designing the weak labelers by domain experts (Ratner et al.] 2016b a). while other researchers seek automatic labeler generation. For instance, Snuba (Varma & Ré. 2016) outputs weak supervisors with an accuracy guarantee on the final ensembled data programming model. Thus far, Snuba has been evaluated various isolated ML tasks, with weak labelers in form of k-nearcst-neigeee neihhbbrs,, decision stumps and logistic regressors, and has been shown to ace in natural language processing and image classification. It also succeeds in more complex applications such as classifying bone tumors medical domain. In our work, we build weak labeling function generation module in Mako based on Snuba, aiming to support a larger variety labeler models for LML, while maintaining the theoretical accuracy guarantee of generated labels.
3 PROBLEM FORMULATION
LML: Given sequence of ML tasks: Tı, T1.T2..... Twe where is unbounded integer, each task Ti has an underlying data space X; and label space Vi. T; has iitroin labeled training data (Xi,rrans Yrtaii)) D,, such that Xi,rrann X," X. and Y, trasnn V, Viiiiteiin following some task- i.i.d. distribution D., The task also has Restest testing data (Xeteats Vi,est) drawn from the same distribution.
Traditionally, an LML model encounters the tasks in order. At T;, it trains on the full set of (X Yr,rai) with supervised learning and obtain model f; : Uj=11 N. + Uj=1 V%. Im- mediately after this training, it predicts the j-th task with Yo3 fi( for each j $ There are two objectives:
1. High test accuracy. Formally, denote testing accuracy of task T, right after the training of task T, on (Xi treva) as
Biitta 1( vk)
aij
Njatest
with DR and ye the k-th label in Yi and Yitest, respectively. One LML objective in the current state-of-the--at is to maximize the peak per-task accuracy up task i defined as
Σ
0; =
03
Peak per task accuracy is defined on all tasks = 1,..,, m and shows the LML performance on the current training task on the basis of the accumulated knowledge.
2. Low catastrophic forgetting. A critical aspect LML is to prevent catastrophic forgetting of previous tasks. The resistance to catastrophic forgetting can be quantified by the average testing accuracy of all tasks encountered far. For each task Tü, denote second objective, final accuracy

---

Under review conference paper ICLR 2022
up task as
i
The second objective maximize a, for all 1, 1.... The gap between peak per-task accuracy and final accuracy expresses how much knowledge previous tasks LML method can preserve through the streams of tasks. Therefore, we define a third metric to be maximized, catastrophic forgetting ratio up to task i, as
(4)
This metric is less than 1 if the LML model loses its performance the earlier tasks, and it is greater than 1 there is positive knowledge transfer from the later tasks to the earlier ones.
Semi-supervised LML with data programming: The key challenge we want to address is the cost of labeling data in all LML tasks. That is, the number of available labeled training data is small. Therefore, we would like to split (Xxstrosn: Yetroin) in the traditional LML problem into disjoint (X,,,, E,, X,,t), where (Xi,z, Yi,z) is a set of labeled training data with restriction on size and X,,r is set of unlabeled data with no such restriction. In other words, [ is small, e.g. [X,,] < 150 for 10-way image classification problem.
Apparently, the LML model now trained with supervisd learning. For task T;, we would like ) to first train generative model n, & + V.
to leverage data programming on (X.,,, Y,, X. This model automatically labels X,, with Y7, n, (X,,))). Then, the LML framework will be trained on LX,,,, Yi,,, X, ). Denote the LML model immediately after this training
J - X + - Di. and y = (Xx,tee)) each task j
In this scenario, we redefine testing accuracy in Equation 1 as
1 MK)
(5)
njttest
where the only difference is that vī is the k-th label in Y3 instead of Yij. From Equation 5 we can then redefine metrics in Equations and 4
0
Our research problem is to design semi-supervised LML framework leveraging data programming that minimizes the performance between using partially labeled data, (X Y,,, X,,, Y,,)), and the upper-bound bound performance using fully labeled data, (X,,,, L,,, Y,,),, in terms of these three metrics while imposing additional knowledge base storage overhead in the LML framework.
4 MAKO: LIFELONG MACHINE LEARNING FRAMEWORK
implemented Mako, a semi-supervised LML framework that can be mounted on top of any ex- isting LML tool, with (Xi.,, Yi..; Xx.::) as input for each task T. The automatic label generation of Y, is done by data programming, consisting of generating weak labeling functions and ensem- bling a generative model. Nevertheless, different from data programming on isolated ML, due to the variation of tasks throughout LML, we need to actively adapt both the weak and the strong models maximize the performance each task. This is accomplished by automatic search architec- ture and training hyperparameter search before weak labeler generation and confidence calibration after the ensembled model is trained. The following subsections explains the Mako workflow step-by-step, as illustrated in Figure]

---

Under review conference paper ICLR 2022
Ayperparameter Search
Programming
Confidence Calibration
Architecturo
Training
Bootstrapingg
Frisebble
Temperature Scaling
CNN
DEN
EWC
Porameters
Labeler
Calibrated Labels
Learning
Model
60è
Lngits
Rates
Generation
Butch Sue
Epochs
è 00000
Threshalding
Testing Acturacy
Repest Until Sufficient Labelers
Satistying the Theoretical Guarantee
Figure I: Mako Framework
4.1 AUTOMATIC HYPERPARAMETER SEARCH
Before producing weak labeling functions, two questions must be answered: (1) How do we choose the model architecture for weak labeling functions? (2) How do train these models after architecture decided? Formally, these questions lead the decision of architectural and training hyperparameters of labelers, which Mako accomplishes automatically.
The only prior knowledge required is a search space, mamually designed, of the architecture and training hyperparameters. For instance, given that the task is image classification, the search space of labeler architecture will include CNNs with the same input and output dimensions, but different configurations of hidden layers. Similarly, the training hyperparameters will be searched on different learning rates, batch sizes and number of epochs.
The automatic tuning of weak labelers therefore traverses the search space until satisfactory con- figuration is found. The stopping criterion is defined as follows: after training ten labelers with a given configuration of architecture and training hyperparameters on different bootstrapped subsets of (X,,t, Y.,.). 9 out of 10 achieves am accuracy on the entire (X.,,, Y2,r.) above some threshold a. This criterion shows the generalization capability of sampled labelers and alternative criteria can be used, such as 8 out of 10 or average accuracy 2 We accept such configuration because we train the labelers
bootstrapped subsets of data during data programming, which will be further explained in 4.2, If no configuration in the search space is satisfactory, we decrease by 0.05 and repeat the search. Hence, we can start from sufficiently high such 0.9. For efficiency, Mako selects the configuration with the smallest number of training epochs among all the acceptable ones.
There exists a efficiency-automattio automation trade-off in this byperparameter search. The less prior knowl- edge we have for the task, the larger search space we need to input to Mako and thus a larger time overhead. In contrast, more manual exploration on the task will lead to a smaller search space and therefore efficiency.
4.2 GUARANTEED DATA PROGRAMMING WITH BOOTSTRAPPING ON TRAINING SET
After deciding the architecture and training hyperparameters of weak labeling functions, Mako kicks off weak labeler training and ensembling of generative model w, for each task T, For the training process, we are inspired by Snuba (Varma & Ré, 2016) due to the fact that has a theoretical accuracy guarantee in the final generative model. Nonetheless, in order to increase variance among the labelers, Snuba trains them different selected subsets of features. This procedure does little favor in LML, where we might encounter complex tasks in the sequence that requires the entire feature space. For instance, in image classification, excluding pixels in training may lead to failure in capturing important feature-target relationships, especially when we want to take advantage of local connectivity of CNNs. Therefore, Mako uses bootstrapping on (Xa,,, Y.,)) and trains each labeler with a sampled subset. The bootstrapping size can be either manually given or searched as a training hyperparameter.
Besides bootstrapping data, the rest of weak labeler generation process remains unchanged from Snuba in order to maintain theoretical guarantee. Snuha has such guarantee because it checks an exit condition on whether the committed lahelers have labeled sufficient number of data points in Xs with acceptable confidence. If so, these labelers guarantee that the final generative model will have learned accuracy on X,,L close to the labeling accuracy on X.u. Formally, this guarantee can be stated as Proposition 1.
Proposition 1 (Theoretical Guarantee of Snuba): Given h committed weak labelers, denote the empirical accuracies of all committed weak labelers on X.L. as a vector L. The generative

---

Under review conference paper ICLR 2022
model training outputs learned accuracies of these labelers on XL,L as as,L, before ensembling. Moreover, the unknown true accuracies of the labelers on X1, after generative model training are denoted as asSU.g. with as L.w, Ls Ug, IR... We have a measured error [|as, Ls,L,glloo I e. Then, if each labeler labels a minimum of
2h
d c)² log
(9)
data points in X.,, with above some given confidence threshold v for all iterations, we can guaran- tee that || 0.,1,gllxx < for all iterations with probability 1 1 8.
The proof of Proposition from the original Snuba paper is included in Appendix A. We would like show that our bootstrapping technique maintains this guarantee, as in Proposition 2.
Proposition 2 (Theoretical Guarantee of Mako Weak Labeler Generation): Despite the modi- fications from Snuba, the Mako weak labeler generation process satisfies the theoretical guarantees described in Proposition 1.
Proof of Proposition Although our weak labelers are generated with (1) different model archi- tectures and (2) sampling by bootstrapping on data instead of features, it still satisfies the definition of weak heuristies required by Snuba. That is, function f X. + 3% U {abstoin), where abstain a special label assigned data points with a confidence lower than the threshold p.
Consequently, by Proposition 2, Mako checks the same theoretical exit condition at each iteration, the final generative model satisfies the same theoretical guarantee as in Snuba. The hyperparameter search described in 4.1 can be understood looking for labelers with high d.L, Based on this guarantce, we will have ||a.U.g d. so I E + 7 with probability 6, and hence we can probablistically achieve high labeling accuracy 01,Ug9
The weak labeling functions are then used to label both X and Xi, and the labels are input for the training of generative model T.. We label X., as well for confidence calibration, which will be explained in the following subsection.
4.3 CONFIDENCE CALIBRATION AND TURESHOLLING
Considering the variability and complexity of LML tasks, we believe the generative model labels can be further improved approach the ground truth. Therefore, Mako takes an extra step adjust the trained generative model n. by confidence calibration on the produced logits on X.UU-
Confidence calibration (Guo et al., 2017) is the process to align the confidence of labeling a data point to the accuracy, which is estimation of the probability that the data point is labeled correctly.
The procedure of confidence calibration is adjusting the logits in order to minimize the expected calibration error (ECE), formalized as
min Ept,, [| PPI[VE,S. = UK.. PÊ.. = p) 10-p) pl]
(10)
Pl,c.
where yk U is the ground truth label of the k-th data point in X1,,, V., .17 is its predicted label by the data programming generative model and P..s.U the associated confidence. However, since the ground truth labels of X,, unknown, this approach is infeasible. This is the reason for training the generative model under the supervision of weak labels on both X,, and X,,,. The model is hence expected to have the similar ECE on labeled and unlabeled data. Mako then performs temperature scaling (Platt) 1999) minimize ECE on X.,,, and then apply the same temperature on X adjust the logits.
After calibration, there could still exist data points labeled with low confidence. We then threshold Xs, with confidence of 1 //tecaasess + 3 for some small positive B. For example, in a 10-way classification problem, pick 3 0.01. We only keep the subset of X with calibrated confidence 2 0.1 + 0.01 0.11. Denote this kept portion of unlabeled data and labels as (X,, Y,
4.4 LML TASK TRAINING AND EVALUATION
Finally, XI,., YLL, Xy,., and eee. will be input into the mounted LML tool for the training task T. The evaluation of the current model will be performed on all the hold-out testing data set (Xteet Yess,))...., "Xtees,,,,,,.t, (Xtest, Y.ests).

---

Under review conference paper ICLR 2022
Mako does not alter the internal procedures of the LML tool. Instead. it generates labels for unla- beled data and then feeds to the LML tool as a black box. This design enables high modularity and the LML tool can be easily replaced. In the experiment section, we will demonstrate Mako's performance when mounting on various of LML paradigms such DF- DFNCNN, DEN and TP. This design also imposes no additional knowledge space storage in LML.
5
EXPERIMENTS
5.1 DATASETS AND LML TASK SEQUENCES
We evaluate Mako LMI. task sequences generated from commonly-used image classification datasets: MNIST (LeCun & Cortes 2010), CIFAR-10 and CIFAR-100 (Krizhevsky 2009).
We mount Mako on existing supervised LML tools: DF-CNN, and TF (Lee et al. 2019 Bulat et al. 2020) to enable semi-supervised learning, and compare the performance to supervised LML on labeled data only as well as all data fully labeled. The task sequences are:
Binary MNIST: for each task, pick classes 0 vs 1,0 2, .... vs 9, with 45 tasks in total.
2. Binary CIFAR-10 10: same construction as binary MNIST.
Additionally, we compare Mako-mounted supervised LML tools to existing semi- supei-sueeeiieed LML:
CNNL., ORDisCo, and DistillMateh (Baucum al. 2017 Wang al. 2021 Smith al. 2021
generate the same task sequences as their papers:
10-way CIFAR-10: the entire CIFAR-10 as one task.
4. 5-way CIFAR-I00 100: for each task, pick each superclass in CIFAR-100, with 20 tasks in total.
5-way MNIST: for each task, pick classes 0-4 and 5-9 in MNIST, with tasks in total.
Details of tasks such as data splits, hyperparameters searched and labeling accuracy are explained in Appendix B.
5.2 COMPARISON TO SUPERVISED LML
For each task, data split into labeled training set, unlabeled training set and hold-out test set. The details of data split and weak labeler models searched are included Appendix
the performance LML methods according to the amount of training data: I. standing for the labeled training set and U with number denoting the number of instances in the unlabeled training set used during training of each task. These tables show peak per-task accuracy, final accuracy and catastrophic forgetting ratio up to the last task (Eqn. 66 7 and []] respectively).
aable[[andDSummattt
Additionally. we trained the LML methods on the same data with true labels instead of the Mako- generated labels quantify the quality of the generated labels. L.L methods achieve better peak
Performance Pinal Acc.
Relative to True Labels
LML.
Data
Per-Task Acc.
Forget. Ratio
Per-Task Acc.
Final Acc.
95.6 = 0.4
96.8 0.6
1.02 + 0.01
L+030
95.9 = 0.4
97.0
0.4
1.01 = 0.01
0.99
1.00
TF
L+U60
96.1 0.3
96.8 0.5
1.01 = 0.01
0.99
0.99
L+U120
96.2 = 0.2
96.9 $9.9=00 0.2
1.01 = 0.00
0.99
0.99
L+U240
96.5 + 0.2
96.4 + 0.2
1.00 + 0.00
0.98
0.98
93.8 0.4
95.4 0.3
1.02 = 0.01
L+030
0.3
96.2 - 0.4
1.02 = 0.01
0.98
1.00
DF-CNN
L+U60
94.9 0.2
96.1 0.7
1.01 - 0.01
0.98
0.99
L+U120
95.2
95.8 0.9
4 0.01
0.98
0.99
L+U240
95.9 - 0.2
94.3 + 1.3
0.99 + 0.01
0.98
0.97
Table I: Supervised LML experiments binary MNIST tasks, showing mean 4: standard deviation.
LML models are trained on either Labeled data only or Labeled data and instances of Unlabeled data with Mako-generatte generated labels, and evaluated three metrics up to task 45 (Eqn. 7 () as well as accuracy metrics relative LMI. models trained the data with true labels instead.

---

Under review conference paper ICLR 2022
Performance
Relative to True Labels
LMI.
Data
Per-Task Acc.
Final Acc.
Forget. Ratio
Per-TTsk Acc.
Final Acc.
81.5 0.2
76.8 + 0.7
0.95 + 0.01
TF
L+U200
83.4 + 0.4
76.8 = 1.6
0.93 = 0.02
0.99
1.00
L+U400
84.3 0.3
76.7 + 1.4
0.92 = 0.02
80.7 0.3
80.4 + 0.7
1.01 + 0.01
DF-CNN
L+0200
82.7 + 0.3
80.0 + 0.5
0.98 + 0.01
0.99
0.99
L+U400
84.0 0.3
80.0 = 0.4
0.96 = 0.01
Table 2: Supervised LML experiments on binary CIFAR-10 10 tasks, mean + standard deviation.
per-task accuracy as more Mako-labeled data is provided for training. This shows that Mako is able to generate useful labels for training while the current task. and especially data distribution, keeps changing in the lifelong learning setting. Training with Mako labels has no significant improvement final accuracy, but, compared to I.M.. trained on the true data, both peak per-task accuracy and final accuracy are at least 97% of the counterparts. Learning curves for each experiment are shown in Appendix C
5.3
COMPARISON TO DEMI-SUPERVISED LML
As discussed in Section 2 there little prior work in supervisd LML settings. We identify three approaches for comparison CNNL,, ORDisCo, and DistillMatch (Baucum et al. 2017; Wang et al. [22T Smith et al. 2021). Each baseline approach contains multiple modules and networks which each require their own tuning: to enable fair comparisons, we replicate the experimental con- ditions (data set, amount of labeled vs. unlabeled data, task definitions, and network architecture) and compare approach to the best results reported in each baseline's original publication. Specif- ically, we replicate the instance-inen incremennal learning experiments CNNL on MNIST and CIFAR- 10, and the class-incremental learning experiments of ORDisCo and DistillMatch on CIFAR-I0 and CIFAR-100, respectively. We briefly describe each experiment below; for more details, see Appendix B for data splits and the original publications for full experimental descriptions.
Ietance-Inerementaa Experiments. We compare semi-supervised learning using Mako an filance-incrementa setting (i.e., where all tasks are present at every epoch, but subsequent epochs contain different batches of unlabeled data), using identical two-convolutional-Bee scribed in (Baucum et al., 2017). The MNIST experiment uses labeled data set of 150 images, with 1000 unlabeled images introduced in each of 30 instance-inerrmmental batches. The CIFAR- 10 experiment uses labeled data set of 2000 samples, with subsequent batches of 1000 unlabeled images. Each Mako experiment was run over random seeds. For additional context, we include the performance of the same network trained in the instance incrremental setting where all data is labeled with the ground truth, representing shown in Table 3 using Mako labels in place CNNL's semi-supervised labeler results compa- rable classification accuracy with better sample efficiency (MNIST), or strictly higher classification accuracy (CIAAR10,, Learning curves for each experiment shown in Appendix D
CNN de-
upper bound semi-superr supervieeeeieed performance. As
MNIST
CIFAR- 10
Approach CNNL
Final Acc.
Batches to Saturation
Final Acc.
Batches to Saturation
90.0
26
45.7
25
Mako Labeled
90.0 = 0.4
3.3 + 1.6
54.2 = 0.5
26.5 - 1.1
Fully Labeled
99.0 + 0.1
17.0 + 4.0
57.6 + 0.5
27.1 + 1.7
Table 3: mnstance-ineremen
semi-supervised LML experiments, showing mean standard devi- ation (where available). Final accuracy (see Fqgn. [7) is measured on the data set's standard held out test set, batches to saturation is measured as the first epoch where a 3-batch sliding window average meets or exceeds the final accuracy
Tass-Incremental Experiments. We compare state-of-the-art LML methods (DF-CNN, DEN, and TF (Lee et al., 2019 Yoon et al. 2018 Bulat et al., (2020) using Mako labels to perform semi- supervised learning in a llass-incremental setting (i.e., the model is sequentially presented with tasks containing new sets of classes). For all experiments below, each Mako-enabled approach

---

Under review conference paper ICLR 2022
over 10 random seeds, and we also include the performance the same using all ground truth labels. We first compare Mako to ORDisCo over five binary CIFAR-I0 tasks using 400 labeled instances, with the remaining data unlaheled. We note that we could not directly replicate the ORDisCo clas- sification network architecture due lack of details in the original publication, but instead show that using Mako labels achieve a higher total classification accuracy with DF-CNN and TF using significantly smaller network (4 convolutional layers for the Mako approaches, compared OR- DisCo's s convolutional layers), shown in Table Additionally, we compare Mako DistillMatch over the 5-way CIFAR-100 100 using 20% labeled data, replicating DistillMatch's ParentClass task. As shown in Table all of the LMI. methods enabled by Mako labeling meet exceed DistillMatch's performance over the learning task. Learning curves for each experiment are shown in Appendix D
CIFAR-10
CIFAR-100
Approach
Final Acc.
Approach
Final Acc.
Baseline
ORDisCo
74.8
DistillMatch
24.4 + 0.4
Mako-laheled DE-CNN
86.8
1.3
Mako-labeled DF-CNN
50.0 + 0.8
Semi-
Mako-labeled DEN
61.4
Mako-labeled DEN
24.1 0.6
Supervised Mako-labcled TF
82.1 1.1122 2.7
Mako-labeled TF
48.9 2.1
Fully-labeted DF-CNN
86.8 + 1.4
Fully-labeled DF-CNN
52.4 + 1.4
Supervised
Fully-labeied DEN
61.4 + 3.7
Fully-labeled DEN
23.7 = 0.4
Fully-labeled TF
82.5 + 4.6
Fully-labeled TF
51.0 + 1.8
Table 4: Class-incremental semi-supervised LML experiments, showing mean standard deviation (where available), Final accuracy (sce Eqn. 7 is measured on the data set's standard held out test evaluated on all tasks after the final task completes training, For a breakdown of individual task accuracy and catastrophic forgetting ratios, see Appendix
CONCLUSION AND FUTURE WORK
In this paper, we identified the challenge that collecting task-level labeled data for LML is expensive. We address this challenge with data programming, where labels automatically generated, and implemented the Mako framework that can be mounted on top of existing LML tools. Mako takes in a limited number of labeled data and an unlimited number of unlabeled data, aiming to minimize the performance gap using the same data but fully labeled. We demonstrated Mako can achieve sufficiently high accuracy per task as well resistance catastrophic forgetting, while costing no additional knowledge base storage, over a set of common image classification LML task sequences.
Future work on this topic can consider how to better resolve the issue of expensive labels at individ- ual LML tasks. This includes how to extend Mako larger variety of LML task sequences. For instance, what alternative methods can be used labeler hyperparameter tuning other than search- ing, what prior knowledge of tasks could assist automatic labeling, and how should the efficiency- capability trade-off of different models be balanced. Outside of the Mako framework. alternative solutions can modify existing LML tools directly, compromising certain degree of modularity while possibly improving the overall LML performance.
Another interesting future direction is to consider the scenario where the labeled data and unlabeled data input per task drawn from different distributions. This be enabled by optimal transport theory, specifically the minimization Sinkhorn distance between two distributions (Cuturi 2013). Ideally, this approach could allow unlabeled data from other datasets to assist LML, resolving the issue of only expensive labels but also expensive data collection.

---

Under review conference paper ICLR 2022
7 REPRODUCIBILITY STATEMENT
We encourage researchers replicate experiments. As discussed in Section 4, have pro- vided the step-by-step explanation of Mako workflow. have included more experiment details in Appendix for convenience. Additionally, we pushed a sample code to an anonymous GitHub com/mako-oooooooooooo mako- anon/makoaaaoaaa All readers welcomed pull our repository (ht //github. cow/
code and execute Mako on the example data themselves.
REFERENCES
M. Baucum, D. Belotto, S. Jeannet, E. Savage, P. Mupparaju, and C. W. Morato. Semi-supervised deep continuous learning. In Proceedings of the International Conference Deep Learning Technologies, pp. 11-18, 2017.
A. Bulat, J. Kossaifi, G. Tzimiropoulos, and M. Pantic. Incremental multi-domain learning with net- work latent tensor factorization. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2020.
R. Caruana. Multitask Learning. PhD thesis, Carnegie Mellon University, 1997.
Z. Chen and B. Liu. Lifelong Machine Leurning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2016.
M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems (NIPS), 2:2292-2300, 2013.
C. Finn, T. Yu, J. Fu, P. Abbeel, and S. Levine. Generalizing skills with semi-supervised reinforce- ment learning. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Y. Gao, Q. She, J. Ma, M. Zhao, W. Liu, and A. L. Yuille. NDDR-CNN: Layer-wise feature fusing multi-task cnn by neural discriminative dimensionality reduction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3205-3214, 2019.
C. Guo, G. Pleiss, Y. Sun, and K. Q Weinberger. On calibration of modern neural networks. Pro- ceedings of the International Conference on Machine Learning (ICML). 70:1321-1330. 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. IEÉE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
A. Kumar and H. Daume. Learning task grouping and overlap in multi-task learning. In Proceedings of the 29th International Conference Machine Learning, pp. 1383-1390. Omnipress, July 2012. ISBN
S. Laine and Aila. Temporal ensembling for semi-supervised learning. Proceedings of the Inter- national Conference on Learning Representations (ICLR). 2016.
Y. LeCun and C. Cortes. Mnist handwritten digit database. 2010. URL htllhtt:////a..eeem ccom/ excb mnist. st.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-hased learning applied to document recog- nition. Proceedings of the IEFE, 86(1) 1)), 1998.
S. Lee, J Stokes, and E. Faton. Learning shared knowledge for deep lifelong learning using deconvo- lutional networks. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 2837-2844, 2019.
B. Liu. Lifelong machine learning: paradigm for continuous learning. Frontier of Computer Science, 11(3):39-331, 2017.
10

---

Under review conference paper ICLR 2022
Lu, A. Kumar, S. Zhai, Cheng, T. Javidi, and R. Feris. bully-adaptive feature sharing in multi- task networks with applications in person attribute classification. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, July 2017.
I. Misra, A. Shrivastava, A. Gupta, and M. Hehert. Cross-stitch networks for multi-task learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994- 4003, 2016.
C. Olivier, S. Bernhard, and Z. Alexander. Semi-supervised learning. IEEE Transactions on Neural Networks, 20, 2006.
J. Pan and Q. Yang. A survey on transfer learning. IEEE Transaction on Knowledge and Data Engineering, 12(10):1345-135) 2015.
J. C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likeli- hood methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and T. Raiko. Semi-supervised learning with ladder networks. Advances in Neural Information Processing Systems (NIPS), 2015.
A. Ratner, S. H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré. Snrokel: rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, (1(3), 2016a.
A. Ratner, S. De Sa, C.and Wu, D. Selsam, and C. Ré. Data programming: Creating large training sets, quickly, Advances in Neural Information Processing Systems (NIPS). pp. 3574-3582, 2016b.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. CoRR, 1ess1666..4671,,20
P. Ruvolo and F. Faton, ELLA: An efficient lifelong learning algorithm. Proceedings of the Inter- national Conference Machine Learning (PMLR), 28(1), 2013a.
P. Ruvolo and E. Eaton. Active task selection for lifelong machine learning. Proceedings of the AAAI Conference on Artificial Intelligence, 27(1), 2013b.
J. Schwarz, W. Czameekii J. Luketina, A. Bruubsa--Bawwingee Y. W. Tch, R. Pascanu, and R. Had- sell. Progress & compress: A scalable framework for continual learning. In Proceedings of the International Conference on Machine Learning, pp. 4528 4528-4437, 2018.
B. Settles. Active learning literature survey. Technical report, Department of Computer Science, University of Wisconsin-Madison. 2009.
J. Smith, J. Balloch, Y. Hsu, and Z. Kira. Memory-efficient semi-supervised continual learning: The world is its own replay buffer. In Proceedings of the International Joint Conference on Neural Networks, 2021.
G. Sun. Y. Cong, and X. Xu. Active lifelong learning with "watchdog". In Thinty-Second AAAI Conference on Artificial Intelligence. 2018
A. Tarvainen and H. Valpola. Mean teachers are better role models: targets improve semi-supervised deep learning results. Advances in Neural Information Process- ing Systems (NIPS), 2017.
consistency
Thrun and T. M. Mitchell. Lifelong robot learning. In The Biology and Technology of Intelligent Autonomous Agents, pp. 165-196. Springer, Berlin, 1995.
P. Varma and C. Ré. Snuha: automating weak supervision label training data. Proceedings of the VLDB Endowment, 12, 2016.
Wang, K. Yang, C. L., I.. Hong, 7. Li, and J. Zhu. Ordisco: Effective and efficient usage of incre- mental unlabeled data for semi-supervised continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition (CVPR), pp. 5392, June 2021.
S. Wang, Z. Chen, and B. Liu. Mining aspect-specific opinion using holistic lifelong topic model. Proceedings of the International Conference on World Wide Web (WWW), pp- 167-176, 2016.
J. Yoon, E. Yang, J. Lee, and S. J. Hwang. Lifelong learning with dynamically expandable networks. Proccedings of the International Conference Learning Representations (ICIRR), 2018.

---

Under review conference paper ICLR 2022
SNUBA THEORETICAL GUARANTEE AND PROOF Proposition 1 in Section 4.2 and its proof are given by the original Snuba paper (Varma & Ré 2016). We adapt their notations our paper to support the propositions Section 4.2 on an arbitrary LML task T, For convenience, we remove the subscript in all notations.
APPENDIX A
Proof of Proposition 1: First, the probability of the fso error between the labeling accuracies of the generative model X and X is greater than equal to 7 can he bounded by the following triangular inequality, with the empirical accuracy of one weak labeler in the middle.
Pr|]|ac.g d1.gl>> 2 7 Σ Pr[ [00, - d2,=lloo + || d1,gيxx 2 7]
(11)
I Pr| loug - d2,elloo + 2 ml
Then, since we have weak labelers, the following union bound holds for an individual labeler j the committed set and its scalar accuracy.
Pr[llau, d1,wllx +
Pr(lougs ان,Wيl + 7l
(12)
In Xt, suppose there are dj data points not labeled abstain by labeler j. the definition of the empirical accuracy weak labeler j
GLad dz Ei(v= = Úg,1)
(13)
where ys is the ground truth label of data point & in the non-abstain subset and vf. is the label given by heuristic j. By combining Equations [2 and [3 and using Hoeffding's inequality the
independent data points, we can get
Pr|] Ag.g o lboc 6 2 7] = Pr[l anwolooo 27-cl
Pr Prilacsd ar, 12-12--- -c
1 E1(me 1(yx = (1xl))
Σ Pr Jassss
27 (1
(14)
d2
2 exp( -2(((-222)))) -))
j=1
S 2h exp( 2( - m ,d))
Equation 14 bounds the probability of Snuba's failure to keep || ||ap.y dg < 7 in one iteration. Assume without loss of generality that we commit one more weak labeler per iteration, so for the all h iterations, apply union bound again:
Pe|
"I, 1.o 2 for any iteration]
Prillan. an, 2 for one iteration]
(15)
212 exp(-2(1 - c)² min(dd...... d,))
Therefore, if we denote this probability of failure in any iteration as 8, have the bound 8 < 2h exp(-2(( e)² min(d1,.... d,))
(16)
and thus obtain Equation 9 by letting d min(d1,....dl
12

---

Under review conference paper ICLR 2022
APPENDIX B
WEAK LABELER MODEL HYPERPARAMETERS EXPERIMENTS
The LML task configurations are listed in Table 5 for the convenience of replicating our procedure. Because even after hyperparameter search (Section 4.1), there are still differences in the architectures and training hyperparameters throughout single L.I.. sequence, have provided shrunken search spaces in weak labeler architecture training hyperparameters.
LML Task Sequence
Data Split of X.,.. Xig, Xitest
Weak Labeler Architecture
Training
# Tasks
Hyperparams
Ir E [1c-4, 1c-2)
Thotches E [5,
Binary MNIST
45
120/1 1880/2000
Arch A
Repochs E [30, 100
bootstrop 30
Ir E [0.8e-3, 1e-3)
Abatohes 30
Binary CIFAR-10
45
400/9600/2000
Arch B
Nepochs E [30, 60)
bootstrap = 350
lr le-2
Arch B final out 10
Abathhes 30
10-way CIFAR-10
WWX030000/10000
Nepocks =
boolstrop 750
Ir E [0.8e-3, 1e-2]
Arch B but final out 5
1batehes 20
5-way CIFAR-100
20
500//000/500
Repocha E [60, 140
bootstrop 200
Ir E [0.8e-3, 1..---
Arch A but final out
Mbatches 10
5-way MNIST
2
150/29750/5000
Vepooha [180, 220]
bootstrop 50
Table 5: LML Task Configurations in Scction 5. Please refer Figure for weak labeler architec- tures.
0on. 1: pxd ael
coiee 2 pra ael.
ONDERD
and pett and
nee-C33
out-42 agrnal
out-27 gimood
outr2 zohivoo
Architecture used for MNIST and Fashion-MNIST
out_TT6, prt, sed,
brack 1: out p=18
track au) pai2
Brack S oul ca04
5-2, pro. sno, fatlen
2 solree
batchnomm
(b) Architecture B: used brased experiments
and, sel
nut cebiad sots barmssm
putto LMt pe rulu
pecaac mut bekhhoen
rou
c2nu 3 peb 5-1, batcmmom
cen: S: get saktnom
(c) Block structure of Architecture B, where block_out_cis the out_c input into the block
Figure 2: Searched Architectures of Weak Labeling Functions
For example, to re-create our binary MNIST LML tasks, one needs to first arrange the 45 tasks as 0 2, 8 vs 9. Then, split the labeled training data, unlabeled training data and testing

---

Under review conference paper ICLR 2022
data 180/11880/000.. All the data splits have balanced classes. Finally, perform our procedure described in Section 4, starting from the hyperparameter search in the shrunken search space.
Figure 2 illlstrates the scarched architectures. The search spaces were inspired by various previous works CNN designs (Lecun al. 1998 He et al. (016)) Nonetheless, the final architectures are much smaller since we need fast training multiple weak lahelers. The notations of the figures are: out_c: output channels/number of filters, k size (width and height) of filter, padding and s: stride.
Figure shows the labeling accuracies of the Mako-labeled data input into LML.
0.9
0.8
Accuracy
0.7
0.8-
0.4
Din MNIST
Bin CHAR
w0-wey CHEAR 10
5 CIFAR, 100
5-way MNIST
Figure 3: Labeling accuracies of the task sequences after confidence calibration and thresholding, with threshold 1/rrum classes + 0.01 for all tasks, iee. labeling accuracies of (Xs, Y.,e)) described in 4.3
14

---

Under review conference paper ICLR 2022
APPENDIX C ADDITIONAL SUPERVISED LML EXPERIMENT ANALYSIS
We visualize peak per-task accuracy and final accuracy of supervised LML experiments (Figure and learning curve final accuracies these experiments (Figure 5 summarized Table and In these plots, we used the same notation I. and U of the main text to specify the training data setting. except TrueU denoting the case trained on the same instances of U but using the true labels instead.
2
0.98
Accuracy
Accuracy
0.966
0.94
0.94
0.92
Partask
0.92
Final Aecur mry
Deak Per-iask Accuracy
tabel
L+U30
L+U80
LAU100
LAU260
Buetren
Tiaal Accuracy
Le
Lethuet/1
Label
LeU30
Leus0
L+u120
Leuze0
Letr00
L+U120
LHU240
(a) TF Binary MNIST
(b) DF-CNN Binary MNIST
0.9
0.05
0.85
Accuracy
Accuracy
0.8
0:75
Poee Dereisk Accuracy Final Accurasy
-beak Perclank Accuraty +final Accuracy
0.7
L+Truel200
L-U200
(c) on Binary CIFAR-10
(d) DF-CNN on Binary CIFAR-10
Figure 4: Supervised LML experiments, showing mean and 95% confidence interval. Peak per-task accuracy increases more mako-labeled data is provided as training, showing the benefit of the generated labels.
2.00
100
032
230
0.2
392
1-100
14100
110033
1042
L11243
(a) TF on Binary MNIST
(b) DF-CNN on Binary MNIST
U10200
128
2.64
L1U230
037
Telk:
Taok
(c) TF Binary CIFAR-10
(d) DF-CNN Binary CIFAR-10
Figure Learning Curve Comparisons
15

---

Under review conference paper ICLR 2022
APPENDIX D ADDITIONAL SEMI-SUPERVISED LML EXPERIMENT ANALYSIS
We include learning curves for each of the ssttace-incrrementll semi-supervised LML experiments Figure 6 and each of the class-incremental semi-supervised LMI. experiments in Figure 7 All training curves show final accuracy up to task i defined in Equation [ where task is the current task being trained. For easier tasks with high labeling accuracy, such CIFAR-10, the Mako- enabled semi-supervised LML methods perform similarly to the equivalent supervised fully-labeled task sequence. Otherwise, the Mako-enabled semi-supervised methods approach fully supervised performance.
1.0
0.6
0.8
Accuracy
Accuracy
0.6
0.4
Maka-labeled
0.2
Mako-labeled
Fully-labeled
Fully-labeled
0.2
5
Instance-intrementall batch
Instance-incrementaa batch
(a) 5-way MNIST Experiment
(b) 10-way CIFAR-10 Experiment
Figure 6: iastance-incrmmenta semi-supervised vs fully supervised learning curve comparisons
0.9
08
Accunicy 0.6
0.8
Mako abe ed dicnn
0.6
Makoo labeted hully-abeled
Malio Fobeled
Pullplace doon
rull-lubeledd den
05-
Binary CIFAR-10 Experiment
0.6
M
0.5
0.4
0.0
Mako aseied dicnn
0.3
Makio labeted
Maliio robreted den hully-kabebedd den
Fullylace we thonn
Fully-kaheled
0.2
5-way CIFAR-100 Experiment
Figure 7: Class-ineremental semi-supervised vs fully supervised learning curve comparisons.
We additionally amalyze the class-incremental semi-supervised L.I.. baseline experiments for catas- trophic forgetting, using the catastrophic forgetting ratio defined in Equation 4 Catastrophic forget- ting measures are not available for the baseline methods ORDisCo and DistllMatch they were not reported in the original publications. The results are shown in Table
For further breakdown of results, we include task-specific performance measures for all of the semi-supervised LML experiments in Tables 7 g
9
and T0
16

---

Under review conference paper ICLR 2022
CIFAR-I0
CIFAR-T00
Approach
Forget. Ratio
Approach
Forget. Ratio
Mako-lab. DE-CNN
0.98 + 0.04
Mako-lab. DF-CNN
0.85 0.18
Semi-
Mako-lab, DEN
0.72 + 0.16
Mako-lab. DEN
0.48 = 0.16
Supervised
Mako-lab. TF
0.92 + 0.09
Mako-lab. TF
0.83 0.16
Fully-lab. DF-CNN
0.98 = 0.04
Fully-lab. DF-CNN
0.83 0.19
Supervised
Fully-lab. DEN
0.72 + 0.17
Fully-lab. DEN
0.48 0.16
Fully-lab. TF
0.93 + 0.10
Fully-lab. TF
0.82 0.16
Table 6: Catastrophic forgetting ratios (see Equation for inassinncementll suni--upervieed LML experiments, showing mean + standard deviation
Mako-labeled (Semi-superrised)
Fully-labeled
(Supervised)
Approach
Task
Final Acc.
Forget. Ratio
Final Acc.
Forget. Ratio
0
88.1 1.4
0.953 + 0.015
86.7 + 5.3
0.944 1 0.061
1
75.9 = 4.8
0.942 = 0.063
76.6 + 1.7
0.944 = 0.024
DF-CNN
2
84.4 = 2.0
0.987 = 0.020
84.7 + 1.1
0.989 = 0.014
3
93.2 0.5
0.999 + 0.007
93.5
0.999 + 0.003
9.1
1.000 = 0.000
92.6
1.000 = 0.000
0
8.2
0.874 = 0.078
11.2
0.853 = 0.119
6.7
0.845 0.085
7.0
0.884 = 0.064
TF
2
8.8
0.922 + 0.105
7.44
0.905 + 0.
3
91.3
0.972 + 0.035
9..4
0.987 + 0.009
92.9
1.000 0.000
9.0
1.000 + 0.000
O
0.7
0.559 + 0.013
4.37
0.574 + 0.045
1
3.3
0.664 0.049
5..8
0.678 = 0.118
DEN
58.1 9.7
0.716 0.111
8.7
0.674 + 0.106
3
60.7 +
0.678 + 0.086
59.9 5.4
0.662 + 0.063
4
87.4 + 3.9
1.000 0.000
0.6
1.000 = 0.000
Table 7: Class-inerementa Binary CIFAR-10 10 experiment results broken down by task, showing
mean + standard deviation
Mako-labeled
(Semi-supervised) Forget. Ratio
Fully-labeles Final Acc.
(Supervised)
Approach
Task
Final Acc.
Forget. Ratio
33.9 5.4
0.663 0.104
31.8 6.0
0.583 0.110
35.1 6.5
0.547 0.108
33.8 + 6.3
0.491 = 0.093
2
32.8 4.0
0.556 + 0.067
32.8 + 5.7
0.509 + 0.081
41.6 6.6
0.640 0.101
4.9
0.562 0.084
37.5 1 7.5
0.530 = 0.104
4.2
0.564 = 0.092
45.8 2 5.4
0.776 + 0.080
7.6
0.788 + 0.117
49.3 5.1
0.831 0.083
5.5
0.804 0.065
46.7
0.768 = 0.049
0.789 = 0.066 53.6
8
57.2
0.901 = 0.048
58.2
0.873 = 0.069
DF-CNN
64.2 2.7
0.930 0.038
2.3
0.923 0.035
10
70.1
0.944 0.047
3.0
0.930 = 0.040
11
53.0 + 3.5
0.917 + 0.055
2.5
0.952 + 0.046
12
61.7 1.7
0.967 0.033
6.72
0.963 0.033
13
54.1
0.970 = 0.034
1..4
0.922 = 0.029
14
33.0 +
0.987 + 0.032
1.9
0.977 = 0.039
15
50.0
1.006 - 0.029
5.5
0.971 + 0.035
16
48.1
1.002 = 0.024
5..0
0.967 = 0.021
17
1.2
1.000 = 0.013
56.1
0.994 = 0.018
18
62.7
2.1
1.001 0.015
67.7
0.996 0.009
19
70.5
- 1.8
1.000 + 0.000
77.3 1.9
1.000 - 0.000
Table 8: Class-incremental 5-way CIFAR-100 experiment, DF-CNN, results broken down by task,
showing mean standard deviation
17

---

Under review conference paper ICLR 2022
Mako-labeles
(Semi-supervised)
Fully lable-labeled (Supervised)
Approach
Task
Final Acc.
Forget. Ratio
Final Acc.
Forget. Ratio
36.9 6.7
0.729 0.135
38.8 + 9.1
0.736 + 0.174
1
43.4 + 4.8
0.692 = 0.082
49.9 9.9
0.750 = 0.138
2
36.2 + 7.0
0.639 0.122
37.5 77.
0.617 = 0.116 0.116
56.4 5.2
0.870 0.085
54.0 5.5
0.791 + 0.072
4
40.6
0.585 = 0.193
42.4 10.7
0.600 = 0.151
5
51.0 7.4
0.839 0.124
50.9 13.6
0.760 = 0.206
6
46.3 7.1
0.767 0.126
5.0
0.832 = 0.085
7
9.3
0.804 = 0.152
50.8
0.804 0.081
8
49.6
0.773 0.111
4..0
0.732 + 0.122
9
7.1
0.885 0.166
5.0
0.828 + 0.095
TF
10
17.5:114
0.777 + 0.154
14.0
0.732 + 0.180
11
8.3
0.842 : 0.140
4.0
0.798 + 0.115
12
53.6 5.2
0.881 0.072
5.8
0.905 + 0.064
13
48.4
0.888 = 0.097
8.2
0.847 = 0.126
14
2.0
0.924 0.070
3.9
0.945 = 0.090
15
4.0
0.836 0.134
4..1
0.928 + 0.046
16
4.0
0.956 1:56:000 0.073
4.2
0.936 = 0.093
17
50.0
10.0+4.
0.939 = 0.083
2.4
0.969 = 0.033
18
62.8
0.974 0.032
67.8 2.2
0.962 + 0.023
19
69.6 2.2
1.000 0.000
76.4 2.7
1.000 + 0.000
Table 9: Vlass-nncremental 5-way CIFAR-100 experiment, TF, results broken down by task, showing
mean + standard deviation
Mako-labeled (Semi-supervised)
Fully-labeled
(Supervised)
Approach
Task
Final Acc.
Forget. Ratio
Final Acc.
Forget. Ratio
21.1 2.9
0.433 0.059
19.6 + 1.1
0.388 : 0.033
18.6
0.373 0.061
22.7 4 2.3
0.455 0.092
2
20.3 = 2.2
0.424 = 0.056
2.4
0.477 = 0.097
20.1 1.6
0.381 0.080
3.0
0.361 = 0.058
24.5 4.7
0.420 101
2.5
0.373 + 0.069
21.9 - 1.4
1.4
0.444 + 0.079
2.5
0.423 + 0.047
19.3 1.0
0.405 0.110
19.6
0.350 +: 0.062
2 2.5
0.490 0.171
2.3
0.452 0.095
8
1.3
0.451 = 0.058
11.
0.472 = 0.058
24.0
0.421 0.077
22.0
0.402 = 0.062
DEN
2.8
0.350 0.048
3.1
0.359 0.046
11
22.9
0.502 + 0.058
22.8
0.477 = 0.050
12
21.9
0.408 = 0.073
2.
0.432 = 0.041
3.0
0.506 0.056
3.3
0.494 0.058
14
20.9
+ 1.5
0.689 + 0.082
2.1
0.737 + 0.083
15
22.8
- 3.0
0.497 + 0.068
20.9
0.479 0.051
21.2
0.8
0.516 0.059
2.11
0.527 0.065
17
22.2
+ 1.6
0.429 - 0.033
21.6 1.7
0.434 = 0.038
18
23.1
2.1
0.416 = 0.065
23.0 + 2.0
0.446 - 0.064
65.2
3.5
1.000 0.000
64.1 1 5.2
1.000 0.000
Table 10: Class-incremental 5-way CIFAR-100 experiment, DEN, results broken down by task,
showing + standard deviation


Reviews:
[{"review_id": "nMm2A_VgTk", "reviewer": "Reviewer_E934", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe definition of the problem, the applicable scenarios and the description of the methods are very clear in this paper. The scenario that the author want to solve where labeled training data is expensive to obtain in some lifelong machine learning tasks is also interesting.   Under the premise of this scenario, the author built a processing framework, completing a series of functions such as hyper-parameters search, pseudo labeling and so on. Experimental results also demonstrate its effectiveness. \n\nWeakness:\n\n1）\tI am confused about the definition of some methods in the article. For example, What does the “weak labeler” refer to in the article?  Does the definition of this part correspond to “weak augmentation” in the semi-supervised problem?. I don’t know what is the “weak label” in classification problem. I hope the author can give some description about this.\n\n2）\tThe article spends a lot of time on the definition of various issues and contexts, It's not even possible for me to figure out where the work is the original work of this article. Even in Sec.4, I found that it seemed to be integrating some of the previous working methods to form a system framework, such as data programming, confidence calibration and so on. It makes me feel that this article is more like a technical report rather than a research work. May be I’m misunderstanding about it, So I hope the author can clearly state the contribution of this work.\n\n3）\tThe paper proposed a framework based on semi-supervised continual learning, the main statement of the article also lies in the application of semi-supervised learning. However, some description about semi-supervised is hard to understand. For example, the most important problem in semi-supervised learning is how to obtain accurate pseudo labels, but I don’t see how the author solves this problem, only by using a previous method data programming? Or does the article just introduce the semi supervised learning into the some scenarios of continuous learning?  If it is the latter, there are many more mature semi supervised algorithms, can you explain data programming is used?\n", "summary_of_the_paper": "This paper propose a wrapper tool that mounts on top of supervised Lifelong Machine Learning (LML) frameworks, leveraging a well-known method data programming. The contributions of this paper can be summarized in three aspects. 1)  Adapting automatic label generation by semi-supervised learning/data programming to LML in some special scenario. 2) Implementing a LML wrapper that can accomplish some tasks under some restrictions. 3) Through detailed experimental results prove the superiority of its method.", "summary_of_the_review": "This paper implemented a semi-supervised continual learning framework that can be mounted on top of any existing LML tool. It integrates some methods to solve the scenario where labeled training data is expensive. The problems and application scenarios proposed in this paper are of practical significance. However, in my opinion, the engineering significance of the paper is greater than its academic significance, it lacks of novelty in the view of research.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jXLNhQAAgfF", "reviewer": "Reviewer_Fba4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n*  The paper is clearly written and the topic is relavent.\n*  Experiments on  several different scenarios \n\n\nWeakness:\n* The technical novelty of proposed methods is very limited. The method is simply the combination of existing methods. For example, the classifer is a collection of weak labeling functions with bootstrapping, e.g., Snuba. In addition, the data programming techniques are not novel as well. It is very similar to the self-training [1], which predicts the labels of unlabeled data during the training process.  The self-training techniques are widely adopted in semi-supervised learning. It would be better to discuss in related work for the difference between the proposed method and self-training and compare to state of art self-training methods in experiment.\n\n* There is lack of insights on the proposed method for mitigate catastrophic forgetting, which is main focus of this paper. \n\n* For the evaluation metrics, the author proposed  catastrophic forgetting ratio up to task i for evaluating the forgetting. However, for most continual learning works, they commonly adopt backward transfer [2] to evaluate the forgetting issue. It would be bettter to show the commonly adopted metrics for a direct comprions to related works. \n\n* For the experiment,  improvement on forgetting seems quite marginal, although the peak per-task accuracy is improved. Also, seems that there are many hyperparameters in the proposed method. It would be great to show the sensitivity of the hyperparameters.\n\n\nReference:\n[1] Uncertainty-aware Self-training for Text Classification with Few Labels. Subhabrata Mukherjee and Ahmed Hassan Awadallah. NeurIPS 2020\n[2] On Tiny Episodic Memories in Continual Learning, Arslan Chaudhry, et al. https://arxiv.org/abs/1902.10486", "summary_of_the_paper": "This paper proposes  data programming method, named Mako,  for semi-supervised continual learning. Mako automatically generates labels for unlabeled data with a set of weak labeling functions, each of the functions is trained on subset of training set with bootstrapping. Experimental on several datasets demonstrate the effectiveness of the proposed methods. ", "summary_of_the_review": "The paper is clearly written, but the technical novelty is quite limited. The analysis and experiments also need to be significantly improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fyaGl2-YeBk", "reviewer": "Reviewer_XNuq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n[1] The lifelong learning without catastrophic forgetting is a critical and interesting issue, the method proposed in this paper is shown to be effective in addressing this issue on some small-scale benchmarks, such as MINIST, CIFAR-10 and CIFAR-100. \n\n[2] The authors conducted comprehensive experiments on LML and proposed several evaluation metrics to quantitatively measure the ability of MAKO in eliminating catastrophic forgetting, improving peak per-task accuracy and final accuracy. \n\n[3] Exploring the lifelong learning using a limited amount of data, i.e. semi-supervised LML, is a new setting and of great interest. \n\n[4] The paper is well-organized. \n\nCons:\n\n[1] Although authors conducted comprehensive experiments on MNIST and CIFAR, the experiments on larger benchmarks may be necessary to show the improvements obtained through using MAKO. \n\n[2] Most of the components of MAKO are based on the prior arts proposed in previous literatures, for example, the method for labeling training data with weak supervision is based on Snuba, therefore, the technical contribution of this paper may be limited. \n\n[3] If the method is resistance to catastrophic forgetting, why the forgetting ratio is always lower than baseline method? Does it mean that there is less positive knowledge transfer from the later tasks to the earlier ones?\n\n[4] In table 1, the more unlabeled data is used during the training process, the worse the final accuracy is, which is quite confusing. The final accuracy and forget ratio are both lower than the baseline method when 40 unlabeled instances are used. If the proposed method is truly effective and is able to avoid catastrophic forgetting, why the performance is lower?\n\nThis metric is less than 1 if the LML model loses its performance on the earlier tasks, and it is greater than 1 if there is positive knowledge transfer from the later tasks to the earlier ones.\n", "summary_of_the_paper": "In contrast to previous works on lifelong machine learning (LML) that put their focus on the supervised learning settings, this paper concentrates on the scenario that only a limited amount of data is available. The proposed method MAKO is mounted on the top of supervised LML model, without introducing additional knowledge based overhead, for better leveraging the unlabeled data. Labeling new data can be realized by using the data programming method which is supervised by the labeled data. The target of this paper is to design a SSL LML framework that minimizes the performance between using partially labeled data, and the upper-bound performance using fully labeled data. Several experiments on standard image classification data sets including MNIST, CIFAR-10 and CIFAR-100 are used to evaluate the the effectiveness of MAKO. ", "summary_of_the_review": "I like the idea of semi-supervised lifelong machine learning and I agree with the authors that previous works on LML only consider the fully-supervised lifelong learning, which require the use of all labeled data and is quite expensive. However, the technical contribution of this paper may be limited. Considering that the performance of the proposed method is not that satisfying either, my current rate of this paper is: \"3: reject, not good enough\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T1hB22vshV3", "reviewer": "Reviewer_ECpE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Pros\nContinual learning with fewer annotated data is an important research task. The proposed framework uses data programming techniques to generate pseudo labels for unlabeled data and makes the task suitable for various fully supervised LML methods. Compared with fully supervised methods, the experimental results seem promising.\n\n#### Cons\n1. Since most benefits come from the data programing techniques (Snuba in this paper), I would like to see more analysis on the annotation process, performance of down-stream tasks is only one of the metrics, e.g.,\n\ta. It is unclear that how many labeling functions are generated in each task? Precision and coverage of each LF in the committed set are more meaningful than labeling accuracy in Fig.3. \n\tb. One key fact of labeling functions in Snuba is diversity. Although the authors trained labeling functions on different sampled sets, the laziness of the neural network makes it prefer to output similar results. How different are the results obtained from different labeling functions? It would be nice to see a numeric comparison.  \n\tc. It seems a sophisticated process to annotate unlabeled data. It would be better to provide/plot average iterations and computational times for [hyperparameter search <--> data programming] for each task. \n\n2. I’m concerned about the module in Sec 4.1 and Sec 4.2:\n\ta. I am not sure whether I grasped the main points of the automatic hyperparameter search and data programming. Does the automatic hyperparameter search module first search the best hyperparameter as a fixed heuristic pattern, then the DP module generates different LF with the same hyperparameters but different sampled subsets?\t\n\tb. How many heuristics are generated for pruning. \nc. Some important technical details are not clear in this paper, e.g., the hyperparameters in Snuba.\n\n#### Misc\n1. Just for curiosity, why do both Mako-labeled DEN and fully-labeled DEN achieve worse performance than the baselines?\n2. I have a doubt about the experimental results of DEN in the last row of Table 7 and Table 10: What’s the reason for the performance booming in the last task?\n\n#### Minor points\n“purposed” in the 1st line of the second paragraph in Sec.2 should be “proposed”?\n", "summary_of_the_paper": "This paper presents an interesting idea of using data programming techniques to enable continual semi-supervised learning with limited labeled data. It proposes a stage-wise pipeline where probabilistic pseudo labels are first produced by a Snuba based Data Programming framework, then calibrates them by the temperature scaling, and finally inputs into the mounted Lifelong Machine Learning (LML) tools. Experiments show that the proposed framework achieves similar performance to fully supervised methods.", "summary_of_the_review": "In conclusion, despite the successful improvement of the LML task, I still have lots of concerns about the labeling results, which is the key to improvements. I would be more convinced if more analysis of labeling results could be done in this paper. For the current version, I recommend a negative score as my initial rating. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes "Mako", which enables continual learning when only a limited amount of labeled data is available (along with a good deal of unlabeled data). Reviewers shared concerns about difficulty in understanding which components of the proposed system were novel, especially given that the most important components seemed to be proposed in past work. Reviewers also had difficulty getting insight on which parts of the system were most useful, and further requested additional experiments on harder benchmarks. There consensus was therefore to reject the paper.
Decision: Reject