Mako: Semi-supervised continual learning with minimal labeled data via data programming

1 INTRODUCTION . Since 1995 , researchers have been actively seeking machine learning ( ML ) frameworks that can better mimic the human learning process by retaining memories from past experiences ( Thrun & Mitchell , 1995 ; Liu , 2017 ) . Hence , in contrast to traditional isolated ML where knowledge is never accumulated , the concept of lifelong machine learning ( LML ) is defined as training ML models for continual learning over task sequences , with a knowledge base to store information that could be helpful in the future . Under this definition , LML can be seen as continual transfer learning ( Pan & Yang , 2015 ) with memory or sequential multi-task learning ( Caruana , 1997 ) . Nevertheless , at the level of individual tasks , the current state-of-the-art of LML still remains largely supervised . A challenge that has rarely been mentioned is that labels can still be expensive and rarely available for each task , such that supervised training can hardly provide acceptable performance . For LML , performance is not only measured at task level , but also for the entire lifetime , with attributes such as resistance to catastrophic forgetting and space-efficiency of knowledge base . We take a more generic approach to semi-supervised lifelong learning and develop a technique that can wrap around any existing continual learning algorithm . Specifically , we propose to use data programming ( Ratner et al. , 2016b ) to label new data under the supervision of a set of weak labeling functions trained from a limited number of labeled data points . Data programming has proven successful for isolated learning , and so this is the first work integrating it with lifelong learning . Our approach , called Mako , sits atop an existing lifelong learner to turn it into a semi-supervised learner . For each task , Mako takes as input a small labeled dataset plus an unlabeled dataset with no restriction on size , generates labels for the unlabeled data and then updates the lifelong learner supervised by both sets . This approach builds upon the widely used Snuba algorithm ( Varma & Ré , 2016 ) , and we show that its theoretical guarantees still hold in the lifelong setting . Mako ’ s procedure leverages automatic hyperparameter tuning , data programming and confidence calibration , in order to adapt automatic labeling to constantly varying LML tasks . We show that Mako produces adequately high performance , in terms of both task accuracy and prevention of catastrophic forgetting , that approaches that of training on fully labeled data without costing extra knowledge base space overhead . We extensively evaluated Mako on various partially labeled LML task sequences generated from commonly used datasets including MNIST , CIFAR-10 and CIFAR-100 ( LeCun & Cortes , 2010 ; Krizhevsky , 2009 ) , while mounting on a diversity of supervised LML frameworks such as Deconvolutional Factorized CNN ( DF-CNN ) , Dynamic Expandable Network ( DEN ) and Tensor Factorization ( TF ) ( Lee et al. , 2019 ; Yoon et al. , 2018 ; Bulat et al. , 2020 ) . We compared the results to supervised learning on fully labeled data , as well as the same partially labeled data with existing semi-supervised LML tools : CNNL , ORDisCo and DistillMatch ( Baucum et al. , 2017 ; Wang et al. , 2021 ; Smith et al. , 2021 ) . Empirically , we show that the performance of LML methods improves as giving more training data even with Mako labels , achieving at least 97 % performance relative to LML methods trained on ground-truth labels and being able to beat existing semi-supervised LML tools by approximately 0.25 higher accuracy . Contributions Summary : 1 . Adapting automatic label generation by semi-supervised learning/data programming to LML in the scenario where labeled training data is expensive to obtain compared to unlabeled for all tasks . 2 . Implementing a LML wrapper , namely Mako , for the scenario of expensive labels , such that , when given partially labeled data of limited size and compared to the current supervised state-ofthe-art with fully labeled data , each LML task sequence ( 1 ) requires significantly fewer labeled data to achieve high accuracy on individual tasks , ( 2 ) maintains similar resistance to catastrophic forgetting and ( 3 ) does not cost extra knowledge base storage . 3 . Extensive experiments on lifelong binary and multiclass image classification on 3 commonly used datasets , including comparison to supervised and semi-supervised LML baselines , accomplishing very close performance to supervised LML on fully labeled data and significantly outperforming baseline semi-supervised LML frameworks . 2 RELATED WORK . Lifelong machine learning ( LML ) is a concept of continual learning among a sequence of tasks . Specifically , given multiple machine learning tasks arriving continuously , the LML framework accumulates the knowledge of previous tasks and utilizes it for future ones ( Chen & Liu , 2016 ; Ruvolo & Eaton , 2013b ) . The method of retaining and adopting the knowledge base varies depending on the characteristics of tasks and learning models . For instance , a latent matrix is used to keep prototypical weights for linear models ( Kumar & Daume , 2012 ; Ruvolo & Eaton , 2013a ) , and for deep neural networks , either weights of layers ( Yoon et al. , 2018 ; Lu et al. , 2017 ; Bulat et al. , 2020 ; Lee et al. , 2019 ) or extracted features of layers ( Rusu et al. , 2016 ; Misra et al. , 2016 ; Gao et al. , 2019 ; Schwarz et al. , 2018 ) are considered as the knowledge base to transfer . Besides research on models to transfer knowledge across tasks , the LML community has also been working on task order sorting for higher overall performance ( Sun et al. , 2018 ; Ruvolo & Eaton , 2013b ) . Nevertheless , all these works assume that each individual task training is supervised with ground-truth labels for the entire training set . In many real-life ML tasks , obtaining labels has been and continues to be expensive ( Settles , 2009 ) . Only a few works have addressed this issue on LML in very specific domains , such as continual sentimental analysis with words partially labeled ( Wang et al. , 2016 ) and reinforcement learning in a sequence of state spaces without explicit rewards ( Finn et al. , 2017 ) . Different from previous work , Mako enables high performance in continual learning on generic ML tasks given partially labeled data , with the cardinality of labels restricted at a small number . To tackle expensive labels in ML , researchers have purposed various methods such as active learning ( Settles , 2009 ) or semi-supervised learning ( Olivier et al. , 2006 ) , which is an ML framework that takes both labeled and unlabeled data as inputs . The unlabeled data can directly assist the training , or can serve as a target data set yet to be labeled . In recent years , with the rising enthusiasm and need in deep learning , various methods have been used to assist semi-supervised learning with DNN models ( Rasmus et al. , 2015 ; Laine & Aila , 2016 ; Tarvainen & Valpola , 2017 ) . One branch of semi-supervised learning , namely data programming or weak supervision ( Ratner et al. , 2016b ) , focuses on auto-generation of labels on an unlabeled dataset with the help of a small labeled dataset . Targeting this problem , researchers leverage weak labeling functions that each can label with an accuracy slightly higher than a coin flip . Then , ensembling algorithms are able to bag these weak labelers and produce highly accurate final labels . In other words , the weak labels supervise the training of the final ensembled generative model . There exists different implementations of such generative model training such as Snorkel ( Ratner et al. , 2016a ) . Semi-supervised learning is an underexplored problem in the LML setting ; all the works mentioned above have addressed problems in isolated ML only . Early work in semi-supervised incremental learning , in which all tasks are trained simultaneously as batches of unlabeled data are incrementally observed by a classifier , has identified challenges pertaining to selection of the label generating model architecture and confidence calibration of the generated labels ( Baucum et al. , 2017 ) , processes which Mako undertakes autonomously through automatic search for weak labelers before ensembling , and confidence calibration after labels are produced . Pioneering work in semi-supervised LML requires training of additional models for out-of-distribution detection ( Smith et al. , 2021 ) or generative replay ( Wang et al. , 2021 ) . Conversely , by focusing on label generation through data programming , our approach achieves similar behavior through a pre-processing workflow that does not impose additional constraints on the LML approach or require additional knowledge base storage . For data programming tools , it has been a long standing issue to obtain weak labeling functions as input . One approach is manually designing the weak labelers by domain experts ( Ratner et al. , 2016b ; a ) , while other researchers seek automatic labeler generation . For instance , Snuba ( Varma & Ré , 2016 ) outputs weak supervisors with an accuracy guarantee on the final ensembled data programming model . Thus far , Snuba has been evaluated on various isolated ML tasks , with weak labelers in form of k-nearest-neighbors , decision stumps and logistic regressors , and has been shown to ace in natural language processing and image classification . It also succeeds in more complex applications such as classifying bone tumors in medical domain . In our work , we build a weak labeling function generation module in Mako based on Snuba , aiming to support a larger variety of labeler models for LML , while maintaining the theoretical accuracy guarantee of generated labels . 3 PROBLEM FORMULATION . LML : Given a sequence of ML tasks : T1 , T2 , . . . , Tm , where m is an unbounded integer , each task Ti has an underlying data space Xi and label space Yi . Ti has ni , train labeled training data ( Xi , train , Yi , train ) ∼ Di , such that Xi , train ∈ X ni , train i and Yi , train ∈ Y ni , train i , following some task-specific i.i.d . distributionDi . The task also has ni , test testing data ( Xi , test , Yi , test ) drawn from the same distribution . Traditionally , an LML model encounters the tasks in order . At Ti , it trains on the full set of ( Xi , train , Yi , train ) with supervised learning and obtain model fi : ⋃i j=1 Xi 7→ ⋃i j=1 Yi . Immediately after this training , it predicts the j-th task with Ŷij = fi ( Xj , test ) for each j ≤ i . There are two objectives : 1 . High test accuracy . Formally , denote testing accuracy of task Tj right after the training of task Ti on ( Xi , train , Yi , train ) as aij = ∑nj , test k=1 1 ( ŷk = yk ) nj , test ( 1 ) with ŷk and yk the k-th label in Ŷij and Yi , test , respectively . One LML objective in the current state-of-the-art is to maximize the peak per-task accuracy up to task i defined as ãi = 1 i i∑ j=1 ajj ( 2 ) Peak per task accuracy is defined on all tasks i = 1 , . . . , m , and shows the LML performance on the current training task on the basis of the accumulated knowledge . 2 . Low catastrophic forgetting . A critical aspect of LML is to prevent catastrophic forgetting of previous tasks . The resistance to catastrophic forgetting can be quantified by the average testing accuracy of all tasks encountered so far . For each task Ti , denote a second objective , final accuracy up to task i as āi = 1 i i∑ j=1 aij ( 3 ) The second objective is to maximize āi for all i = 1 , . . . , m.The gap between peak per-task accuracy and final accuracy expresses how much knowledge of previous tasks LML method can preserve through the streams of tasks . Therefore , we define a third metric to be maximized , catastrophic forgetting ratio up to task i , as ci = 1 i i∑ j=1 aij ajj ( 4 ) This metric is less than 1 if the LML model loses its performance on the earlier tasks , and it is greater than 1 if there is positive knowledge transfer from the later tasks to the earlier ones . Semi-supervised LML with data programming : The key challenge we want to address is the cost of labeling data in all LML tasks . That is , the number of available labeled training data is small . Therefore , we would like to split ( Xi , train , Yi , train ) in the traditional LML problem into disjoint ( Xi , L , Yi , L , Xi , U ) , where ( Xi , L , Yi , L ) is a set of labeled training data with restriction on size and Xi , U is a set of unlabeled data with no such restriction . In other words , |Xi , L| is small , e.g . |Xi , L| ≤ 150 for a 10-way image classification problem . Apparently , the LML model is now trained with semi-supervised learning . For task Ti , we would like to leverage data programming on ( Xi , L , Yi , L , Xi , U ) to first train a generative model πi : Xi 7→ Yi . This model automatically labels Xi , U with Y πi , U = πi ( Xi , U ) . Then , the LML framework will be trained on ( Xi , L , Yi , L , Xi , U , Y πi , U ) . Denote the LML model immediately after this training as fπi : ⋃i j=1 Xi 7→ ⋃i j=1 Yi , and Ŷ πij = fπi ( Xj , test ) for each task j ≤ i . In this scenario , we redefine testing accuracy in Equation 1 as aπij = ∑nj , test k=1 1 ( ŷ π k = yk ) nj , test ( 5 ) where the only difference is that ŷπk is the k-th label in Ŷ π ij instead of Ŷij . From Equation 5 , we can then redefine metrics in Equations 2 , 3 and 4 as ãπi = 1 i i∑ j=1 aπjj ( 6 ) ā π i = 1 i i∑ j=1 aπij ( 7 ) c π i = 1 i i∑ j=1 aπij aπjj ( 8 ) Our research problem is to design a semi-supervised LML framework leveraging data programming that minimizes the performance between using partially labeled data , ( Xi , L , Yi , L , Xi , U , Y πi , U ) , and the upper-bound performance using fully labeled data , ( Xi , L , Yi , L , Xi , U , Yi , U ) , in terms of these three metrics while imposing no additional knowledge base storage overhead in the LML framework .

Summary: This paper propose a wrapper tool that mounts on top of supervised Lifelong Machine Learning (LML) frameworks, leveraging a well-known method data programming. The contributions of this paper can be summarized in three aspects. 1)  Adapting automatic label generation by semi-supervised learning/data programming to LML in some special scenario. 2) Implementing a LML wrapper that can accomplish some tasks under some restrictions. 3) Through detailed experimental results prove the superiority of its method.