Mako: Semi-supervised continual learning with minimal labeled data via data programming

1 INTRODUCTION . Since 1995 , researchers have been actively seeking machine learning ( ML ) frameworks that can better mimic the human learning process by retaining memories from past experiences ( Thrun & Mitchell , 1995 ; Liu , 2017 ) . Hence , in contrast to traditional isolated ML where knowledge is never accumulated , the concept of lifelong machine learning ( LML ) is defined as training ML models for continual learning over task sequences , with a knowledge base to store information that could be helpful in the future . Under this definition , LML can be seen as continual transfer learning ( Pan & Yang , 2015 ) with memory or sequential multi-task learning ( Caruana , 1997 ) . Nevertheless , at the level of individual tasks , the current state-of-the-art of LML still remains largely supervised . A challenge that has rarely been mentioned is that labels can still be expensive and rarely available for each task , such that supervised training can hardly provide acceptable performance . For LML , performance is not only measured at task level , but also for the entire lifetime , with attributes such as resistance to catastrophic forgetting and space-efficiency of knowledge base . We take a more generic approach to semi-supervised lifelong learning and develop a technique that can wrap around any existing continual learning algorithm . Specifically , we propose to use data programming ( Ratner et al. , 2016b ) to label new data under the supervision of a set of weak labeling functions trained from a limited number of labeled data points . Data programming has proven successful for isolated learning , and so this is the first work integrating it with lifelong learning . Our approach , called Mako , sits atop an existing lifelong learner to turn it into a semi-supervised learner . For each task , Mako takes as input a small labeled dataset plus an unlabeled dataset with no restriction on size , generates labels for the unlabeled data and then updates the lifelong learner supervised by both sets . This approach builds upon the widely used Snuba algorithm ( Varma & Ré , 2016 ) , and we show that its theoretical guarantees still hold in the lifelong setting . Mako ’ s procedure leverages automatic hyperparameter tuning , data programming and confidence calibration , in order to adapt automatic labeling to constantly varying LML tasks . We show that Mako produces adequately high performance , in terms of both task accuracy and prevention of catastrophic forgetting , that approaches that of training on fully labeled data without costing extra knowledge base space overhead . We extensively evaluated Mako on various partially labeled LML task sequences generated from commonly used datasets including MNIST , CIFAR-10 and CIFAR-100 ( LeCun & Cortes , 2010 ; Krizhevsky , 2009 ) , while mounting on a diversity of supervised LML frameworks such as Deconvolutional Factorized CNN ( DF-CNN ) , Dynamic Expandable Network ( DEN ) and Tensor Factorization ( TF ) ( Lee et al. , 2019 ; Yoon et al. , 2018 ; Bulat et al. , 2020 ) . We compared the results to supervised learning on fully labeled data , as well as the same partially labeled data with existing semi-supervised LML tools : CNNL , ORDisCo and DistillMatch ( Baucum et al. , 2017 ; Wang et al. , 2021 ; Smith et al. , 2021 ) . Empirically , we show that the performance of LML methods improves as giving more training data even with Mako labels , achieving at least 97 % performance relative to LML methods trained on ground-truth labels and being able to beat existing semi-supervised LML tools by approximately 0.25 higher accuracy . Contributions Summary : 1 . Adapting automatic label generation by semi-supervised learning/data programming to LML in the scenario where labeled training data is expensive to obtain compared to unlabeled for all tasks . 2 . Implementing a LML wrapper , namely Mako , for the scenario of expensive labels , such that , when given partially labeled data of limited size and compared to the current supervised state-ofthe-art with fully labeled data , each LML task sequence ( 1 ) requires significantly fewer labeled data to achieve high accuracy on individual tasks , ( 2 ) maintains similar resistance to catastrophic forgetting and ( 3 ) does not cost extra knowledge base storage . 3 . Extensive experiments on lifelong binary and multiclass image classification on 3 commonly used datasets , including comparison to supervised and semi-supervised LML baselines , accomplishing very close performance to supervised LML on fully labeled data and significantly outperforming baseline semi-supervised LML frameworks . 2 RELATED WORK . Lifelong machine learning ( LML ) is a concept of continual learning among a sequence of tasks . Specifically , given multiple machine learning tasks arriving continuously , the LML framework accumulates the knowledge of previous tasks and utilizes it for future ones ( Chen & Liu , 2016 ; Ruvolo & Eaton , 2013b ) . The method of retaining and adopting the knowledge base varies depending on the characteristics of tasks and learning models . For instance , a latent matrix is used to keep prototypical weights for linear models ( Kumar & Daume , 2012 ; Ruvolo & Eaton , 2013a ) , and for deep neural networks , either weights of layers ( Yoon et al. , 2018 ; Lu et al. , 2017 ; Bulat et al. , 2020 ; Lee et al. , 2019 ) or extracted features of layers ( Rusu et al. , 2016 ; Misra et al. , 2016 ; Gao et al. , 2019 ; Schwarz et al. , 2018 ) are considered as the knowledge base to transfer . Besides research on models to transfer knowledge across tasks , the LML community has also been working on task order sorting for higher overall performance ( Sun et al. , 2018 ; Ruvolo & Eaton , 2013b ) . Nevertheless , all these works assume that each individual task training is supervised with ground-truth labels for the entire training set . In many real-life ML tasks , obtaining labels has been and continues to be expensive ( Settles , 2009 ) . Only a few works have addressed this issue on LML in very specific domains , such as continual sentimental analysis with words partially labeled ( Wang et al. , 2016 ) and reinforcement learning in a sequence of state spaces without explicit rewards ( Finn et al. , 2017 ) . Different from previous work , Mako enables high performance in continual learning on generic ML tasks given partially labeled data , with the cardinality of labels restricted at a small number . To tackle expensive labels in ML , researchers have purposed various methods such as active learning ( Settles , 2009 ) or semi-supervised learning ( Olivier et al. , 2006 ) , which is an ML framework that takes both labeled and unlabeled data as inputs . The unlabeled data can directly assist the training , or can serve as a target data set yet to be labeled . In recent years , with the rising enthusiasm and need in deep learning , various methods have been used to assist semi-supervised learning with DNN models ( Rasmus et al. , 2015 ; Laine & Aila , 2016 ; Tarvainen & Valpola , 2017 ) . One branch of semi-supervised learning , namely data programming or weak supervision ( Ratner et al. , 2016b ) , focuses on auto-generation of labels on an unlabeled dataset with the help of a small labeled dataset . Targeting this problem , researchers leverage weak labeling functions that each can label with an accuracy slightly higher than a coin flip . Then , ensembling algorithms are able to bag these weak labelers and produce highly accurate final labels . In other words , the weak labels supervise the training of the final ensembled generative model . There exists different implementations of such generative model training such as Snorkel ( Ratner et al. , 2016a ) . Semi-supervised learning is an underexplored problem in the LML setting ; all the works mentioned above have addressed problems in isolated ML only . Early work in semi-supervised incremental learning , in which all tasks are trained simultaneously as batches of unlabeled data are incrementally observed by a classifier , has identified challenges pertaining to selection of the label generating model architecture and confidence calibration of the generated labels ( Baucum et al. , 2017 ) , processes which Mako undertakes autonomously through automatic search for weak labelers before ensembling , and confidence calibration after labels are produced . Pioneering work in semi-supervised LML requires training of additional models for out-of-distribution detection ( Smith et al. , 2021 ) or generative replay ( Wang et al. , 2021 ) . Conversely , by focusing on label generation through data programming , our approach achieves similar behavior through a pre-processing workflow that does not impose additional constraints on the LML approach or require additional knowledge base storage . For data programming tools , it has been a long standing issue to obtain weak labeling functions as input . One approach is manually designing the weak labelers by domain experts ( Ratner et al. , 2016b ; a ) , while other researchers seek automatic labeler generation . For instance , Snuba ( Varma & Ré , 2016 ) outputs weak supervisors with an accuracy guarantee on the final ensembled data programming model . Thus far , Snuba has been evaluated on various isolated ML tasks , with weak labelers in form of k-nearest-neighbors , decision stumps and logistic regressors , and has been shown to ace in natural language processing and image classification . It also succeeds in more complex applications such as classifying bone tumors in medical domain . In our work , we build a weak labeling function generation module in Mako based on Snuba , aiming to support a larger variety of labeler models for LML , while maintaining the theoretical accuracy guarantee of generated labels . 3 PROBLEM FORMULATION . LML : Given a sequence of ML tasks : T1 , T2 , . . . , Tm , where m is an unbounded integer , each task Ti has an underlying data space Xi and label space Yi . Ti has ni , train labeled training data ( Xi , train , Yi , train ) ∼ Di , such that Xi , train ∈ X ni , train i and Yi , train ∈ Y ni , train i , following some task-specific i.i.d . distributionDi . The task also has ni , test testing data ( Xi , test , Yi , test ) drawn from the same distribution . Traditionally , an LML model encounters the tasks in order . At Ti , it trains on the full set of ( Xi , train , Yi , train ) with supervised learning and obtain model fi : ⋃i j=1 Xi 7→ ⋃i j=1 Yi . Immediately after this training , it predicts the j-th task with Ŷij = fi ( Xj , test ) for each j ≤ i . There are two objectives : 1 . High test accuracy . Formally , denote testing accuracy of task Tj right after the training of task Ti on ( Xi , train , Yi , train ) as aij = ∑nj , test k=1 1 ( ŷk = yk ) nj , test ( 1 ) with ŷk and yk the k-th label in Ŷij and Yi , test , respectively . One LML objective in the current state-of-the-art is to maximize the peak per-task accuracy up to task i defined as ãi = 1 i i∑ j=1 ajj ( 2 ) Peak per task accuracy is defined on all tasks i = 1 , . . . , m , and shows the LML performance on the current training task on the basis of the accumulated knowledge . 2 . Low catastrophic forgetting . A critical aspect of LML is to prevent catastrophic forgetting of previous tasks . The resistance to catastrophic forgetting can be quantified by the average testing accuracy of all tasks encountered so far . For each task Ti , denote a second objective , final accuracy up to task i as āi = 1 i i∑ j=1 aij ( 3 ) The second objective is to maximize āi for all i = 1 , . . . , m.The gap between peak per-task accuracy and final accuracy expresses how much knowledge of previous tasks LML method can preserve through the streams of tasks . Therefore , we define a third metric to be maximized , catastrophic forgetting ratio up to task i , as ci = 1 i i∑ j=1 aij ajj ( 4 ) This metric is less than 1 if the LML model loses its performance on the earlier tasks , and it is greater than 1 if there is positive knowledge transfer from the later tasks to the earlier ones . Semi-supervised LML with data programming : The key challenge we want to address is the cost of labeling data in all LML tasks . That is , the number of available labeled training data is small . Therefore , we would like to split ( Xi , train , Yi , train ) in the traditional LML problem into disjoint ( Xi , L , Yi , L , Xi , U ) , where ( Xi , L , Yi , L ) is a set of labeled training data with restriction on size and Xi , U is a set of unlabeled data with no such restriction . In other words , |Xi , L| is small , e.g . |Xi , L| ≤ 150 for a 10-way image classification problem . Apparently , the LML model is now trained with semi-supervised learning . For task Ti , we would like to leverage data programming on ( Xi , L , Yi , L , Xi , U ) to first train a generative model πi : Xi 7→ Yi . This model automatically labels Xi , U with Y πi , U = πi ( Xi , U ) . Then , the LML framework will be trained on ( Xi , L , Yi , L , Xi , U , Y πi , U ) . Denote the LML model immediately after this training as fπi : ⋃i j=1 Xi 7→ ⋃i j=1 Yi , and Ŷ πij = fπi ( Xj , test ) for each task j ≤ i . In this scenario , we redefine testing accuracy in Equation 1 as aπij = ∑nj , test k=1 1 ( ŷ π k = yk ) nj , test ( 5 ) where the only difference is that ŷπk is the k-th label in Ŷ π ij instead of Ŷij . From Equation 5 , we can then redefine metrics in Equations 2 , 3 and 4 as ãπi = 1 i i∑ j=1 aπjj ( 6 ) ā π i = 1 i i∑ j=1 aπij ( 7 ) c π i = 1 i i∑ j=1 aπij aπjj ( 8 ) Our research problem is to design a semi-supervised LML framework leveraging data programming that minimizes the performance between using partially labeled data , ( Xi , L , Yi , L , Xi , U , Y πi , U ) , and the upper-bound performance using fully labeled data , ( Xi , L , Yi , L , Xi , U , Yi , U ) , in terms of these three metrics while imposing no additional knowledge base storage overhead in the LML framework .

Summary: This paper presents an interesting idea of using data programming techniques to enable continual semi-supervised learning with limited labeled data. It proposes a stage-wise pipeline where probabilistic pseudo labels are first produced by a Snuba based Data Programming framework, then calibrates them by the temperature scaling, and finally inputs into the mounted Lifelong Machine Learning (LML) tools. Experiments show that the proposed framework achieves similar performance to fully supervised methods.