Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning

1 INTRODUCTION . The lottery ticket hypothesis ( LTH ) ( Frankle & Carbin , 2019 ) suggests the existence of an extremely sparse sub-network , within an overparameterized dense neural network , that can reach similar performance as the dense network when trained in isolation with proper initialization . Such a subnetwork together with the used initialization is called a winning ticket ( Frankle & Carbin , 2019 ) . The original LTH studies the sparse pattern of neural networks with a single task ( classification ) , leaving the question of generalization across multiple tasks open . Following that , a few works ( Morcos et al. , 2019 ; Mehta , 2019 ) have explored LTH in transfer learning . They study the transferability of a winning ticket found in a source task to another target task . This provides insights on one-shot transferability of LTH . In parallel , lifelong learning not only suffers from notorious catastrophic forgetting over sequentially arriving tasks but also often comes at the price of increasing model capacity . With those in mind , we ask a much more ambitious question : Does LTH hold in the setting of lifelong learning when different tasks arrive sequentially ? Intuitively , a desirable “ ticket ” sub-network in lifelong learning ( McCloskey & Cohen , 1989 ; Parisi et al. , 2019 ) needs to be : 1 ) independently trainable , same as the original LTH ; 2 ) trained to perform ∗Equal Contribution . 1 competitively to the dense lifelong model , including both maintaining the performance of previous tasks , and quickly achieving good generalization at newly added tasks ; 3 ) found online , as the tasks sequentially arrive without any pre-assumed order . We define such a sub-network with its initialization as a lifelong lottery ticket . This paper seeks to locate the lifelong ticket in class-incremental learning ( CIL ) ( Wang et al. , 2017 ; Rosenfeld & Tsotsos , 2018 ; Kemker & Kanan , 2017 ; Li & Hoiem , 2017 ; Belouadah & Popescu , 2019 ; 2020 ) , a popular , realistic and challenging setting of lifelong learning . A natural idea to extend the original LTH is to introduce sequential pruning : we continually prune the dense network until the desired sparsity level , as new tasks are incrementally added . However , we show that the direct application of the iterative magnitude pruning ( IMP ) used in LTH fails in the scenario of CIL since the pruning schedule becomes critical when tasks arrive sequentially . To circumvent this challenge , we generalize IMP to incorporate a curriculum pruning schedule . We term this technique top-down lifelong pruning . When the total number of tasks is pre-known and small , then with some “ lottery ” initialization ( achieved by rewinding ( Frankle et al. , 2019 ) or similar ) , we find that the pruned sparse ticket can be re-trained to similar performance as the dense network . However , if the number of tasks keeps increasing , the above ticket will soon witness performance collapse as its limited capacity can not afford the over-pruning . The limitation of top-down lifelong pruning reminds us of two unique dilemmas that might challenge the validity of lifelong tickets . i ) Greedy weight pruning v.s . all tasks ’ performance : While the sequential pruning has to be performed online , its greedy nature inevitably biases against later arriving tasks , as earlier tasks apparently will contribute to shaping the ticket more ( and might even use up the sparsity budget ) . ii ) Catastrophic forgetting v.s . small ticket size : To overcome the notorious catastrophic forgetting ( McCloskey & Cohen , 1989 ; Tishby & Zaslavsky , 2015 ) , many lifelong learning models have to frequently consolidate weights to carefully re-assign the model capacity ( Zhang et al. , 2020 ) or even grow model size as tasks come in ( Wang et al. , 2017 ) . Those seem to contradict our goal of pruning by seeing more tasks . To address the above two limitations , we propose a novel bottom-up lifelong pruning approach , which allows for re-growing the model capacity to compensate for any excessive pruning . It therefore flexibly calibrates between increasing and decreasing tickets throughout the entire learning process , alleviating the intrinsic greedy bias caused by the top-down pruning . We additionally introduce lottery teaching to overcome forgetting , which regularizes previous task models ’ soft logit outputs by using free unlabeled data . That is inspired by lifelong knowledge preservation techniques ( Castro et al. , 2018 ; He et al. , 2018 ; Javed & Shafait , 2018 ; Rebuffi et al. , 2017 ) . For validating our proposal , we conduct extensive experiments on CIFAR-10 , CIFAR-100 , and TinyImageNet datasets for class-incremental learning ( Rebuffi et al. , 2017 ) . The results demonstrate the existence and the high competitiveness of lifelong tickets . Our best lifelong tickets ( found by bottom-up pruning and lottery teaching ) achieve comparable or better performance across all sequential tasks , with as few as 3.64 % parameters , compared to state-of-the-art dense models . Our contributions can be summarized as : • The problem of lottery tickets is formulated and studied in lifelong learning ( class incremental learning ) for the first time . • Top-down pruning : a generalization of iterative weight magnitude pruning used in the original LTH over continual learning tasks . • Bottom-up pruning : a novel pruning method , which is unique to allow for re-growing model capacity , throughout the lifelong process . • Extensive experiments and analyses demonstrating the promise of lifelong tickets , in achieving superior yet extremely light-weight lifelong learners . 2 RELATED WORK . Lifelong Learning A lifelong learning system aims to continually learn sequential tasks and accommodate new information while maintaining previously learned knowledge ( Thrun & Mitchell , 1995 ) . One of its major challenges is called catastrophic forgetting ( McCloskey & Cohen , 1989 ; Kirkpatrick et al. , 2017 ; Hayes & Kanan , 2020 ) , i.e. , the network can not maintain expertise on tasks that they have not experienced for a long time . 2 This paper ’ s study subject is class-incremental learning ( CIL ) ( Rebuffi et al. , 2017 ; Elhoseiny et al. , 2018 ) : a popular , realistic , albeit challenging setting of lifelong learning . CIL requires the model to recognize new classes emerging over time while maintaining recognizing ability over old classes without access to the previous data . Typical solutions are based on regularization ( Li & Hoiem , 2017 ; Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ; Aljundi et al. , 2018a ; Ebrahimi et al. , 2019 ) , for example , knowledge distillation ( Hinton et al. , 2015 ) is a common regularizer to inherit previous knowledge through preserving soft logits of those samples ( Li & Hoiem , 2017 ) while learning new tasks . Besides , several approaches are learning with memorized data ( Castro et al. , 2018 ; Javed & Shafait , 2018 ; Rebuffi et al. , 2017 ; Belouadah & Popescu , 2019 ; 2020 ; Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018 ) . And some generative lifelong learning methods ( Liu et al. , 2020 ; Shin et al. , 2017 ) mitigate catastrophic forgetting by generating simulated data of previous tasks . There also exist a few architecture-manipulation-based lifelong learning methods ( Rajasegaran et al. , 2019 ; Aljundi et al. , 2018b ; Hung et al. , 2019 ; Abati et al. , 2020 ; Rusu et al. , 2016 ; Kemker & Kanan , 2017 ) , while their target is dividing a dense model into task-specific parts for lifelong learning , rather than localizing sparse networks and the lottery tickets . Pruning and Lottery Ticket Hypothesis It is well-known that deep networks could be pruned of excess capacity ( LeCun et al. , 1990b ) . Pruning algorithms can be categorized into unstructured ( Han et al. , 2015b ; LeCun et al. , 1990a ; Han et al. , 2015a ) and structured pruning ( Liu et al. , 2017 ; He et al. , 2017 ; Zhou et al. , 2016 ) . The former sparsifies weight elements based on magnitudes , while the latter removes network sub-structures such as channels for more hardware friendliness . LTH ( Frankle & Carbin , 2019 ) advocates the existence of an independently trainable sparse subnetwork from a dense network . In addition to image classification ( Frankle & Carbin , 2019 ; Liu et al. , 2019 ; Wang et al. , 2020 ; Evci et al. , 2019 ; Frankle et al. , 2020 ; Savarese et al. , 2020 ; You et al. , 2020 ; Ma et al. , 2021 ; Chen et al. , 2020a ) , LTH has been explored widely in numerous contexts , such as natural language processing ( Gale et al. , 2019 ; Chen et al. , 2020b ) , reinforcement learning ( Yu et al. , 2019 ) , generative adversarial networks ( Chen et al. , 2021b ) , graph neural networks ( Chen et al. , 2021a ) , and adversarial robustness ( Cosentino et al. , 2019 ) . Most of them adopt unstructured weight magnitude pruning ( Han et al. , 2015a ; Frankle & Carbin , 2019 ) to obtain the ticket , which we also follow in this work . ( Frankle et al. , 2019 ) analyzes large models and datasets , and presents a rewinding technique that re-initializes ticket training from the early training stage rather than from scratch . ( Renda et al. , 2020 ) further compares different retraining techniques and endorses the effectiveness of rewinding . ( Mehta , 2019 ; Morcos et al. , 2019 ; Desai et al. , 2019 ) pioneer to study the transferability of the ticket identified on one source task to another target task , which delivers insights on one-shot transferability of LTH . One latest work ( Golkar et al. , 2019 ) aimed at lifelong learning in fixed-capacity models based on pruning neurons of low activity . The authors observed that a controlled way of “ graceful forgetting ” after training each task can regain network capacity for new tasks , meanwhile not suffering from forgetting . Sokar et al . ( 2020 ) further compresses the sparse connections of each task during training , which reduces the interference between tasks and alleviates forgetting . 3 LOTTERY TICKET FROM SINGLE-TASK LEARNING TO CIL 3.1 PROBLEM SETUP In CIL , a model continuously learns from a sequential data stream in which new tasks ( namely , classification tasks with new classes ) are added over time , as shown in Figure 1 . At the inference stage , the model can operate without having access to the information of task IDs . Following ( Castro et al. , 2018 ; He et al. , 2018 ; Rebuffi et al. , 2017 ) , a handful of samples from previous classes are stored in a fixed memory buffer . More formally , let T1 , T2 , · · · represent a sequence of tasks , and the ith task Ti contains data that fall into ( ki − ki−1 ) classes Ci = { cki−1+1 , cki−1+2 , · · · , cki } , where k0 = 0 by convention . Let Θ ( i ) = { θ ( i ) , θ ( i ) c } denote the model of the learner used at task i , where θ ( i ) corresponds to the base model cross all tasks from T1 to Ti , and θ ( i ) c denotes the task-specific classification head at Ti . 3 Thus , the size of θ ( i ) is fixed , but the dimension of θ ( i ) c aligns with the number of classes , which have been seen at Ti . In general , the learner has access to two types of information at task i : the current training dataD ( i ) , and certain previous information P ( i ) . The latter includes a small amount of data from previous tasks { Tj } i−11 stored in the memory buffer , and the previous model Θ ( i−1 ) at task Ti−1 . This is commonly used to overcome the catastrophic forgetting issue of the current task i against the previous tasks . Based on the aforementioned setting , we state the CIL problem as below . Problem of CIL . At the current task i , we aim to learn a full model Θ ( i ) = { θ ( i ) , θ ( i ) c } based on the information ( D ( i ) , P ( i ) ) such that Θ ( i ) not only ( I ) yields the generalization ability to the newly added data at task Ti but also ( II ) does not lose its power to the previous tasks { Tj } i−11 . We note that the aforementioned problem statement applies to CIL with any fixed-length learning period . That is , for n time stamps ( one task per time ) , the validity of the entire trajectory { Θ ( i ) } n1 is justified by each Θ ( i ) from the CIL criteria ( I ) and ( II ) stated in ‘ Problem of CIL ’ .

Summary: The research question of this paper is the existence of an extremely sparse network with an initial weight assignment that can be trained online to perform multiple tasks to compete with a dense network, in a lifelong continual learning configuration. Another research question of this paper is how to identify this sparse network and achieve competitive performance. To address these questions, the authors proposed to incrementally introduce new non-zero weights when learning incoming tasks (Figure 2 and Equation 1). The network considered by the authors has a common base for all models and a head for individual tasks.