Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning
Authors: Tianlong Chen
Zhenyu Zhang
Sijia Liu
Shiyu Chang
Zhangyang Wang
Venue: ICLR 2021 Poster (2021)
Keywords: lottery tickets
winning tickets
lifelong learning

Abstract: The lottery ticket hypothesis states that a highly sparsified sub-network can be trained in isolation, given the appropriate weight initialization. This paper extends that hypothesis from one-shot task learning, and demonstrates for the first time that such extremely compact and independently trainable sub-networks can be also identified in the lifelong learning scenario, which we call lifelong tickets. We show that the resulting lifelong ticket can further be leveraged to improve the performance of learning over continual tasks. However, it is highly non-trivial to conduct network pruning in the lifelong setting. Two critical roadblocks arise: i) As many tasks now arrive sequentially, finding tickets in a greedy weight pruning fashion will inevitably suffer from the intrinsic bias, that the earlier emerging tasks impact more; ii) As lifelong learning is consistently challenged by catastrophic forgetting, the compact network capacity of tickets might amplify the risk of forgetting. In view of those, we introduce two pruning options, e.g., top-down and bottom-up, for finding lifelong tickets. Compared to the top-down pruning that extends vanilla (iterative) pruning over sequential tasks, we show that the bottom-up one, which can dynamically shrink and (re-)expand model capacity, effectively avoids the undesirable excessive pruning in the early stage. We additionally introduce lottery teaching that further overcomes forgetting via knowledge distillation aided by external unlabeled data. Unifying those ingredients, we demonstrate the existence of very competitive lifelong tickets, e.g., achieving 3-8% of the dense model size with even higher accuracy, compared to strong class-incremental learning baselines on CIFAR-10/CIFAR-100/Tiny-ImageNet datasets. Codes available at https://github.com/VITA-Group/Lifelong-Learning-LTH.

Published as a conference paper at ICLR 2021
LONG LIVE THE LOTTERY: THE EXISTENCE OF WIN- NING TICKETS IN LIFELONG LEARNING
Tianlong Chen"", Zhenyu Zhang", Sijia
Chang', Zhangyang Wang
'University of Texas at Austin, UUniversity of Science and Technology of China
'Michigan State University, "MIT-IBM Watson AI Lab, IBM Research
(ti [tianleng. chen, atlaswang) Outenag edu, zzyl 996 99w.0mai...m.mm.m..............m uste. edu 1 liusi jiijjis ji50jii....... edu, chang& change ibm. com
ABSTRACT
The lottery ticket hypothesis states that a highly sparsified sub-network be trained in isolation, given the appropriate weight initialization. This paper ex- tends that hypothesis from one-shot task learning, and demonstrates for the first time that such extremely compact and independently trainable sub-networks can be also identified in the lifelong learning scenario, which we call lifelong tickets. We show that the resulting lifelong ticket can further be leveraged to improve the performance learning over continual tasks. However, it is highly non-trivial to conduct network pruning in the lifelong setting. Two critical roadblocks arise: i) As many tasks now arrive sequentially, finding tickets in a greedy weight pruning fashion will inevitably suffer from the intrinsic bias, that the carlier emerging tasks impact more; ii) As lifelong learning is consistently challenged by catastrophic forgetting, the compact network capacity of tickets might amplify the risk of for- getting. In view of those, we introduce two pruning options, e.g., top-down and bottom-up, for finding lifelong tickets. Compared to the top-down pruning that extends vanilla (iterative) pruning over sequential tasks, we show that the bottom- up one, which can dynamically shrink and (re- (ree)de model capacity, effectively avoids the undesirable excessive pruning in the early stage. We additionally intro- duce lottery teaching that further overcomes forgetting via knowledge distillation aided by external unlabeled data. Unifying those ingredients, we demonstrate the existence of very competitive lifelong tickets, c.g., achieving 8% of the dense model size with even higher accuracy, compared strong fass-incremental Iearn- datasets. Codes available
ing baselines on CIFAR- at http://hhh...baooooooooaaaaaaoaa
INTRODUCTION
The lottery ticket hypothesis (LTH) (Frankle & Carbin 2019) suggests the existence of extremely sparse sub-network, within an overparameterized dense neural network, that can reach similar per- formance as the dense network when trained in isolation with proper initialization. Such a sub- network together with the used initialization is called winning ticket (Frankle & Carbin 2019). The original LTH studies the sparse pattern of neural networks with a single task (classifccatio), leaving the question of generalization across multiple tasks open. Following that, few works (Morcos et al.) 2019. Mehta 2019) have explored LTH in transfer learning. They study the trans- ferability of winning ticket found in source task to another target task. This provides insights on one-shot transferability of LTH. In parallel, lifelong learning not only suffers from notorious catas- trophic forgetting over sequentially arriving tasks but also often comes the price of increasing model capacity. With those in mind, we ask a much more ambitious question:
Does LTH hold in the setling of lifelong learning when different lasks arrive sequentially?
Intuitively, a desirable "ticket" sub-network in lifelong learning (McCloskey & Cohen 1989 Parisi
et al. 2019) needs to be: 1) independently trainable, same as the original LTH; 2) trained to perform
"Equal Contribution.

---

Published as a conference paper at ICLR 2021
competitively to the dense lifelong model, including both maintaining the performance of previous tasks, and quickly achieving good generalization at newly added tasks; 3) found online, as the tasks sequentially arrive without any pre-assumed order. We define such sub-network with its initialization as a lifelong lottery ticket.
This paper seeks to locate the lifelong ticket in class-incrementne learning (CIL) WWag al. 2017
Rosenteld & T 2018: Kemker & Kanan 2017: Li & Hoiem [017 Belouadah & Popescu
2019 2020), popular, realistic and challenging setting of lifelong learning. A natural idea to extend the original LTH is to introduce sequential pruning: we continually prune the dense network until the desired sparsity level, as new tasks are incrementally added. However, we show that the direct application of the iterative magnitude pruning (IMP) used in LTH fails in the scenario of CIL. since the pruning schedule becomes critical when tasks arrive sequentially. To circumvent this challenge, we generalize IMP to incorporate a curriculum pruning schedule. We term this technique top-down lifelong pruning. When the total number of tasks is pre-known and small, then with some "lottery" initialization (achieved by rewinding (Frankle al. 2019) or similar), we find that the pruned sparse ticket can be re-trained to similar performance as the dense network. However, if the number of tasks keeps increasing, the above ticket will soon witness performance collapse as its limited capacity cannot afford the over-pruning.
The limitation of top-down lifelong pruming reminds us of two unique dilemmaa that might chal- lenge the validity of lifelong tickets. i) Greedy weight pruning v.s. all tasks' performance: While the sequential pruning has to be performed online, its greedy nature inevitably biases against later arriving tasks, as earlier tasks apparently will contribute to shaping the ticket more (and might even use up the sparsity budget). ii) Catastrophic ous catastrophic forgetting (McCloskey & Cohen 1989; Tishby & Zaslavsky 2013). many lifelong learning models have to frequently consolidate weights to carefully re- re-assign the model capacity (Zhag et al., (020) or even grow model size as tasks come in (Wang et al. 2017). Those seem to contradict our goal of pruning by seeing more tasks.
v.s. small ticket size: To overcome the notori-
To address the above two limitations, we propose a novel bottom-up lifelong pruming approach, which allows for re-growing the model capacity to compensate for any excessive pruning. It there- fore flexibly calibrates between increasing and decreasing tickets throughout the entire learning process, alleviating the intrinsic greedy bias caused by the top-down pruning. We additionally in- troduce lottery teaching overcome forgetting, which regularizes previous task models' soft logit outputs by using free unlabeled data. That is inspired by lifelong knowledge preservation techniques (Cstro et al. 2018 He al., 2018: Javed & Shafait 2018 Rebuffi et al. 2017).
10, CIFAR-100, and Tiny- ImageNet datasets for class-incremental learning (Rebuffi et al., 2017). The results demonstrate the existence and the high competitiveness of lifelong tickets. Our best lifelong tickets (found by bottom-up pruning and lottery teaching) achieve comparable or better performance across all quential tasks, with as few as 3.64% parameters, compared to state-of-the-art dense models. Our contributions can be summarized as:
For validating our proposal, we conduct extensive experiments on CIFAR-10.
The problem of lottery tickets is formulated and studied in lifelong learning (class incremental learning) for the first time.
Top-down pruning: a generalization of iterative weight magnitude pruning used in the original LTH over continual learning tasks.
Bottom-up pruning: a novel pruning method, which is unique to allow for re-growing model capacity, throughout the lifelong process.
Extensive experiments and analyses demonstrating the promise of lifelong tickets, in achieving superior yet extremely liiht- lifelong learners.
RELATED WORK
Lifelong Learning A lifelong learning system aims to continually learn sequential tasks and ac- commodate new information while maintaining previously learned knowledge (Thrun & Mitchell 1995). One of its major challenges is called catastrophic forgetting (McCloskey & Cohen 1989 Kirkpatrick et al. 2017 Hayes & Kanan 2020), i.e., the network cannot maintain expertise on tasks that they have not experienced for long time.
2

---

Published as a conference paper at ICLR 2021
This paper's study subject is class- incrementll learning (CIL) (Rebuffi et al. 2017 Elhoseiny al. 2018: a popular, realistic, albeit challenging setting of lifelong learning. CIL requires the model to recognize new classes emerging over time while maintaining recognizing ability over old classes without access to the previous data. Typical solutions are based on regularization (Li & Hoiem
2017 Kirkpatrick al.] 2017 Zenke et al. 2017 Aljundi et al., 20188 Ebrahimi al. 2019). for example. knowledge distillation (Hinton et al. 2015) is a common regularizer to inherit previous knowledge through preserving soft logits of those samples (Li & Hoiem 2017) while learning new tasks. Besides, several approaches are learning with memorized data (Castro et al. 20188] Javed & 2018 2018 Rebufh et al., 2 [elouadah & Popescu, 2019 [020]] Lopez- Pazzz-aaz& & Ranzato 2
Shafait
Chaudhry et 2018). 8). And some generative lifelong learning methods (Liu et al. 2200;] Shin et al., 2017) mitigate catastrophic forgetting by generating simulated data of previous tasks. There also exist few cechiteture-maiipulation--
lifelong learning methods (Rajasegaran et al., 2019)
Aljundi et al., 2018b; Hung et al., 2019; [Abati et al., [0200] Rusu et al., 2016; Kemker & Kanan 2017). while their target is dividing a dense model into task-specific parts for lifelong learning, rather than localizing sparse networks and the lottery tickets.
Pruning and Lottery Ticket Hypothesis It is well-known that deep networks could be pruned of excess capacity (LeCun et al. 1990b). Pruning algorithms can be categorized into unstructured (Han et al. 20155 LeCun et al. 1990a Han et al. 2015a) and structured pruning (Liu et al. 2017, 2016). The former sparsifies weight elements based on magnitudes, He et al. 2017 Zhou et al.
while the latter removes network sub-structures such as channels for more hardware friendliness.
LTH (Frankle & Carbin 2019) advocates the existence of independently trainable sparse sub-
network from a dense network. In addition to image classification (Frankle Carbin
2019]
Liu
et al. 2019 Wang al. 2020 Evei al. 2019: Frankle al., 2020 Savarese et al. 2020 You et al.
2020
et [221] Chen et 20220a. LTH has been explored widely in numerous contexts, such
as natural language processing (Gale et al.) 2019 Chen et al., (022b), reinforcement learning Yuu
iet al. 2019). generative adversarial networks (Chen et al.] (0221b), graph neural networks (Chen al. 2021a), and adversarial robustness (Cosentino et al. 2219), Most of them adopt unstructured
weight magnitude pruning (Han et al.] 2015a Frankle & Carbin 2019) to obtain the ticket, which we also follow in this work. (Frankle al. (219)) analyzes large models and datasets, and presents rewinding technique that re-initializes ticket training from the early training stage rather than from scratch. (Renda et al.] (2020) further compares different retraining techniques and endorses the
effectiveness of rewinding. (Mehta 2019 Morcos et al. 2019 Desai et al.) 2019) pioneer to study the transferability of the ticket identified on one source task to another target task, which delivers insights on one-shot transferability LTH.
One latest work (Golkar et al. 2019) aimed at lifelong learning in fixed-capacity models based on pruning neurons of low activity. The authors observed that controlled way of "graceful forgetting" after training each task can regain network capacity for new tasks, meanwhile not suffering from forgetting. (Sokar al. (2000) further compresses the sparse connections of each task during training. which reduces the interference between tasks and alleviates forgetting.
3 LOTTERY TICKET FROM SINGLE-TASK LEARNING TO CIL
3.1 PROBLEM SETUP
In CIL, a model continuously learns from a sequential data stream in which new tasks (namely, classification tasks with new classes) are added over time, as shown in Figure [ At the inference stage, the model can operate without having access to the information of task IDs. Following (Castro et al., 2018 He et al. 2018 Rebulfi et al.) 2017.. a handful of samples from previous classes are stored in a fixed memory buffer.
Class [c).
Class
[o -1
Task T
J
Task Ts
Class-incremenntal Learner
TaskTaf
Class
(ch=l.--..cs
Figure 1: Basic CIL Setting.
More formally, let Ti, ...... represent a sequence of tasks, and the ith task Ta contains data that
fall into (k:- k; classes C, = {Ck, Ch-142: Ck., }, where ko 0 by convention. Let
( = (0()) y denote the model of the learner used at task i, where ( corresponds to the base model cross all tasks from Ti to Tú, and ! denotes the task-specific classification head at Ti-
3

---

Published as a conference paper at ICLR 2021
Thus, the size of O(0) is fixed, but the dimension of 0 aligns with the number of classes, which have been seen at Tj In general, the learner has access to two types of information at task i: the current training data D(1), and certain previous information p The latter includes a small amount of data from previous tasks {T 1-111111 stored in the memory buffer, and the previous model (i-1) at task T, This is commonly used to overcome the catastrophic forgetting issue of the current task i against the previous tasks. Based on the aforementioned setting, we state the CIL problem as below.
Problem of CIL. At the current task i, we aim to learn full model ( = { e based on the information (D(t), p(() such that ( not only (I) yields the generalization ability to the newly added data at task Ti but also (II) does not lose its power to the previous tasks {T
We note that the aforementioned problem statement applies to CIL with any fixed-length learning period. That is, for time stamps (one task per time), the validity of the entire trajectory (O()) J1 is justified by each ((1) from the CIL criteria (l) and (II) stated in 'Problem of CIL..
3.2 LIFELONG LOTTERY TICKETS
was shown by LTH (Frankle& & Carbin 2019) that a standard (unstructured) pruning technique can uncover the so-called winning ticket, namely, a sparse sub-network together with proper initialization that can be trained in isolation and reach similar performance as the dense network. In this paper, we aim prune the base model ((+) over time. And we ask: Do there exist winning lickets in lifelong learning? If yes, how to obtain them? To answer these questions, prerequisite is to define the notion of lottery tickets lifelong learning, which we call lifelong lottery tickets.
Following LTH (Frankle & Carbin 2019). lottery ticket consists of two parts: 1) binary mask m {0, 11 (lo obtained from one-shot or iterative pruning algorithm, and 2) initial weights or rewinding weights Op. The ticket (m, 0o) is a winning ticket if training the subnetwork m @ 0g (o denotes element-wise product), identified by the sparse pattern with initialization Oo. wins the initialization lottery to match the performance of the original (fully trained) network. In CIL, at the presence of sequential tasks ) from the perspective of dynamical system:
define lifelong lottery tickets (m()), o0")) recursively
m (1-1) + A(D(), p(o) (t-)), and
(0))) O00)
(I)
where A denotes pruning algorithm used the current task ( based on the information D(O, p(() and (+-1), ((0) denotes the initialization prior to training the model at (1), and of denotes a rewinding point at ( In Eq. (.. we interpret the (non-trivial) pruning operation A by weight perturbations, with values drawn from 0, 1}. to the previous binary mask. Here - denotes the removal of a weight, O signifies to keep a weight intact, and represents the addition of a weight. Moreover, the introduction of weight rewinding is spurred by the so-called rewinding ticket (Rendal cct al., 2020; Frankle et al. 2020 For example, if o/@ = @ then we pick the model weights learnt at the previous task T to initialize the training at ( We also note that O can be regarded as the point rewound to the earliest stage of the lifelong learning. Based on Eq. I.. we then state the definition of winning tickets in CIL.
Lifelong winning tickets.
Given sequence of tasks (T:).. the lifelong lottery tickets
(mm)) (1) )0 given by (1) are winning tickets if they can be trained in isolation to match the CII. performance (i.e., criteria and II) of the corresponding full model {0(3) where n N+
In the next section, we will design the lifelong pruning algorithm A. together with ticket initializa- tion schemes formulated in Eq.
4 PROPOSED PRUNING METHOD TO FIND LIFELONG WINNING TICKETS
4.1 REVISITING IMP OVER SEQUENTIAL TASKS: TOP-DOWN (TD) PRUNING
In order to find the potential tickets at the current task 7(,, it is natural to specify A in Eq. | as the iterative magnitude pruning (IMP) algorithm (Han al. 2015a to prune the model from
4

---

Published as a conference paper at ICLR 2021
m (1-1) o ((i-1), Following (Frankle & Carbin 2019: Renda et al. 2020). IMP iteratively prunes pat (%) non-zero weights of (i-1) C (6-1) over (0) rounds at 7 T Thus, the number of non-
zero weights in the obtained mask m (8) given by ((1 p-.ó> yol (la)) However, in the application of IMP to the sequential tasks (T y. we find that the schedule of IMP over sequential
tasks, in terms of (, is critical make pruning successful in lifelong learning. We refer readers
to Appendix A2. for detailed justifications.
Curriculum schedule of TD pruning is key to success The conventional method is to set {n(*) as a uniform schedule, namely, IMP prunes a fixed portion of non-zeros at each task. However, this direct application fails quickly as the number of incremental tasks increases, implying that "not all tasks are created equal" in the learning/pruning schedule. Inspired by the recent observation that training with more classes helps consolidate more robust sparse model (Morcos al., 2019), we propose a curriculum pruning schedule, in which IMP is conducted more aggressively for new tasks arriving later, with (4) 2 n,(-1), until reaching the desired sparsity. For example, if there are 12 times of pruning on five sequentially arrived tasks, arrange them in linearly increasing way, i.e., (T::1,72:1 T:::2.74:3..T
Note that TD pruning relies on the heuristic curriculum schedule, and thus inevitably greedy and suboptimal over continual learning tasks. In what follows, we propose more ad advanced pruning scheme, bottom-up (BU) pruning, that obeys a different principle of design.
4.2 BOTTOM-UP (BU) LIFELONG PRUNING: AN ADVANCED SCHEME
- J- Romddum Iniialioniim/ Connection
J Weights Connection from Ti
J: Weights Connection from Tu-
Weighta / Cammattin from T--4
Buttom-up Lifelong Prusing
Reference Capruned Madel
Chrotsveuual Leanty %
Cla nerenerree Leaning
Clavinnnmoral Lamiiz
T
Rovinding
Ti
T2
Ts
Tx
ta
IMP
IMP
IMP
nall grtd
&
PI gits
Figure 2: Framework of our proposed bottom-up (BU) lifelong pruning which is based on sparse model consolidation. Tickets founded by BU pruning keep expanding for each newly added task.
Why we need more than top-down pruning? TD pruning is inevitably greedy and suboptimal. Earlier added tasks contribute more to shaping the final mask, due to the nested dependency be- tween intermediate masks. In the later training stage, we often observe the network is already too heavily pruned to learn more tasks. Inspired by the recently proposed model consolidation (Zhang ett al. 2020), we propose the BU alternative of lifelong pruning, to dynamically compensate for the excessive pruning by re-growing previously reduced networks.
Full reference model & rewinding point For BU lifelong pruning, we maintain full (unpruned) model g(6)
ref as reference throughout lifelong learning. First, g(i) provides a reference performance RR(() obtained (1)) Once the validation accuracy of the current sparse model is worse than the reference performance, the sparse model is considered to still have sufficient capacity and can be further pruned. Otherwise, capacity expansion is needed. On the other hand, the reference model offers rewinding point for network parameters, which preserves knowledge of all previous tasks
prior to ( It naturally extends the rewinding concept (Frankle et al. 2019) to lifelong learning.
BU pruning method BU lifelong pruning expands the previous mask m (f-1) to m m() Different from TD pruning, the model size grows along the task sequence, namely, [| I1o 2 [|mm611) Ilo-
Thus, BU pruning enforces A in Eq. 4 to draw non-negative perturbations. As illustrated in Fig-
ure 2 for each newly added Ti, we first re-train the previous sparse model (i-1) @ ((1-1) under
5

---

Published as a conference paper at ICLR 2021
the current information (D(1) p(()) and calculate the validation accuracy R( If R() is above than the reference performance R. rref we proceed to keep the sparse mask m (5) = (i-)) intact and use re-trained ( as (() at Tú Otherwise, an expansion from (+-1) is required to ensure sufficient learning capacity. To do so, we restart from the full reference model 0 and iteratively prune its weights using IMP until the performance gets just below R R Here the previous non-zero weights localized by m (8-1) are excluded from the pruning scope of IMP but the values of those non-zero weights could be re-trained. As a result, IMP will yield the updated mask m (6) with a larger size than m (1-1) We repeat the aforementioned BU pruning method when a new task arrives.
Although never observed in our CIL experiments, potential corner case of expansion is that the ticket size may hit the size of the full model. We consider this as an artifact of limited model capacity and suggest future work of combining lifelong tickets with (full) model growing (Wang al. 2017).
Ticket initialization Given the pruning mask found by the BU (or TD) pruning method, we next
determine the initialization scheme of lifelong ticket. We consider three specifications of 0
Eq. I to initialize the sparse model (+) for re-training the found tickets. They include: (I)
0% = ( i.e., the original "from the same random" initialization (Frankle & Carbin 2019). (II)
random re-initialization Oreinit which is independent of o()) and (III) previous-task rewinding, i.e. 0 O(i-1), The initialization schemes I-III together with m(() yield the following tickets m( T (1) BU (or TD) tickets. namely, m found by BU (or TD) pruning with initialization I; (2) random BU (or TD) tickets, namely, m with initialization II; (3) task-rewinding BU (or TD) tickets, namely, m(( with initialization III. In experiments,
will show that both BU (or TD) tickets and their rask-rewinding (TR) counterparts are winning tickets, which outperform unpruned full CIL models. Compared BU with TD pruning, TR-BU tickets surpass the best TD tickets.
4.3 LOTTERY TEACHING: A PLUG-IN REGULARIZATION
Catastrophic forgetting poses a severe challenge class-incremental learning, especially for com- pact models. (Castro et al. 2018 He el al. 2018 Javed & Shafait 2018 Rebuffi et al. 2017) are early attempts for undertaking the forgetting dilemma by introducing knowledge distillation regu- larization (Hintnn et al., 2015). which employs handful of stored previous data in addition to new task data. (Zhang et al.
2020) takes advantage of unlabeled data to handle the forgetting quandary.
We adapt their philosophy (Li & Hoiem 2017 Hinton et al. 2015 Zhang et al. 2020) to presenting lottery teaching, enforcing previous information into the new tickets via knowledge distillation term on external unlabeled data. Lottery teaching consists of two steps: i) we query more similar unlabeled data "for free" from a public source, by utilizing a small number of prototype samples from previous tasks' training data. In this way, the storage required for previous tasks could be minimal, while the queried surrogate data functions similarly for our purpose; ii) we then enforce the
output soft logits of the current subnetwork o0(+,, o. each queried unlabeled sample to be close to the logits from previously trained subnetwork (m-(11) @ 0(-1) o((-1),, via knowledge distillation (KD) regularization based on the K-L divergence. For all experiments of our methods hereinafter, we by default append the lottery teaching as it is widely beneficial. An ablation study will also follow in Section 5.3.
EXPERIMENTS
Experimental Setup We briefly discuss the key facts used in our experiments and refer readers to Appendix A2.3 for more implementation details. We evaluate our proposed lifelong tickets on three datasets: CIFAR-IO, C00,R-100, and Tiny- ImageNet. We adopt ResNet18 (He et al.) 2016)) as our backbone. We evaluate the model performance by standard testing accuracy (SA) averaged over three independent runs.
CIL baseline: We consider strong baseline framework derived from (Zhang et al. 2019 recent state-offtth--at (SOTA) method introduced for imbalanced data training (see more illustrations in Appendix AL. We implement (Zhang al. 2019) for CII., and compare with two latest CIL.

---

Published as a conference paper at ICLR 2021
Our results demon- strate the adapted CIL method from (Zhang et al. 2 outperforms the others significantly, (1.65% SA better than IL2M and 4.88% SA better than iCaRL on CIFAR-10 establishing new SOTA bar. The proposed lottery teaching further improves the performance of the baseline adapted from Zhang et al. (2019), given by 4.4% SA improvements on CIFAR-10. Thus, we use (Zhang al.) 2019). combined with/without lottery teaching, to train the original (unpruned) CIL model.
SOTAs: iCaRL (Rebuffi et al. 2017) and IL2M (Belouadah & Popescu 2019
CIL pruning: To the best of our knowledge, we are not aware of any effective CIL pruning baseline comparable to ours. Thus, we focus on the comparison among different variants of our methods. We also compare the proposal with the ordinary IMP, showing its incapability in CIL pruning. Further- more, we demonstrate that given our proposed pruning frameworks, standard pruning methods such as IMP and & pruning (by imposing & sparsity regularization) then turn to successful.
Results on TD tickets We begin by showing that TD pruning is non-trivial in CIL. We find that the ordinary IMP (Han et al. 2015a) fails: It leads to 10.21% % SA degradation (from 72.79% to 62.58% for SA) with 4.40% parameters left. By contrast, our proposed lifelong tickets yield substantially better performance which even surpasses the full dense model, with fewer parameters left than the ordinary IMP (Han et al. 2015a).
In what follows, We evaluate TD lifelong pruning using different weight rewindings, namely, i) TD tickets; ii) random TD tickets; iii) task- book-rewinding TD tickets; iv) late-rewinding TD tickets; and v) Fine-tuming The late- retiieewnn tickets is strong baseline claimed in Mchta (0119).
Figure 3 and Table A4 demonstrate the high competitiveness of our proposed TD ticket (blue lines). It matches and most of the time outperforms the full ( dash lines). Even with only 6.87% model parameters left, the TD ticket still surpasses the dense model by 0.49% SA. The lask-rewinding tickets, in second place, exceeds the dense model un- til reaching the extreme sparsity of 4.40%. Moreover, we see late-rewinding TD tickets dominate over other rewinding/fine-unn tuning op- tions, echoing the finding in single-task learn-
74
(SA)
72
70-
Full Model
TD Tickets
6B
random TD Tickets
66
task-rewindingg TD Tickets
late-rewinding TD Tickets
Fine-tuning
62
100
51.2
48.96
687
2.25
Remaining Weights x
Figure 3: Evaluation performance (standard accuracy) top-down lifelong tickets on CIFAR-10. 10.
ing (Frankle et al. 2019).
However, TD pruning cannot afford a lot more incremental tasks due to its greedy weight (over- prunnnng Our results show that TD lickets pruned from only tasks Ti and T/ clearly overfit the first two tasks, even after incrementally learning the remaining three tasks. In this inappropriate pruning schedule (in contrast to Ti Ts scheme), the resultant ticket drops to 59. 28% SA which is 13.51% lower than the dense model, as shown in Table A3 More results can be found in the appendix. Therefore, bottom-up lifelong pruning is proposed, remedy for relieving laborious tuning of pruning schedules.
Results on BU lifelong tickets The bottom-up lifelong pruning allows the sparse network to regret if they could not deal with the newly added tasks, which compensates for the excessive pruning and reaches a substantially better trade-off between sparsity and generalization ability. Compared to TD pruning, it does not require any heuristic pruning schedules.
In Table I we first present the performance of BU tickets, random BU tickets, and task-rewinding BU (TR-BU) tickets, as mentioned in Section 4.2 As we can see, TR-BU tickets obtain the supreme performance. A possible explanation is that rewiidennn (i.e., g = g(i- maintains full in- formation of learned tasks which mitigates the catastrophic forgetting, while other weight rewinding points lack sufficient task information to prevent compact models from forgetting. Next, we observe that TR tickets significantly outperform the full dense model by 0.52% SA with only 3.64%
Both are implemented using official codes. The comparison has been strictly controlled to be fair, including dataset splits, same previously stored data, due diligence in hyperparameter tuning for each, etc.
i Moore comparisons with the latest CIL SOTAs are referred to the Appendix A1.1 Full model, denoting the performance of the dense CII. model (Zhang 2019 with lottery teaching.
7

---

Published as a conference paper at ICLR 2021
Table 1: Comparison results across full dense model, BU Tickets with different ticket initialization, and & BU
Tickets when training incrementally on CIFAR-10. Ti denotes the learned sequential tasks Ti ~ Te.
Dataset
CIFAR-I0 (Standard Accuracy Remaining Weights
( 10
Settings
Ti (%) 100%
Tiwz (%) 100%
Tima (%) 100%
T (9) 100%
Timo 100%
Full Dense Model
97.75
89.10
82.83
76.99
72.79
Ti (%) 2.81%
Ti-a (96) 3.11%
T1-3
(93)33
Tisa (9) 3.64%
Vi- (%) 3,64%
BU tickets random BU tickets
98.05
86.77
75.87
70.81
68.77
96.55
82.08
77.97
72.84
71.17
TR-BU tickets
98.05
88,90
81.37
74.66
73.31
Ti (%) 1.80%
Tiwa (%) / 2.93%
Ti-a 2.93%%
Ti-a (%) /4.05%
Tiwa (%) 5.16%
& BU tickety
96.80
87.05
77.58
74.53
72.88
Standard Acturecy (SA) %
Standard Accuracy (5A)
TR-BU Tickets 20%)
TD Tickets (9 (9.66%)
90-
60
80
TR-BU Tickets (3.64%)
TO Tickets (4.40%)
40
1
2
3
6
7
10
The number of Learned Tasks
The number Learned Tasks
Remaining Weights
BG. 50.00
TR-BU Tickets
Remalninn Inin Weights
TR-BU Tickets
64,00
TD Tickets
TD Tickets
32.77
TEZI
2.81
3.11
3.40
1.64
4.403.64
-12
12
40
Ti=2
71-3
T1-4
T
1.
T--2 11
T1-4
7155
71
7--7
T1-3
71-30
Figure 4: Performance and sparsity comparison between TR-BU tickets and TD tickets when training models incrementally. Left: CIFAR-10. Right: CIFAR-100. Upper: Comparison of SA. Boutom: Comparison of remaining weights in tickets. Above all, tickets located by TD pruning continue to shrink with the growth of incremental tasks. On the contrary, tickets founded by BU pruning keep expanding for each newly added task.
parameters left and é BU tickets obtain matched performance the full dense model with 5.16% remaining parameters. It suggests that IMP, & and even other adequate pruning algorithms can be plugged into our proposed BU pruning framework to identify the lifelong tickets.
In Figure 4 we present the performance comparison between TR-BU tickets (the best subnetworks in Table Dh and TD tickets. TR-BU tickets are identified through bottom-up lifelong pruning, whose sparse masks continue to subtly grow along with the incremental tasks, from sparsity 2.81% at the first task to sparsity 3.64% the last task. As we can see, any incremental learning stage, TR-BU tickets attain a superior performance with significantly fewer parameters. Particularly, after learning all tasks, TR-BU tickets surpass TD tickets by 101% SA with 0.76% fewer weights on CIFAR- 10; 3.07% with 2.46% fewer weights on CIFAR-100. Results demonstrate TR-BU tickets have better generalization ability and parameter-efffciency compared with TD tickets. In addition, on Tiny- ImageNet in Table A7 TR-BU tickets outperform full model with only 12.08% remaining weights. is worth to mention that both TR-BU tickets and TDD rickets have superior performance than full dense model. We refer readers to Table A5 and A6]in in the appendix for more detailed results.
A6]
From the above results, we further observe that TR-BU tickets achieve comparable accuracy to full models which have more than 30x times in network capacity, implying that bottom-up lifelong pruning successfully discovers extremely sparse sub-networks, and yet they are powerful enough to inherit previous knowledge and generalize well on newly added tasks. Furthermore, our proposed lifelong pruning schemes can be directly plugged into other CIL models to identify the lifelong ticket, shown in Appendix AT
Ablation studies In what follows, summarize results ablation studies and refer readers to the appendix AA.2.I for more details. Figure 5 we show the essential role of the curriuulum schedule in TD pruning compared the uniform pruning schedule. We notice that the curriculum
8

---

Published as a conference paper at ICLR 2021
100
100
100
Pruning Schedule of 6.87%
Curniclum Schedule of 10.74%
Standard Accuntry (5A) %
Uniform
Standard Accuntcy (SA)
90
Remaining Woionns %
Remaining Weight %
60
60
80
40
40
70
Tickets
Curriculum Schedule Tickets)
Tickets w.o. Lottery Teaching
Uniform Schedule (TD Tickets)
50
The number of Leamed Tasks
The number Leamed Tasks
Figure 5: Left: the results of TD Tickets with/without lottery teaching. Right: the comparison of TD tickets
(10.74%) obtained from uniform and curriculum pruning schedule. Experiments conducted CIFAR-IO.
pruning scheme generates stronger TD rickets than the uniform pruning in terms of accuracy, which confirms our motivation that pruning heavier in the late stage of lifelong learning with more classes is beneficial. In Table A8 we demonstrate the effectiveness of our proposals against different num- bers of incremental tasks. In the Figure[5] we show that lottery teaching injects previous knowledge through applying knowledge distillation external unlabeled data, and greatly alleviates the catas- trophic forgetting issue in lifelong pruning (i.e., after learning all tasks, utilizing lottery teaching obtains 4.34% SA improvement fixed storage capacity to store 128 unlabeled images queried from public sources at each training iteration. We find that leveraging newly queried unlabeled data offers better generalization-abiityy than storing historical data in past tasks. The latter only reaches 70.60% SA on CIFAR-10 10, which is 2.19% worse than the of unlabeled data.
CIFAR-10) It is worth mentioning that set a buffer of
CONCLUSION
We extend the Lottery Ticket Hypothesis to lifelong learning, in which networks incrementally learn from sequential tasks. We pose top-down and bottom-up lifelong pruning algorithms to identify life- long tickets. Systematical experiments are conducted to validate that located tickets obtain strong(er) generalization ability across incremental learned tasks, compared with unpruned models. Our fu- ture work aims to explore lifelong tickets with the (full) model growing approach.
REFERENCES
Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learn- ing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3931-3940, 2020.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 139-154, 2018a.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. arXiv preprint 2018b.
Anonymous.
Why lottery ticket wins? a theoretical perspective of sample complexity on sparse neural networks. In Submitted to International Conference Learning Representations, 2021. URL htll/ttps///openeeiiiee net worumoiddddooo. 8p2 8PG2233YT under review.
Eden Belouadah and Adrian Popescu. I12m: Class incremental learning with dual memory. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Eden Belouadah and Adrian Popescu. Scail: Classifier weights scaling for class incremental learn- ing. In The IEEE Winter Conference on Applications of Computer Vision, pp. 1266-1275. 2020.
Francisco M Castro, Manuel J Marín-Jiménn Jimééz, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 233-248, 2018.

---

Published as a conference paper at ICLR 2021
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1112.00420, 2018.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv preprint arXiv:2012.06900 2020a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint A.Xv:2207..12233, 2020b.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks, 2021a.
Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. [GAN]s can play lottery tick- ets too. In International Conference on Learning Representations, 2021b. URL https: 7/ 7/openrevien ew..wwwww. wwt. saet/forum?? farrumiidee d=1ommhcciAMMccc.oo _
Justin Cosentino, Federico Zaiter, Dan Pei, and Jun Zhu. The search for sparse, robust neural networks. arXiv preprint arXiv: 19/2.0266, 2019.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint 1910.12000.000 2019.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Mareus Rohrbach. Uncertainty-guided continual learning with bayesian neural networks. arXiv preprint arXiv: wrXiv:1906.02225 2019.
Mohamed Elhoseiny, Francesca Babiloni, Rahaf Aljundi, Mareus Rohrbach, Manohar Paluri, and Tinne Tuytelaars. Exploring the challenges towards lifelong fact learning. In Asian Conference on Computer Vision, pp. 66-84. Springer, 2018
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: /0732, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2019. URL /ttpttas///peeneev.eee net forrmm///ormmmmdeeom - b3RcF7
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. The lottery ticket 2019.
hypothesis scale. arXiv preprint
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network
training. In International Conference Learning Representations, 2020. URL https:
//openreview wwt.:ee///ooooo http:/orrmmmidd farrrmmid=H. d= 1 WS
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint ar0iv: 0955444 2019.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint arXiv:/903 1903044600 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1135-1143, Curran Asso- ciates, Inc., 2015b.
Tyler I. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear dis- criminant analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 220-221, 2020.
Chen He, Ruiping Wang, Shiguang Shan, and Xilin Chen. Exemplar- supported generative repro- duction for class incremental learning. In British Machine Vision Conference, 2018.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint NXXV:1503.0253: 2015,
Ching- Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pp. 13669-13679, 2019.
Khurram Javed and Faisal Shafait. Revisiting distillation and incremental classifier learning. In Asian Conference on Computer Vision, pp. 3-17. Springer, 2018.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning.
arXiv preprint arXiv:1711.0055 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baaiskka-Barinska. et al. Overcom- ing catastrophic forgetting neural networks. Proceedings of the national academy of sciences, 114(13):3511-3526 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's s thesis, Department of Computer Science, University of Toronto, 2009.
Science,,
Alex Krizhevsky, lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In D. S. Touretzky (ed.), Advances in Neural Information Processing Systems 2, pp. 598-605. Morgan-Kann Kaufmann, 1990a. URL httt:th/:///ppetee
wnips..p.............. cc/ http://aper/22200 wttppppc.ccppppe//500-aaop/
bra n- dawage. .ddf
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-299 2017.
Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. Generative feature replay for class-incremental learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 226-227, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang, Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2019. htts:: //openreview. www.:nett..oommmoomm d=rJl nm3C5mm
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pp. 6467-6476, 2017.
Haoyu Ma, Tianlong Chen, Kung-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2101.03255, 2021.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
11

---

Published as a conference paper at ICLR 2021
Rahul Mehta. Sparse transfer learning via winning lottery tickets. arXiv preprint arXiv: 1905.07788 2019.
Ari Morcos, Haonan Yu, Michela Pagannni, and Yuandong Tian. One ticket to win them all: gener- alizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Infor- mation Processing Systems, pp.
2019.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Jathushan Rajasegaran, Munawar Hayat, Salman H Khan, Fahad Shahbaz Khan, and Ling Shao. Random path selection for continual learning. In Advances in Neural Information Processing Systems, pp. 12669-12679) 2019.
Sylvestre- Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruming. In International Conference on Learning Representations, 2020. URL https: //openzevies wwwttforuwmieeeeeeeeeeeeeeeeeeee
JNKKV
Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE transac- tions on pattern analysis and machine intelligence, 2018.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.0461, 2016.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification, 2020.
eww. ftatt/oorrmmddd-BBeeeeeee
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999 2017.
Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Spacenet: Make free space for continual learning. arXiv preprint arXiv:2007.07617, 2020.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Roboties and autonomous systems, 15(1-2):25 46. 1995.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, (9(11))1988-1900 2008
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Growing a brain: Fine-tuning by increas- ing model capacity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2471-2480, 2017.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early- tickets: Toward more efficient training of deep networks. In International Conference on Learning Representations, 2020. URL https:
//openreview.o wwt.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. arXiv preprint 10K6::900.02/770 096602768, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987, 2017.
12

---

Published as a conference paper at ICLR 2021
Junjie Zhang, Linggiao Liu, Peng Wang, and Chunhua Shen. To balance or not to balance: A simple cimple-ye-effeective approach for learning with long-tailed distributions. arXiv, pp. arXiv-1912, 2019.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. Class-incremental learning via deep model consolidation. In The IEEE Winter Conference on Applications of Computer Vision, pp. 1131-1140, 2020.
Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less more: Towards compact enns. European Conference on Computer Vision, pp. 662-677. Springer, 2016.
13

---

Published as a conference paper at ICLR 2021
A1 MORE EXPERIMENT RESULTS
A1.1 MORE BASELINE RESULTS
Comparison with the Latest CIL SOTAS We find (Zhang al. 2019) can be naturally intro- duced to ilass-nncremental learning, which tackles the intrinsic training bias between a handful of previously stored data and large amount of newly added data. It adopts random and class- baas-balancedd sampling strategies, combined with an auxiliary classifier, to alleviate the negative impact from im- balanced classes. Extensive results, shown in Table A2] demonstrates that adopting (Zhang et al.) 2019) the simple baseline surpasses previous SOTAs iCaRL (Rebuffi et al. (017)) and IL2M (Belouadah Popescu, 2019) by a significant performance margin (1.65% SA better than IL2M and 4.88%/77.00 SA better than iCaRL on ing a new SOTA bar. With the assistance lottery teaching, (Zhang et al.) 2019) obtains an extra performance boost, 4.4% SA CIFAR-10 10 and 7.34% SA on CIFAR-100.
respectively + establish-
It is worth mentioning that a lifelong ticket also exists in other CIL models. Take IL2M on CIFAR- 10 as example, bottom-up (BU) ticket achieves accuracy 68.92% with 11.97% parameters vs. the dense unpruned model with an accuracy of 66.74%
Table A2: Comparison between our dense model and two previous SOTA CIL methods on CIFAR- 10 and CIFAR- 100. Reported performance it the final accuracy for each task T. Simple baseline donates the dense CIL model (Zhang et al. (2019). Full model represents our proposed framework which combines lottery teaching technique with the simple baseline.
CIFAR-10
Ti (%)
Ts (%)
Ty (%)
Ta (%)
Ta (%)
Average (%)
Methods
iCaRL
76.45
79,00
75.70
50.85
35,55
63.51
1L2M
78,20
64.05
60.40
38.95
92.10
66.74
Simple Baseline Full Model
75.05
71.50
54.25
52.05
89.10
68.39
79.70
79.12
68.23
63.45
73.43
72.79
CIFAR-100
Ti (%)
Ty (%)
Ta (%)
T (%)
T5 (%)
T7 (%)
TF (%)
Tg (%)
Ta (%)
Tio (%)
Average (%)
Methods
iCaRl.
5.90
7.50
4.50
2,80
9.00
8.00
28.20
38,50
59.60
80,20
24.42
11.2M
19.90
24.10
19.80
12.90
21.30
21.70
29.90
34,80
40.30
89.80
31.45
Simple Baseline Full Model
32.10
23.00
22.70
33.80
40.20
32,02
29.04
33.94
32,54
27.94
32.74
29.64
47.94
45.34
47.24
67.24
39.36
Pruning Schedule Important As shown Table A3 an inappropriate pruning schedule across
Ti ~ T2, the resultant ticket drops to 59. 28% accuracy which is 13.51% lower than the dense model. On the contrary, the adequate scheme across Ti ~ To in Table A3 generates a TD winning ticket with a higher test accuracy (+0.49% SA) and extreme fewer parameters (6.87%), compared with the dense CIL model.
Table A3: Evaluation performance of TD tickets (6.87%) pruned from different task ranges.
TD Tickets (6.87%)
CIFAR-10
Pruning Schedule
Ti (%)
Tz (%)
Ta (%)
T4 (%)
Ta (%)
Average (%)
Prune actoss Ti Tz
74.05
88.80
78.25
28.40
26.90 70.00
59.28
Prune across Ti ~ Ts
78.90
82.15
71.55
63.80
73.28
A1.2 MORE LIFELONG TICKETS RESULTS
Top-down Lifelong Tickets We also report several performance reference baselines: (a) Full model, denoting the achievable performance of the dense CIL model (Zhang et al., 2019) combined with lottery teaching.
(b) CILtower denoting vanilla CIL model without using lottery teaching
'To ensure fair compassion, iCaRL and IL2M both are implemented with their official codes. The compar- ison has heen strictly controlled to he fair, including dataset splits, same previously stored data, due diligence in hyperparameter tuning for each, etc.
A14

---

Published as a conference paper at ICLR 2021
storing/utilizing previous data in any form; (c) MT upper training dense model using full data from all tasks simultancously in a multi-task learning scheme. While it not CIL (and much easier accuracy "upper bound" for (dense) CIL ; (d) MT by di-
to learn), consider MT upper rectly pruning upper to obtain its lottery ticket (Frankle & Carbin 2019). The detailed evaluation performance of TTD) fickety at different sparsity levels CIFAR-TU TO are collected in Table A4
74
Stardard Accuracy
Standard Accuracy
70
Clloaar
668
Model
TD Tickets
TD Tickets
Tickets
66
task-rewindding TD Tickets
task-rewinding TD Tickets
bet-rrewnnding
64
late-rewinding
Fine Fine-tuning
Finestuning
100
51.2
10.74
6:57
100
51.2
40.96
10:74
6.67
2.5
Remaining Weights %
Remaining Weights %
Figure A6: Evaluation performance (standard accuracy) of top-down lifelong tickets. The right figure zooms in the red dash-line box the left figure.
Table A4: Evaluation performance of TD tickets at different sparsity levels on CIFAR-10. Re- ported performance is the final accuracy for each task T. Differences (+/-) are calculated w.r.t. the full/dense model performance.
TD Tickets on CIFAR-10
Remaining Weights
Ti (%)
T2 (%)
Ta (%)
T4 (%)
Ts (%)
Average (%)
MT upper (100.00%)
97.48
97.48
93.33
90.83
94.03
94.63
Cow (100.00%)
0.00
0.00
0.00
0.00
93.70
18.74
100.00%
79.70
79.12
68.23
63.45
73.43
72.79
32.77%
84.30
80.05
70.70
67.75
69.55
74.47 + 1.68
10.74%
78.90
77.75
76.30
63.55
71.05
73.51 0.72
6.87 %
78.90
82.15
71.55
63.80
70.00
73.28 + 0.49
4.40%
82.25
78.55
65.10
65.38
70.23
72.30 -.49
2.25%
78.20
78.30
69.50
58.20
65.00
69.84 2.45
Bottom-up Lifelong Tickets As shown in Table A3 and Table A6 even compared with the best TD tickets in terms of the trade-off between sparsity and accuracy, TR-BU tickets consistently remain prominent on both CIFAR-10 (a slightly higher accuracy and 3.23% fewer weights) and CIFAR- 100 (2.37% higher accuracy and 4.88% fewer weights), From the results, we further observe that TR-BU tickets achieve comparable accuracy to full models which have more than 30x times in network capacity, implying that bottom-up up lifelong pruning successfully discovers extremely sparse nub-networks, and yet they are powerful enough to inherit previous knowledge and generalize well on newly added tasks.
A1.2.1
MORE ABLATION RESULTS
Uniform v.s. Curriculum I Lifelong Pruning We discuss different pruning schedules of top-down lifelong pruning, which play an essential role in the performance of TD rickets. From the right figure in Figure A7 we notice that the curriculum pruning scheme generates stronger TD tickets than the uniform pruning terms of accuracy, which confirms our motivation that pruning heavier in the late stage of lifelong learning with more classes is beneficial.
The Number of Incremental Tasks Here we study the influence of increment times our lifelong learning settings. shows the results of TR- TR-BU2g tickets incrementally learn from 20 tasks (5
A15

---

Published as a conference paper at ICLR 2021
Table A5: Evaluation performance of TR-BU/TD tickets on CIFAR-10. T i (1,,,3, 2,5,, do- nates that models have learned from ", incrementally. sparsity.
I
represents the current network |
Compact Weights
Ti (%)
limiin ?ill
Ti-a (%)
Imil: T
Ti-a (%)
CIFAR-IO
li TP]c
T (%)
liml T996
Ti-45 (%)
Imile T[M
MTugeer
94.63
100.00%:
ClLioverr
97.60
100.00%
49.80
100.00%
32.57
100.00%
23.23
100.00%
18.74
100.00%)
Full Model
97.75
100.00%
89.10
100.00%
82.83
100.00%
76.99
100.00%
72.79
100,00%
TD nickets (6.87%)
98.05
80.00%
87.55
64.00%%
80.20
40.96%
73.21
16.78%
73.28
6.87%
(4.40%)
32.77%
13.42%
BU tickets (3.641%)
96,80
2.81%
88.90
3.11%
81.37
3.40%
74.66
3.64%
73.31
3.64%
Table A6: Evaluation performance of TR-B TR-BU/TD tickets when training incrementally on CIFAR-100.
Compact Weights
(56
Imih T|OT6
Ti-2 (5)
limile
Ti-s (S))
CIFAR-100
limils T
Ti-4 (9)
limiia
Ti-s (8)
limlic
MT MTppperr
Cll Clliaoee
87.40
100 00%
44.65
100.00%
28.67
100.00%
20.08
100.00%
17.44
100.00%6
Full Model
88.30
100 00%
74.90
100,00%
63.70
100.00%6
53.58
100.00%
48
100.00%
TD Tickets (12.08%)
88.34
80.00%
71.60
64.00%
58.80
51.2%
48.88
40.96%
43.60
32.77%
TD Tickets (9.66%)
88.20
80.00%
71.50
64.00%
58.73
51.2%
48.45
40.96%
44.08
32.77%
BU (7.20%)
5.50%
5.78%
6.07%
6.07%
6.07%
Compact Weights
mwlb
Te..r (96)
[]melo
Ti-s (92)
Ti (%)
T (9)
MT upger ClLivarr
74.11
100.00%
14,07
100.00%
12.64
100.00%
10.91
100.00%
9.66
100.00%
8.64
100.00%
Full Model
45.82
100.009
44.07
100.00%
42.35
100.00%
40.93
100.00%
39.36
100.00%:
Tickets 08%)
16.78%
22.08%
TD Tickets (9 66%)
41.18
26.21%
39.37
20.97%
38.06
16.78%
37.90
13.42%
36.72
9 9.66%
BU Tickets (7.20%)
47.58
6.35%
43.59
6.63%
41.35
6.63%
39.80
6.92%
39.79
7.20%
Table A7: Evaluation performance of TR-BU/TD tickets when training incrementally on Tiny- ImageNet.
CIFAR- 100
Compact Weights
(96
[[mile
Ti-2 (%)
Ti-a (%)
(10)
Ti. (8)
[[mlio [elia
Tius (%)
Hel
Full Model
73,70
100,00%
59.60
100.00%
52.07
100.009
43.85
100.00%
41.32
100.00%
Tickets (12.08%)
10.74%
11.01%
11.28%
11.28%
11.28%
Compact Weights
Meelle
Tiur (8))
Ti-a (9)
Ioolls
Ty (%)
Linii
(%)
Full Model
37.32
100.00%
34.69
100.00%
30.23 32.41
100.00%
29.94
100.00%
28.29
100.00%
BU Tickets (12.08%)
36.93
11.28%
36.43
11.54%
11.54%
29.16
11.81%
28.33
12.08%
100
100
100
Pruning Schedule of 6.07%
Curriculum Schedule of 10. 74%
Srandara PCC Accuarry (SA) %
Uniform Schedule 10.74%
940
Rema nina Weight 9%
Remaining Weight %
Standard Accuarty
60
80
80
40
40
70-
Tiickts Teaching
20
Curriculum Schedule (TD Tickets)
Tickets w.o. Lattery Teaching
Uniform Schedule (TD Tickets!
60
The number of Leamed Tasks
The number of Leamed Tasks
Figure A7: Left: the results of TD Tickets with/without lottery teaching. Right: the comparison of TD tickets (10.74%) obtained from uniform and curriculum pruning schedule. Experiments are conducted on CIFAR-10.
classes per task); Table presents the results of TR-BU10 tickets incrementally learn from 10 tasks (10 classes per task). Comparing between two tickets, TR- BU- tickets reach 6.55% higher accuracy at the expense of 1.77% more parameters. Possible reasons behind it are that: i) the increasing of incremental learning times aggravates the forgetting issue, which causes TR- BUg tickety fall in
A16

---

Published as a conference paper at ICLR 2021
Table A8: Evaluation performance of TR-BU Tickets when models incrementally learn 20 tasks on CIFAR-100.
Compact Weights
7i (58
Ti-2 (%)
1/m
[]]]s
Ti-a (%)
CIFAR-100
[mmll
19
Ti-4 (%)
[|]]|
m
Ti.o (96)
|
Full Modkl
100.00%
100.00%
100.00%
100.00%
52.16
100.0005
TR-BU Ticketr (5.43%)
86.80
2.81%
75.20
3.11%
66.47
3,40%
60.75
3.40%
53.68
3,40%
Compact Weights
Ti-. (%)
[imila
Ti-r (%)
1m
1001
Ti (8)
limila
Ti-o (86
||0||
mm
Fi-co (5)
limila
Full Model TR-BU Tickets (5.43%)
50.10
100.00% 3.40%
46.80
100.00%
43.35
100.00% 3.98%
41.71
100.00%
38.62 38.70
100,00%
3.69%
44.10
43.29
4.27%
4,27%
Compact Weights
TI (96)
llmll:
7i (%)
Ti-as
[tealls
Ti-14 (%)
Ti-us
T
Full Model
36.38
100.00%
35.77
100.00%
35.14
100.00% 4.27%
34.66
100.0005
35.41
100.00%
TR-BU (5.43%)
4.27%
4,27%
34,62
33.87
4,27%
35,48
Compuct Weights
Ti-as (98)
Ti (%)
To-as (9)
leuilla
T8P
Te-10 (%)
TiP
Ti-2o (%)
Full Model
34.43
100,00%
33,98
100.00%
34.14
100.00% 5.14%
32.65
100.00%
33.13
100.0094
TR-BU Ticketr (5.43%)
34.64
4.56%
34.13
4,85%
33,69
31,84
5.14%
33.24
5.43%
worse accuracy decay; ii) at each incremental stage, TR-BU10 tickets learn more knowledge (10 v.s. 5 classes per task), which requires a large network capacity.
With v.s. Without Lottery Teaching Comparison results between TD tickets with lottery teaching and the ones without lottery teaching are collected in this section. As shown in Figure AA (left figure), the performance of TD tickets without lottery teaching (black dash curves), quickly falls into worse decay along with the times of incremental learning increase. After learning all tasks, utilizing lottery teaching obtains a 4.34% accuracy improvement on CIFAR-10. 10. It suggests that our proposed lottery teaching injects previous knowledge through applying knowledge distillation external unlabeled data, and greatly alleviates the catastrophic forgetting issue.
A2
MORE METHODOLOGY AND IMPLEMENTATION DETAILS
2
MORE LIFELONG PRUNING DETAILSS
Rardoon Initializaiion Consecion
J: Weights / Connection from T.
> Weightss Connection from T
Weights / Connection from Tu.s
Tup-down Lifelong Pruning
Siee
Ti
Ta
Ta
Ta
IMP
IMP
g
g
2 (e3)
Figure A8: Framework of our proposed top-down lifelong pruning algorithms. The top-down (TD) lifelong pruning performs like iterative magnitude pruning (IMP) by unrolling the sequential tasks. Tickets located by TD pruning continue shrink with the growth of incremental tasks.
More Technical Details of Top-down Pruning In our implementation, we set pat 20% as (Frankle & Carbin 2019 Renda et 2020) and adjust (n( to control the pruning sched- ule of IMP over sequential tasks. The atorementioned lifelong pruning method is illustrated in Figure A8 and we call it top-down lifelong pruning since the model size is sequentially reduced, namely, ( l10 $
Pruning Algorithms ing in Algorithm|] [ and respectively. For pruning hyperparameters, we follow the original LTH's setting (Frankle & Carbin 2019); i.e. Ap 20%. If we change Ap to 40%, it will drop 2.04% accuracy at the same sparsity level on CIFAR-10.
We summarize the workflow of the top-down pruming and bottom-up prun-
A2.2 MORE CLASS-INCEEMENAAI LEARNING DETAILS
Lottery teaching regularization In order to mitigate the catastrophic forgetting effect, we apply knowledge distillation (Hinton et al. 2015) RKD enforce the similarity between previous ý and
A17

---

Published as a conference paper at ICLR 2021
current soft logits unlabeled data. We state RKD as follows:
RKD Ø) = -((((v), t(())
= Σ ((y), logt(g), (());
(g,) 1//T
where t(y): Σ 1/T,
= our case, following the standard setting in (Hinton et al. 2015
Li & Hoiem 2017).
Algorithm 1: Top-Down Pruning
Algorithm 2: Bottom-Up Pruning
Input:
Full dense model f(O,, 0! xx), desired sparsity P... samples from a storage S and sequential tasks Ti-n. soft logits from previous model on queried unlabeled data, pruning ratio Ap Output: An updated sparse model
Input: f@g. 6 x), Pm x, soft logits and Ap defined in Algorithm [
f0,, 0! x) has learned Timi and has performance Ri i E {1,
Output: An updated sparse model
f(O @ m, 0(")
Set i 1 and mask m 0 E Rllole
f(O m, )!") :x)
Train f(0, @ m, [ :x) with data from S
Set i and mask m
and T.. Calculate accuracy R1.
Train f(0. c x) with data from S and Ti. while |m < Pm and i S n do 0
while i < and |milo < |0 do
if R. 2 R2 or |m|le = 0 then
Continue
Iterative weight magnitude (IMP) pruning Ap and obtaining new mask m, where ||m/||u < <|ml|o
else
Start from f(e,, 0 0x) x). m =
repeat
Rewind weight to 0g
pruning Ap of 0, ) (m m). obtain new mask m." where
m=m
Retrain f(e, O m, 0 x) on the current task Ti and S. Lottery teaching is applied (A knowledge distillation constrain with soft logits)
|| 110 ||miloo and m m
Retrain f(0,-1 O m o. xx) and calculate accuracy R, m = m
10
11
12
until R, ~ R2 and set m = m"
Set i+11
end
end
Set = i+1
end
Our Dense Full CIL Model We consider a strong baseline framework derived from (Zhang al. 2019) with our proposed lottery teaching as our dense full CIL model. It adopts random and class- balanced sampling strategies, auxiliary classifier, and the knowledge distillation regularizer RKD- For incrementally learning task Tr, the training objective is depicted as:
0o !0)) (( 72 x C(O. C(0.0 00) + (()
((0,0!)) Exxy)ED, [xE(ff() !0), x),
m EXED,
[RKD(F(0,0) 000 x). Uc)
C(0,0 (0) = (xx))E, (xe(f(O, (00) x),
+ X EXED. RKD( 0 x), Vc)]
where D, is the class-balanced sampled dataset, D, represents the randomly sampled dataset, and
D, stands for the queried unlabeled dataset. g. and e! are the main and auxiliary classifiers. ÿe and S. are soft logits on previous tasks the main and auxiliary classifiers. We adopt 7 = 1, 0.5 in experiment according grid search.
A18

---

Published as a conference paper at ICLR 2021
A2.3 MORE OTHER IMPLEMENTATION DETAILS
Datasets and Task Splittings We evaluate our proposed lifelong tickets on CIFAR-10. 10, CIFAR- 100, and Tiny-ImageNee datasets, all being standard and state-of-the-art benchmarks for CIL Krizhevsky & Hinton] 2009). For all three datasets, we randomly split the original training dataset into training and validation with ratio of 9 On CIFAR-10. we divide the 10 classes into splits of 2 classes with a random order (10/2 = 5 tasks); On CIFAR- 100, we divide the 100 classes into splits of 10 classes with a random order (100/10 = tasks); On Tiny- Iima-emagenet, we divide the 200 classes into splits of 20 classes with random order (200/200 = 10 tasks). In this way, when models learn new incoming task, the dimension of classifiers will increase by 2 for CIFAR-10, 10 for CIFAR-100, and 20 for Tiny-ImageNel Additionally. 100 images, 10 images, and 5 images per class of learned tasks will be stored for CIFAR-10, CIFAR-100, and Tiny- ImageNet respectively.
from 80 Million Tiny Image dataset (Torralba et al., [0088) and for Tmny-ImageNet are from ImageNet dataset Krizhevsky al. 2012). At each incremental learning stage, 4, 500, 450 and 450 images per class of learned tasks will be queried, based on the feature similarity with stored prototypes (m(1-1) ((111), 0 in top-down Pruning and (0(--1), - in bottom-up Pruning the ith CIL stage for CIFAR-I0, CIFAR-100, and Tiny-ImageNee respectively. The feature similarity is defined by & norm distance.
Unlabeled Dataset All queried unlabeled data for COTAR-OO/CIAA--00
Training and Evaluation Models are trained using Stochastic Gradient Descent (SGD) with 0.9 momentum and 5 10 weight decay. For 100 epochs training, a multi-step learning rate schedule is conducted, starting from 0.01, then decayed by 10 times at epochs 60 and 80. During the iterative pruning, we retrain the model for 30 epochs using a fixed learning rate of 10-4, The batch size for both labeled and unlabeled data is 128. We pick the trained model of the highest validation accuracy and report their performance on the hold-out testing set.
Other Training Details (i) CIFAR-I0 and CIFAR-100 can be download thtttps: www. cs Ecronto. edu/ www//kkrii///eaee /ttaz//ccaaar./ra html (ii) 80 Million Tiny Image dataset is referred to thttp: //horatic cs nyu. edu wwtpeettttttt//////////////tattt
htt/ex.eeeel (iii) All of our experiments are conducted on NVIDIA GTX 1080-Ti GPUs.
A. DISCUSSION
Challenges of Theoretical Analysis and Future Work The theoretical justificatiin of the lot- tery ticket hypothesis is very limited, except for very shallow networks (Anonymous) (02T). In the meantime, class-incremental learning makes the theoretical analysis more difficult. It is chal- lenging lifelong learning problem, and the current progress lies in the empirical side rather than the thcoretical side. The theoretical analysis is out of scope for this paper and we would like to explore it in the future.
A19


Reviews:
[{"review_id": "evHg6vPdZ6q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[EDIT AFTER DISCUSSIONS] I thank the authors for their answers to my comments. Having read the various threads, I confirm my score and see interesting work in this paper.\n[/EDIT]\n\n##########################################################################\nSummary: this paper extends the lottery ticket hypothesis to life-long learning. The paper proposes a top-down and bottom-up approach to network pruning and shows that the bottom-up pruning reaches SOTA performance on several datasets while reducing the network size to a few percent of the full model size. The paper also shows higher performance against SOTA for class-incremental learning. \n\n##########################################################################\nReasons for score: To the best of my knowledge, this paper brings novel contributions to the community.  The approach is sound, well-explained, and evaluated rigorously.  The Open Questions section presents open questions but these do not represent a blocker to publication in my opinion.\n\n##########################################################################\nPros: The paper has the following advantages:\n\n- Novelty: to the best of my knowledge, the paper brings a novel contribution to the problem of LTH for life-long learning\n\n- Clarity: the paper is well-structured, clear, and easy to read\n\n- Rigor: the work presented in the paper is rigorous. An ablation study is included and several in-depth analysis are presented. Due diligence has been done on experimental setup. The appendix contains numerous experimentation details (providing code would be even better)\n\n- Impact: the paper brings a significant contribution to the literature by beating or reaching SOTA on lifelong learning.\n\nThere are several open questions in the Rebuttal section, which, according to me, should not challenge my score.  However, I am interested in the opinion of the authors and other reviewers on these questions.\n\n##########################################################################\nCons: I do not see major limitations to this paper, apart from the code not being released, which reduces the reproducibility of the paper.  This section contains only minor editorial recommendations.\n\nThe first sentence of the abstract should read \"The lottery ticket analysis states that...\" instead of \"demonstrates that...\" since a hypothesis cannot \"demonstrate\" something.  It is a minor detail but since this is the first sentence of the paper, it has a significant impact on the reader's impression of the paper.\n\nTypo page 3 in \"can be trained same well in isolation\" (this phrase does not make sense)\n\nTypo page 4 in \"Why we need beyond top-down pruning\" (this phrase does not make sense)\n\nGraphs on Figure 3 (and in the Appendix) are hard to read for small values of remaining weights.  Many scaling the x-axis differently would help.\n\nTypo page 6 in \"learning the rest three tasks\" (does not make sense)\n\nPage 6, the sentence \"Therefore, the bottom-up lifelong pruning debuts, ...\" does not make sense\n\nThe Related work section is put at the end of the paper, which can make sense (some paper do this regularly).  However, for this particular paper, I would tend to think that moving it up in the paper (close to the beginning) could make sense too.\n\n\n##########################################################################\nOpen questions\n\nMy understanding when reading page 4 section \"Curriculum schedule\" is that TD pruning requires the knowledge of the number of tasks.  Is that correct?  How would it extend to an unlimited or unknown number of tasks?\n\nThe rewinding point approach seems to require maintaining the full model in parallel to the optimized one.  If that is true, it seems to defeat the purpose of optimizing the model.  Am I missing something?  Also, in the real world, this could have memory implications that could make the approach less practical.\n\nThe ticket size seems not to have a theoretical upper bound in this approach.  Is this correct?\n\nResults seem out of noise for most experiments, but it would be nice to have confidence intervals, in particular for the claim that TR-BU outperform the dense model by 0.52% (page 6)\n\nFrom Figure 4, it seems that the ticket sizes seem to converge for TR-BU and TD as the number of tasks grows.  Is that what is happening?  Any theoretical analysis of this?\n\n#########################################################################\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S9WnPYbFwl", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\n \nThe paper provides an interesting extension of the lottery ticket hypothesis in the lifelong learning setup, showing the existence of these tickets for class incremental learning. The paper also explores top-down and bottom-up tickets. The authors performed experiments on CIFAR10,CIFAR100, and Tiny-ImageNet datasets showing the effectiveness of the proposed ticket strategy.   \n\n\n##########################################################################\n\nReasons for score: \n\n \nOverall, I am leaning positive. I find the idea of investigating lottery tickets in continual learning is valuable to study and understand. My major concern is about the clarity of the paper and some additional experiments (see cons below). Hopefully, the authors can address my concern in the rebuttal period. \n\n \n##########################################################################\nPros: \n\n \n1. The paper focuses on one of the most important machine intelligence tasks, continual learning and more specifically class incremental learning, and studies how to find winning lottery tickets inspired from lottery ticket hypothesis paper (Frankle, Carbin, 2019). \n\n \n2. The authors proposed top-down and bottom-up tickets as strategies to find these tickets and showed that can perform on bar and sometimes better compared to the full mode.\n \n3. This paper provides experiments on CIFAR10, CIFAR100, min-ImageNet including quantitative results, to show the effectiveness of the proposed initialization. \n\n \n##########################################################################\n\nCons: \n\n\n1. The paper uses episodic memory, a small number of examples from the previous setting. It is not clear whether this observation would generalize to regularization or generative approaches that do not require so. Meaning, lifelong learning methods are desired not to assume access to previous task data (i.e., regularization based approaches like EWS[R1], LWF[R10], Intelligent Synapses[R9], MAS[R3], UCB[R0], generative approaches includes [R6, R13])\n\n2. Related work section can be enriched. There is a lot of work that has been done in continual learning. \n\na) I attach below some representative references but it will be position this work within regularization, memory-based, and structural continual learning methods. \n\nb) it will be good to also show experimentally how these CIL tickets may generalize in regularization approaches and/or generative approaches mentioned above. Currently, experiments are restricted to iCaRL and IL2M. \n\n3) It seems that iterative CIL pruning requires retraining of the entire sequence. Does not this seem to break the natural setup of revisiting previous task data, again and again even if we prune only once?  I understood that however the goal is to probably show the existence of these CIL tickets but it is still not so clear how to make this more practical and realistic in CIL setting. \n\n \n##########################################################################\n\nQuestions & Clarifications\n--------------------------------------\n1) continual pruning algorithm A may assign -1,0,1 but some of these weights may not make sense based on m^(i-1). For example, if a mask in some location is already 0, producing -1 is not valid. In the context of this paper, do the authors observe the case of -1, when? will be good to analyze these learning dynamics during the continual learning process. \n\n2) \"we find that the schedule of IMP over sequential\ntasks, in terms of fn(i)g, is critical to make pruning successful in lifelong learning\" could you elaborate on this experimentally? what went wrong in some of your experiments for other choices and why?   \n\n\n3) it could be cleared to update Algorithm 1 and 2 to include the tasks loop. \n\nminor\n--------\n1) SA is sometimes confusing. In some cases, it is spelled out. sometimes not, will be good to fix it. \n2) \"Compared BU with TD pruning, TR-BU tickets surpass the best TD tickets.\" \nComparing? TR-BU abbreviation is not defined before this point but defined later. It will be nice to clarify. \n\n[R1] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., ... & Hassabis, D. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13), 3521-3526.\n\n[R2]Zenke, Friedemann, Ben Poole, and Surya Ganguli. \"Continual learning through synaptic intelligence.\" Proceedings of machine learning research 70 (2017): 3987.\n\n[R3]Memory Aware Synapses: Learning what (not) to forget. (ECCV’18)\n R Aljundi, F Babiloni, M Elhoseiny, M Rohrbach and T Tuytelaars \n\n[R4]Exploring the Challenges towards Lifelong Fact Learning.  (ACCV’18)\nM Elhoseiny, F Babiloni, M Paluri, R Aljundi, M Rohrbach and T Tuytelaars\n\n[R5]David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" In Advances in neural information processing systems, pp. 6467-6476. 2017.\n\n[R6]Shin, Hanul, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. \"Continual learning with deep generative replay.\" In Advances in Neural Information Processing Systems, pp. 2990-2999. 2017.\n\n[R7]Rusu, Andrei A., Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. \"Progressive neural networks.\" arXiv preprint arXiv:1606.04671 (2016).\n\n[R8]Efficient Lifelong Learning with A-GEM  (ICLR’19)\n        Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, Mohamed Elhoseiny\n\n[R9]Uncertainty-guided Continual Learning with Bayesian Neural Networks (ICLR’20)\n        Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, Marcus Rohrbach\n\n[R10] Li, Zhizhong, and Derek Hoiem. \"Learning without forgetting.\" IEEE transactions on pattern analysis and machine intelligence 40.12 (2017): 2935-2947\n\n[R11] Kemker, Ronald, and Christopher Kanan. \"Fearnet: Brain-inspired model for incremental learning.\" arXiv preprint arXiv:1711.10563 (2017).\n\n[R12] Hayes, Tyler L., and Christopher Kanan. \"Lifelong machine learning with deep streaming linear discriminant analysis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020.\n\n[R13] Liu, Xialei, et al. \"Generative Feature Replay For Class-Incremental Learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n\n\n \n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HTWfXDnTjpA", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The research question of this paper is the existence of an extremely sparse network with an initial weight assignment that can be trained online to perform multiple tasks to compete with a dense network, in a lifelong continual learning configuration. Another research question of this paper is how to identify this sparse network and achieve competitive performance. To address these questions, the authors proposed to incrementally introduce new non-zero weights when learning incoming tasks (Figure 2 and Equation 1). The network considered by the authors has a common base for all models and a head for individual tasks.\n\nWhile the topic that combines lifelong learning and network sparsity is definitely interesting, the development of this paper is incremental,  and there lacks some theoretical justification on why introducing a new task will both keep the network sparse and reuse weights of the previous networks. So the paper is slightly below the acceptance threshold for now.\n\nPros:\n\n+ Interesting topic.\n+ The proposed schema works, as shown in the experiments.\n\nCons:\n\n- There needs work to satisfactorily define the new lifelong winning ticket framework. For example, in a multi-tasks configuration, why can a new task be learned by adding sparse non-zero weights (+retraining) with the performances of previous tasks preserved? If we simply rely on restarting from scratch when adding sparse new weights doesn't do the work, then what is the contribution? In what situation do we need to restart from scratch and in what situation can new tasks be learned from the current sparse network incrementally? \n- Lack of theoretical and experimental justification about how the proposed concept will hold or fail.\n\n To improve the paper, I would like to see more experiments, and preferably theoretical discussions.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work extends the lottery ticket hypothesis to lifelong learning and, in particular, it tackles the problem of class incremental learning. This is an important and difficult problem, and of great interest to the community. The authors considered top down and bottom-up pruning strategies. The proposed approaches were validated on existing benchmarks (CIFAR10,CIFAR100, and Tiny-ImageNet), reaching state-of-the-art results, and showing that catastrophic forgetting could be alleviated. While some questions remain in terms of practical relevance, they authors showed the existence of winning tickets in the continual setting. There were concerns regarding clarity and requests for additional experiments, but all were convincingly addressed and the clarifications provided by the authors in their rebuttal further strengthened the paper.
Decision: Accept (Poster)