SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qiang Gao $^{1,\ast}$ , Xiaojun Shan $^{2,\ast}$ , Yuchen Zhang $^{2}$ , Fan Zhou $^{2,\dagger}$
$^{1}$ Southwestern University of Finance and Economics, Chengdu, China
$^{2}$ University of Electronic Science and Technology of China, Chengdu, China
qianggao@swufe.edu.cn, {xiaojunshan@std., yuchenzhang@std., fan.zhou@}uestc.edu.cn
As there exist competitive subnetworks within a dense network in concert with Lottery Ticket Hypothesis, we introduce a novel neuron-wise task incremental learning method, namely Data-free Subnetworks (DSN), which attempts to enhance the elastic knowledge transfer across the tasks that sequentially arrive. Specifically, DSN primarily seeks to transfer knowledge to the new coming task from the learned tasks by selecting the affiliated weights of a small set of neurons to be activated, including the reused neurons from prior tasks via neuron-wise masks. And it also transfers possibly valuable knowledge to the earlier tasks via data-free replay. Especially, DSN inherently relieves the catastrophic forgetting and the unavailability of past data or possible privacy concerns. The comprehensive experiments conducted on four benchmark datasets demonstrate the effectiveness of the proposed DSN in the context of task-incremental learning by comparing it to several state-of-the-art baselines. In particular, DSN enables the knowledge transfer to the earlier tasks, which is often overlooked by prior efforts.
Continual learning (CL), as one of the human-like lifelong learning or incremental learning paradigms, has received enormous attention in Artificial Intelligence (AI) community due to its capability to incrementally learn a sequence of tasks in a (deep) neural network and to keep accumulating knowledge throughout its lifetime [1]. According to the context of whether the task identity is provided during model training and inference, the majority of prior works categorize continual learning into three main practical problems by specific settings, including task-incremental learning, class-incremental learning, and task-free incremental learning. This study was prepared for Task Incremental Learning or TIL for short. Not limited to TIL, existing solutions for incremental learning are put forward to solve the catastrophic forgetting (CF) issue [2] that is a common phenomenon in CL scenarios. Usually, these approaches can be broadly categorized into three directions, i.e., regularization-based [3, 4, 5], rehearsal-based [6, 7, 8], and architecture-based [9, 10, 11, 12].
Motivations: Recently, several studies have demonstrated that deep neural networks are usually over-parameterized, whereby the redundant or unused weights can be pruned, allowing efficient computation and on-par or even better performance [13, 14, 15]. Likewise, researchers stated this phenomenon as Lottery Ticket Hypothesis (LTH) [16, 17], i.e., a randomly-initialized neural network contains a subnetwork such that, when trained in isolation, can match the performance of the original network. With this in mind, more recent efforts turn to randomly initialize a large but sparse neural network and employ it to sequentially discover compact subnetworks for (task) incremental learning,
†Corresponding author (fan.zhou@uestc.edu.cn).
aiming at providing room to learn new tasks as well as enabling forward knowledge transfer from the prior tasks [18, 19, 20]. Specifically, they concentrate on weight-wise search using an adaptive binary mask that determines which weight should be activated (winning tickets). Nevertheless, we consider that it is not preferable to maintain such a binary mask for retrieving a subnetwork, especially in incremental learning. Since it will destroy the continuous structure of a layer-dependent subnetwork. Meanwhile, maintaining a binary mask whose size equals the number of parameters also results in significant resource consumption. In addition to this, knowledge transfer across different tasks is the core work of incremental learning. Existing solutions primarily seek to enable the knowledge transfer from earlier tasks to the new coming task (i.e., forward knowledge transfer), along with considering the CF problem [21, 22, 23]. Like human cognitive processes [24], we debate whether the newly learned knowledge has the potential to help improve past tasks (i.e., backward knowledge transfer). Intuitively, it is natural and practical that we can replay the old samples from earlier tasks, like rehearsal-based approaches, to trigger the ability to tackle old tasks much better. However, this is not applicable in the actual incremental learning scenario due to computational overhead and privacy concerns. With those in mind, we ask a much more ambitious question: Does LTH hold in the setting of TIL concerning elastic knowledge transfer when different tasks arrive sequentially?
Contributions: To answer this question, we propose a novel and practical approach named Data-free Subnetworks (DSN), enabling the elastic knowledge transfer across the tasks that sequentially arrive. Motivated by LTH, we assume that there exists a hypernetwork (Fig. 1 (a)) that contains a sequence of competitive "ticket" sub-networks, where each of them can perform well on the affiliated task with the knowledge transfer in mind. DSN mainly contains two components, i.e., neuron-wise mask and data-free memory reply. Specifically, we first randomly initialize a hypernetwork and incrementally learn the model parameters and task-specific masks, where each mask determines which neurons and their corresponding weights should be used for a new coming task (Fig. 1 (b)). To allow the forward knowledge transfer on a new task, we use the mask to adaptively select the neurons that have been used in the earlier tasks and make their corresponding weights unchangeable for the purpose of addressing the CF problem. To permit the backward knowledge transfer, we first measure the mask similarity scores and craft the impressions of the most similar task via data-free memory replay. In contrast to sample replay paradigms, we treat the subnetwork as a past experience and use it to recall past impressions, avoiding incrementally preserving the past actual samples. Then, we fine-tune the most similar task and make backward knowledge transfer possible(Fig. 1 (c)). We summarize our contributions as follows:
- In concert with Lottery Ticket Hypothesis (LTH), we introduce a novel and practical task-incremental learning solution DSN, which aims to learn a compact subnetwork (winning ticket) for each task from the perspective of knowledge transfer, including forward and backward transfer.
- We devise a neuron-wise mask mechanism to adaptive select neuron-affiliated weights to transfer the learned knowledge to the new task, where the used neuron-affiliated weights in the past tasks are frozen to eliminate the CF problem. Besides, our proposed data-free replay mechanism regards the trained subnetwork as a past experience and uses it to craft impressions regarding the past samples, which does not require holding any actual samples related to past tasks.
- The comprehensive experiments conducted on four benchmark datasets demonstrate the effectiveness of our proposed DSN against the state-of-the-art baselines.
Catastrophic Forgetting. Existing approaches to address the CF problem can be broadly categorized into three directions. (1) Regularization-based approaches that add a selective penalty in the loss function, which penalizes the variation of network parameters depending on its importance in performing previous tasks [3, 25, 4, 26, 27]. EWC operates the Fisher information matrix to estimate the importance weights [3]. Synaptic Intelligence (SI) provides an online approximation of parameter importance by assessing their contributions to the variations of total loss [28]. (2) Rehearsal-based approaches typically incrementally hold a few representative old training samples as a small memory buffer and replay it to retain the past knowledge when learning a new task [6, 7, 8, 29, 30, 31]. For example, GEM proposes to further guarantee an equal number of old training samples per class [32]. DGR presents a framework that replays generated data sampled from the old generative model while learning each generation task to inherit the previous knowledge [6]. Also, other
(a) Dense Initialization
(b) Selection for Task  $t$
Figure 1: Task Incremental Learning with DSN: (a) We define the task network as a hypernetwork that can maintain the training of multiple tasks. And it will be randomly initialized (blue line) for task 1 training (pink circle and line); (b) We use the Neuron-wise Mask to produce a subnetwork for task  $t$  training (purple circle and line); (c) If task 1 is the most similar to task  $t$  by similarity measurement of masks, we update the architecture of task 1 based on the performance of Data-free Replay.
(c) Consolidation for Task 1
researchers [33, 34, 35] perform data-free generative replay to alleviate memory space or privacy concerns. For instance, [34] employs two teacher models to enhance the quality of generated samples. However, it requires an additional generative model, yielding additional trainable parameters. [35] can generate old samples merely through the main network. Besides, [33, 34, 35], which directly employ synthesized samples to fine-tune the model, have limitations in achieving forget-free performance and are susceptible to interference from other tasks. (3) Architecture-based approaches assign different model parameters to different tasks, freeze the parameters for old tasks by masking them out when training the new task, and expand the network when necessary [10, 11, 12, 8, 19]. DEN [36] can dynamically adjust its network capacity as it trains on a sequence of tasks to learn a compact overlapping knowledge-sharing structure among tasks. WSN aims to explore the subnetworks sequentially for a sequence of tasks from a sparse neural network [19]. It also maintains a binary mask whose size equals the number of parameters, resulting in significant resource consumption. We argue the architecture-based approaches provide us with the possibility to explore better subnetworks while they either confront the redundant neurons (weights) or shallow knowledge transfer.
Knowledge Transfer. Recently, researchers and practitioners have argued that the ability to learn from a sequence of tasks is in essence to complement and reinforce each other through knowledge transfer. Several studies have proposed to improve task performance by maximizing shareable knowledge [37, 38, 39, 40]. For instance, Bayes model [41, 42] and regression method [43] concentrate on transferring the learned knowledge to facilitate learning in the new domains or new tasks. BNS [12] presents a Task-CL algorithm that can adaptively determine the network structure for each new task and selectively transfer previously learned knowledge. However, it employs massive past samples to address the negative knowledge transfer concern. HAT [44] and WSN [19] selectively reuse the optimized parameters via a binary masking mechanism, enabling significant forward knowledge transfer. However, backward knowledge transfer is not considered. [45] focuses on a few-shot scene, which is accomplished by jointly learning model weights and adaptive non-binary soft masks, with the major subnetwork minimizing catastrophic forgetting and the minor subnetwork preventing overfitting. Also, it fails to take backward knowledge transfer into consideration. [46] proposes two distillation-based objectives for class incremental learning. This method can utilize the feature space structure of the previous model to preserve the representation of previous classes. Besides, a controlled transfer is introduced to approximate and condition the current model on the semantic similarities between incoming and prior classes, maximizing positive forward transfer and minimizing negative backward transfer. However, it can not achieve a positive backward transfer. [38] and [47] preserve previous data as a replay buffer for retraining old tasks, which raises concerns regarding data privacy and memory overhead. [38] utilizes a CL-plugin to explicitly identify and transfer useful features/knowledge from previous tasks to new ones, designed specifically for LLMs. [48] characterizes task correlations, identifies positively correlated old tasks, and selectively modifies the learned model of the old tasks when learning new tasks. In sum, the above solutions concentrate on either addressing severe negative transfer using pruning methods or using additional data to help knowledge consolidation. In contrast, our DSN attempts to select subnetworks from a sparse network for a sequence of tasks while using a data-free mechanism to enable the possibly positive knowledge transfer to the old tasks.
Without loss of generality, a classical TIL scenario, typically in a supervised learning setting (e.g., classification), deals with a set  $D = \{D_{1}, D_{2}, \dots, D_{T}\}$  of  $T$  tasks that come in turn to the learner or task network (e.g., a deep neural network). Let  $D_{t} = \{(x_{t}^{i}, y_{t}^{i})\}_{i=1}^{n_{t}}$  denote the dataset of task  $t$ , containing  $n_{t}$  pairs of raw instances and independent label space. In our context, assume that we have a neural network, named hypernetwork  $\mathcal{H}$ .  $\mathcal{H}(\cdot, \theta(\boldsymbol{n}))$  is composed of layered neurons  $\boldsymbol{n}$  where learnable weights  $\theta$  bridge them together. TIL follows the common continual learning paradigm and attempts to sequentially learn a set of task  $T$  with the following objective for each task  $t$ :
$$
\boldsymbol {\theta} ^ {*} = \underset {\boldsymbol {\theta}} {\operatorname {m i n i m i z e}} \frac {1}{n _ {t}} \sum_ {i = 1} ^ {n _ {t}} \mathcal {L} \left(\mathcal {H} \left(x _ {t} ^ {i}; \boldsymbol {\theta}\right), y _ {t} ^ {i}\right), \tag {1}
$$
where  $\mathcal{L}(\cdot, \cdot)$  is a loss function such as cross-entropy loss. It is important to note that  $D_{t}$  for task  $t$  is only accessible during task  $t$  learning, and any instances in  $D_{t}$  cannot be accessed while learning future tasks. Furthermore, assume that task identity is given in both the training and inference processes in the TIL scenario. In addition, we follow most of the previous studies and employ multi-head settings for TIL, with each task having its own single but non-overlapping head.
The Lottery Ticket Hypothesis (LTH) hypothesizes the existence of competitive subnetworks in a randomly initialized dense neural network [16], which motivates us to ingeniously devise an incremental learner that employs an over-parameterized deep neural network to allow more room for future task learning. Therefore, our goal is to discover an optimal subnetwork along with a neuron-wise mask  $m_t \in \{0, 1\}^{|\mathbf{n}|}$  for each task  $t$  such that  $|m_t^*| \ll |\mathbf{n}|$ , summarized as follows:
$$
\boldsymbol {\theta} ^ {*}, \boldsymbol {m} _ {t} ^ {*} = \underset {\boldsymbol {\theta}, \boldsymbol {m} _ {t}} {\operatorname {m i n i m i z e}} \frac {1}{n _ {t}} \sum_ {i = 1} ^ {n _ {t}} \mathcal {L} \left(\mathcal {H} \left(x _ {t} ^ {i}, \boldsymbol {\theta} (\boldsymbol {n} \odot \boldsymbol {m} _ {t})\right), y _ {t} ^ {i}\right) - \mathcal {L} \left(\mathcal {H} \left(x _ {t} ^ {i}; \boldsymbol {\theta}\right), y _ {t} ^ {i}\right), y _ {t} ^ {i}) \tag {2}
$$
$$
s. t. \left| \boldsymbol {m} _ {t} ^ {*} \right| \ll | \boldsymbol {n} |.
$$
3.2 Neuron-wise Mask
To obtain an optimal subnetwork from the hypernetwork  $\mathcal{H}$  for any new coming task, we are motivated by HAT [44] and inhibitory synapse [49] and devise a neuron-wise differentiable mask mechanism to search a small set of neurons with the objective Eq.(2). Specifically, for each layer in the hypernetwork  $\mathcal{H}$ , let each layer associated with a learnable embedding  $e_t^l$  ( $l \in \{1, 2, \dots, L\}$ ) before training a coming task  $t$ , where  $L$  is the number of layers. To determine if a neuron in  $\pmb{n}$  should be used for task  $t$ , we adopt the binary mask  $\pmb{m}_t$  to produce the architecture of the subnetwork. However, neurons typically have only two states, i.e., activated or not activated. To this end, we use the following function to bridge the gap between the layer embedding  $e_t^l$  and layer mask  $\pmb{m}_t^l \in \pmb{m}_t$ :
$$
\boldsymbol {m} _ {t} ^ {l} = \sigma (\gamma \cdot \boldsymbol {e} _ {t} ^ {l}), \tag {3}
$$
where  $\sigma$  is the sigmoid function.  $\gamma$  is the scaling factor to control the sharpness of the function that can make the binarized progressively [44, 50]. We note that the optimal mask  $m_{t}^{*}$  can be received by jointly optimizing the model parameters  $\theta$  of task  $t$ , as the task learning is conditioned on mask  $m_{t}$ . For instance, the forward propagation of hypernetwork  $\mathcal{H}$  during the training of task  $t$  can be expressed as follows:
$$
\text {f o r w a r d}: \boldsymbol {h} _ {t} ^ {l} = \boldsymbol {h} _ {t} ^ {l} \odot \boldsymbol {m} _ {t} ^ {l}, \tag {4}
$$
where  $h_t^l$  denotes the output of neurons in the layer  $l$ . Correspondingly, the backward propagation can be summarized as follows,
$$
\text {b a c k w a r d}: \theta_ {l i j} = \theta_ {l i j} - \frac {\partial \mathcal {L}}{\partial \theta_ {l i j}} \odot \max  \left(m _ {t} ^ {l, i}, m _ {t} ^ {l - 1, j}\right), \tag {5}
$$
where  $\theta_{lij} \in \pmb{\theta}$  refers to a parameter (i.e., weight) between the neuron  $i$  of  $l$ -th layer and the neuron  $j$  of  $(l-1)$ -th layer.  $m_t^{l,i}$  denotes the mask value of  $i$ -th neuron.  $\max(m_t^{l,i}, m_t^{l-1,j})$  indicates that the parameter will not be updated if neither the neuron  $i$  of  $l$ -th layer nor the neuron  $j$  of  $l-1$ -th
layer is activated for task  $t$ . Otherwise, the parameter will be updated. However, the above backward propagation inevitably results in the catastrophic forgetting issue since the weights trained in the previous tasks could be changed. Hence, we regard the used neurons in the previous tasks as the synapses that only take the role of sending messages between different layers. To this end, we devise a cumulative mask to maintain the signal that all previously learned parameters should be frozen. The cumulative mask  $m_{\leq t}$  for any task  $t$  can be recursively obtained as follows,
$$
\boldsymbol {m} _ {\leq t} = \max  (\boldsymbol {m} _ {\leq t - 1}, \boldsymbol {m} _ {t}). \tag {6}
$$
Note that we set  $m_{\leq 0} = 0$ , indicating all neurons can be used before training task 1. Thus, the Eq.(5) can be revised as follows,
$$
\text {b a c k w a r d}: \theta_ {l i j} = \theta_ {l i j} - \frac {\partial \mathcal {L}}{\partial \theta_ {l i j}} \odot \max  \left(m _ {t} ^ {l, i} \odot \left(1 - m _ {<   t} ^ {l, i}\right), m _ {t} ^ {l - 1, j} \odot \left(1 - m _ {<   t} ^ {l, j}\right)\right). \tag {7}
$$
It is worth noting that mask operation may also pose capacity problems. To this end, we add a regularization term  $\mathcal{L}_r$  to the objective  $\mathcal{L}$  to hold more room for future tasks as follows,
$$
\mathcal {L} _ {r} = \eta \frac {\sum_ {l = 1} ^ {L - 1} \sum_ {i = 1} ^ {n _ {l}} m _ {t} ^ {l , i} \left(1 - m _ {<   t} ^ {l , i}\right)}{\sum_ {l = 1} ^ {L - 1} \sum_ {i = 1} ^ {n _ {l}} \left(1 - m _ {<   t} ^ {l , i}\right)}, \tag {8}
$$
where  $n_l$  denotes the neuron number of  $l$ -th layer in hypernetwork  $\mathcal{H}$  and  $\eta$  is a hyperparameter that controls the capacity preference for the current task  $t$ . Note that we provide more details of the hyperparameter setting in Appendix A.
3.3 Data-free Replay
To enhance the knowledge transfer from the current task to the earlier tasks, data-free replay recalls network experiences to address the unavailability of old samples or possible data privacy concerns. Recent zero-shot learning solutions [51, 35] provide us with a new perspective. That is, we craft past knowledge through the subnetwork rather than seeking past samples.
Output Space Modeling. Let  $\pmb{o}_t$  denote the output space (i.e., Softmax space for classification problems) of task  $t$  over the subnetwork  $\mathcal{H}(\cdot, \theta(\pmb{n} \odot \pmb{m}_t))$ . Recent studies [51, 35] reveal that any output representation can be sampled from a Dirichlet distribution as its ingredients fall in the range of [0,1] and their sum is 1. Specifically, the distribution to represent the (Softmax) outputs  $\pmb{o}_t^c$  of  $c$ -th class can be modeled as  $\text{Dir}(C_t, \beta \times \alpha^c)$ , where  $c \in \{1, 2, \dots, C_t\}$  is the class index regarding task  $t$ ,  $\alpha^c \in \mathbb{R}^{C_t}$  is the concentrate vector to model  $c$ -th class,  $\beta$  is a scaling parameter [52, 51], and any real value in  $\alpha^c$  is greater than 0. It should be noted that there could exist interactive correlations between different classes in a single task since the samples related to different classes may be similar. To mitigate the risk of ignoring the inherent distribution of different classes of samples, we are inspired by [51] and additionally preserve a class similarity matrix  $M_t$  after training each task  $t$  to describe the correlation between different classes. As  $M_t$  of task  $t$  is a class-wise table, it will not bring the significant capacity issue. We detail the motivation of the similarity matrix in Appendix B.
Replay with Impression Craft. Due to the unavailability of old task samples, like the human memory mechanism, we attempt to synthesize the input of  $\mathcal{H}(\cdot ,\theta (\pmb {n}\odot \pmb {m}_t))$  , which can be denoted as an impression regarding a raw sample in an earlier task. To be specific, we synthesize an impression set  $\mathcal{I}_t$  based on the Dirichlet sampling. For any synthesized sample  $\hat{x}_t^{c,i}$  corresponding to  $c$  -th class in task  $t$  , we initialize  $\hat{x}_t^{c,i}$  to random noise, e.g., by sampling from a uniform distribution. And we optimize it with the following objective over a sampled outputs  $\hat{o}_t^c$  and the subnetwork  $\mathcal{H}(\cdot ,\theta (\pmb {n}\odot \pmb {m}_t))$  , i.e.,
$$
\hat {x} _ {t} ^ {c, i} = \underset {x} {\operatorname {a r g m i n}} \mathcal {L} _ {I C} (\mathcal {H} (\cdot , \boldsymbol {\theta} (\boldsymbol {n} \odot \boldsymbol {m} _ {t}), \tau), \hat {\boldsymbol {o}} _ {t} ^ {c}), \tag {9}
$$
where  $\tau$  is a temperature value [53] used in the output (Softmax) head. In this manner, we can successively produce impressions for each class. Intuitively, we can produce the impressions of different classes equally. However, the subnetwork may show different performances in different classes. That is, we need to consider the emergence of hard classes. Thus, we generate a biased number of impressions of different classes based on the accuracy performance. For each task learning, we report the error rate of each class and normalize them as the distribution of the sampling rate.