SlowGuess/NAACL_2021_OCR huggingface dataset:
Order	Task Sequence
1	ag → yelp → yahoo
2	yelp → yahoo → ag
3	yahoo → ag → yelp
4	ag → yelp → amazon → yahoo → dbpedia
5	yelp → yahoo → amazon → dbpedia → ag
6	dbpedia → yahoo → ag → amazon → yelp
7	yelp → ag → dbpedia → amazon → yahoo
Table 2: Seven random different task sequences used for experiments. The first 6 are used in Setting (Sampled). The last 4 are used in Setting (Full).
- Replay (Wang et al., 2019; de Masson d'Autume et al., 2019): Finetune model augmented with an episodic memory. Replay examples from old tasks while learning new tasks.
- Regularization: On top of Replay, with an L2 regularization term added on the hidden state of the classifier following BERT.
- MBPA++ (de Masson d'Autume et al., 2019): augment BERT model with an episodic memory module and store all seen examples. MBPA++ performs experience replay at training time, and uses K-nearest neighbors to select examples for local adaptation at test time.
- LAMOL (Sun et al., 2019): train a language model that simultaneously learns to solve the tasks and generate training samples, the latter is for generating pseudo samples used in experience replay. Here the text classification is performed in Q&A formats.
- Multi-task Learning (MTL): The model is trained on all tasks simultaneously, which can be considered as an upper-bound for continual learning methods since it has access to data from all tasks at the same time.
We use pretrained BERT-based-uncased from HuggingFace Transformers (Wolf et al., 2020) as our base feature extractor. The task generic encoder and task specific encoder are both one linear layer followed by activation function  $\text{Tanh}$ , their output size are both 128 dimensions. The predictors built on encoders are all one linear layer followed by activation function softmax.
All experiments are conducted on NVIDIA RTX 2080 Ti with 11GB memory with the batch size of
8 and the maximum sequence length of 256 (use the first 256 tokens if one's length is beyond that). We use AdamW (Loshchilov and Hutter, 2019) as optimizer. For all modules except the task id predictor, we set the learning rate  $lr = 3e^{-5}$ ; for task id predictor, we set its learning rate  $lr_{task} = 5e^{-4}$ . The weight decay for all parameters are 0.01.
For experience replay, we set the store ratio  $\gamma = 0.01$ , i.e. we store  $1\%$  of seen examples into the episodic memory module. Besides, we set the replay frequency  $\beta = 10$ , which means we do experience replay once every ten steps.
For information disentanglement, we mainly tune the coefficients of the regularization loss. For batches from memory buffer  $\mathcal{M}$ , we set  $\lambda^g$  to 2.5, select best  $\lambda^s$  from  $\{1.5, 2.0, 2.5\}$ . For batches from current training set  $S$ , we set  $\lambda^g$  to 0.25, select best  $\lambda^s$  from  $\{0.15, 0.20, 0.25\}$ .
We evaluate models after training on all tasks and report their average accuracies on all test sets as our metric. Table 3 summarizes our results in Setting (Sampled). While continual finetuning suffered from severe forgetting, experience replay with  $1\%$  stored examples achieves promising results, which demonstrates the importance of experience replay for continual learning in NLP. Beyond that, simple regularization turns out to be a robust method on the basis of experience replay, which shows consistent improvements on all 6 orders. Our proposed Information Disentanglement Based Regularization (IDBR) further improves regularization consistently under all circumstances.
Table 4 compares IDBR with previous SOTA: MBPA++ and LAMOL in Setting (Full). Note that although we use the same training/testing data, there is some inherent differences between our settings and previous SOTA methods. Despite the fact that MBPA++ applies local adaptation when testing, IDBR still outperforms it by an obvious margin. We achieve comparative results with LAMOL, despite that LAMOL requires task identifiers during inference which makes its prediction task easier.
6.1 Impact of the Lengths of Task Sequences
Comparing results of length-3 sequences and length-5 sequences in Table 3, we found that the gap between IDBR and multi-task learning became bigger when the length of task sequence changed from 3 to 5. To better understand how IDBR grad
(a) Task Generic Space
Figure 2: t-SNE visualization of task generic hidden space and task specific hidden space of IDBR.
(b) Task Specific Space
ually forgot, we followed Chaudhry et al. (2018) to measure forgetting  $\mathcal{F}_k$  after trained on task  $k$  as follows:
$$
\begin{array}{l} \mathcal {F} _ {k} = \mathbb {E} _ {j = 1 \dots t - 1} f _ {j} ^ {k}, \\ f _ {j} ^ {k} = \max  _ {l \in \{1 \dots k - 1 \}} a _ {l, j} - a _ {k, j} \\ \end{array}
$$
where  $a_{l,j}$  is the is the model's accuracy on task  $j$  after trained on task  $l$ . On order 4, 5 and 6, we calculate the forgetting every time after IDBR was trained on a new task and summarize them in Table 5. For continual learning, we hypothesize that the model is prone to suffer from more severe forgetting as the task sequence becomes longer. We found that although there was some big drop after training on the 3rd task, IDBR maintained stable performance as the length of task sequence increased, especially after training on 4-th and 5-th task, the forgetting increment was relatively small, which demonstrated the robustness of IDBR.
6.2 Visualizing Disentangled Spaces
To study whether our task generic encoder  $G$  tends to learn more generic information and task specific encoder  $S$  captures more task specific information, we used t-SNE (van der Maaten and Hinton, 2008) to visualize the two hidden spaces of IDBR, using the final model trained on order 2, and the results are shown in Figure 2, where Figure 2a visualizes task generic space and Figure 2b visualizes task specific space. We observe that compared with task specific space, generic features from different tasks were more mixed, which demonstrates that the next sentence prediction helped task generic space to be more task-agnostic than task specific space, which was induced to learn separated representations for different tasks. Considering we only employed two simple auxiliary tasks, the effect of information disentanglement was noticeable.
Model	Length-3 Task Sequences	Length-5 Task Sequences
Order	1	2	3	Average	4	5	6	Average
Finetune	25.79	36.56	41.01	34.45	32.37	32.22	26.44	30.34
Replay	69.32	70.25	71.31	70.29	68.25	70.52	70.24	69.67
Regularization	71.50	70.88	72.93	71.77	72.28	73.03	72.92	72.74
IDBR	71.80	72.72	73.08	72.53	72.63	73.72	73.23	73.19
MTL	74.16	74.16	74.16	74.16	75.09	75.09	75.09	75.09
Table 3: Summary of results on Setting (Sampled) using averaged accuracy after training on the last task. All results are averaged over 3 runs. The  $p$ -values of paired  $t$ -test between nine numbers of Regularization and IDBR are 0.018 on Length-3 and 0.009 on Length-5, demonstrating the significant differences.
Model	TT	TI	LA	PM	Length-5 Task
Order	4	5	6	7	Average
MBPA++ †	✓	BERT	70.7	70.2	70.9	70.8	70.7
MBPA++ ††	✓	BERT	74.9	73.1	74.9	74.1	74.3
LAMOL ††	✓	✓	GPT-2	76.1	76.1	77.2	76.7	76.5
IDBR	✓	BERT	75.9	76.2	76.4	76.7	76.3
Table 4: Summary of results on Setting (Full) using averaged accuracy after training on the last task. Our results are averaged over 2 runs.  $\dagger$  means we fetch numbers from de Masson d'Autume et al. (2019).  $\dagger \dagger$  means we fetch numbers from Sun et al. (2019). TT: whether task-id is available during training. TI: whether task-id is available during inference. LA: whether need local adaptation during inference. PM: pretrained models used for continual learning.
Order	4	5	6	Average
After 2 tasks	0.64	1.63	0.07	0.78
After 3 tasks	3.18	2.56	1.56	2.43
After 4 tasks	3.60	2.17	2.20	2.66
After 5 tasks	3.46	2.33	2.88	2.89
Table 5: Forgetting measure (Chaudhry et al., 2018) calculated every time after finishing training on a new task. All results are averaged over 3 runs.
Model	Accuracy
Regularization	73.03
IDBR w/o Lnsp	73.17
IDBR w/o Ltsk	73.29
IDBR	73.72
Effect of Disentanglement In order to demonstrate that each module of our information disentanglement helps the learning process, we performed ablation study on the two auxiliary tasks using order 5 as a case study. The results are summarized in Table 6. We found that both task-id prediction and next sentence prediction contribute to the final
Table 6: Comparison among using task-id prediction only, next sentence prediction only and both of them. All results are averaged over 3 runs.
Model	4	5	6	Avg
Reg only on s	72.05	72.54	72.61	72.40
Reg only on g	72.01	72.98	72.73	72.57
Reg on both	72.63	73.72	73.23	73.19
Table 7: Comparison among using regularization on task specific space only, task generic space only and both of them. All results are averaged over 3 runs.
performance. Furthermore, the performance gain was much larger by combing these two auxiliary tasks together. Intuitively, the model needs both tasks to disentangle the representation well, since it is easy for the model to ignore one of the spaces if the constraint is not imposed appropriately. The results show that the two tasks are likely complimentary to each other in helping the model learn better disentangled representations.
Impact of Regularization To study the effect of regularization on task generic hidden space  $g$  and task specific hidden space  $s$ , we performed an ablation study which only applied regularization on  $g$  or  $s$ , and compared the results with regularization on both in Table 7. We found that regularization on both spaces results in a much better performance than regularization on one of them only, which demonstrates the necessity of both regulariz
Rules	1	2	3	Average
Random	71.52	72.60	73.03	72.38
K-Means	71.80	72.72	73.08	72.53
Table 8: Comparison between different selection rules: select stored examples randomly or select by K-Means. All results are averaged over 3 runs.
ers. While we may expect to give more tolerance to specific space for changing, we found that no regularization on it would lead to severe forgetting of previously learnt good task specific embeddings, hence it is necessary to add a regularizer over this space as well. Beyond that, we also observed that under most circumstances, adding regularization on the task generic space  $g$  results in a more significant gain than adding regularization on the task specific space  $s$ , consistent with our intuition that task generic space changes less across tasks and thus preserving it better helps more in alleviating catastrophic forgetting.
Impact of K-Means To demonstrate our hypothesis that when the memory budget is limited, selecting the most representative subset of examples is vital to the success of continual learning, we performed an ablation study on order 1,2,3 using IDBR with and without K-Means. The result is shown in Table 8. From the table, we found that using K-Means helps boost the overall performance. Specifically, the improvement brought by K-Means was larger on those challenging orders, i.e. orders on which IDBR had worse performance. This is because for these challenging orders, the forgetting is more severe and the model needs more examples from previous tasks to help it retain previous knowledge. Thus with the same memory budget constraint, diversity across saved examples will help the model better recover knowledge learned from previous tasks.
In this work, we introduce an information disentanglement based regularization (IDBR) method for continual text classification, where we disentangle the hidden space into task generic space and task specific space and further regularize them differently. We also leverage K-Means as the memory selection rule to help the model benefit from the augmented episodic memory module. Experiments conducted on five benchmark datasets demonstrate that IDBR achieves better performances compared
to previous state-of-the-art baselines on sequences of text classification tasks with various orders and lengths. We believe the proposed approach can be extended to continual learning for other NLP tasks such as sequence generation and sequence labeling as well, and plan to explore them in the future.
We would like to thank the anonymous reviewers for their helpful comments, and the members of Georgia Tech SALT group for their feedback.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139-154.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai, and Jiajun Chen. 2019. Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6008-6019, Florence, Italy. Association for Computational Linguistics.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussa. 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523-6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. 2018. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV).
Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147-2157, Online. Association for Computational Linguistics.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2017. Style transfer in text: Exploration and evaluation. arXiv preprint arXiv:1711.06861.
Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Continual relation learning via episodic memory activation and reconsolidation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6429-6440, Online. Association for Computational Linguistics.
Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020. Meta-learning with sparse experience replay for lifelong language learning. CoRR, abs/2009.04891.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR.
Khurram Javed and Martha White. 2019. Metalearning representations for continual learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434, Florence, Italy. Association for Computational Linguistics.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3526.
Yuan Li, Chunyuan Li, Yizhe Zhang, Xiujun Li, Guoqing Zheng, Lawrence Carin, and Jianfeng Gao. 2020. Complementary auxiliary classifiers for label-conditional text generation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8303-8310.
Zhizhong Li and Derek Hoiem. 2018. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2947.
David Lopez-Paz and Marc' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281-297. Oakland, CA, USA.
Arun Mallya and Svetlana Lazebnik. 2018. Packet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7765-7773.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109-165. Elsevier.
Abiola Obamuyide and Andreas Vlachos. 2019. Metalearning improves lifelong relation extraction. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 224-229.
David N Perkins, Gavriel Salomon, et al. 1992. Transfer of learning. International encyclopedia of education, 2:6452-6457.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Amal Rannen, Rahaf Aljundi, Matthew B Blaschko, and Tinne Tuytelaars. 2017. Encoder based lifelong learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 1320-1328.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, , and Gerald Tesauro. 2019. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations.
Mark B Ring. 1998. Child: A first step towards continual learning. In Learning to learn, pages 261-292. Springer.
Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. 2019. Adversarial decomposition of text representation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 815-825, Minneapolis, Minnesota. Association for Computational Linguistics.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. arXiv preprint arXiv:1606.04671.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pages 2990-2999.
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2019. Lamol: Language modeling for lifelong language learning. In International Conference on Learning Representations.
Sebastian Thrun. 1998. Lifelong learning algorithms. In Learning to learn, pages 181-209. Springer.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579-2605.
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sentence embedding alignment for lifelong relation extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 796-806, Minneapolis, Minnesota. Association for Computational Linguistics.
Zirui Wang, Sanket Vaibhav Mehta, Barnabas Poczos, and Jaime Carbonell. 2020. Efficient meta lifelong-learning with limited memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 535-548, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pages 5824-5836. Curran Associates, Inc.
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649-657.
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. 2018. Adversarily regularized autoencoders. In International Conference on Machine Learning, pages 5902-5911. PMLR.