SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
\text {A v e r a g e} F 1: A v g F 1 (t) = \frac {1}{t} \sum_ {i = 1} ^ {t} F 1 _ {t, i} \tag {7}
$$
where  $F1_{t,i}$  is the F1 score, either macro F1 or micro F1, achieved by the model on the  $i$ th task after being trained on the  $t$ th task. The larger this metric, the better the model. We compare our methods with the following methods:
Fine-tuning: The model contains two bidirectional LSTM encoders (Hochreiter and Schmidhuber, 1997; Zhou et al., 2016; Liu et al., 2016) to encode the tweet and the group separately. The score of the group is calculated as the cosine distance between the hidden state of the tweet encoder and that of the group encoder. This model is also the backbone model of all the methods described below, except Fine-tuning + BERT. The model is directly fine-tuned on the stream of tasks, one after another, by the ranking loss function in 1.
Fine-tuning+BERT: The training framework is the same as above, but each encoder is replaced by a pre-trained BERT model (Devlin et al., 2019) followed by a linear layer. The linear layers are fine-tuned during training.
Fine-tuning+RMR (Random Memory Replay): We augment the fine-tuning method with an additional memory module. Same as in section 4.2, the memory is divided equally for each task, but instead of using LB-SOINN, the  $K$  samples are randomly sampled from the current training data and then rewrite  $K$  random slots in the old memory. EWC: EWC is a regularization-based method, adding a penalty term  $\sum_{i} \frac{\lambda}{2} F_i (\theta_i - \theta_i^*)^2$  to the ranking loss function  $1$ .  $F_i$  is the diagonal of the Fisher information matrix  $F$ ,  $\theta$  is the model parameter, and  $i$  labels each parameter.  $\theta^*$  is the model parameter when the model finishes training on the previous task.  $\lambda$  is set to 2e6 in our experiments.
GEM: We use the episodic memory in the original paper: the memory is populated with  $m$  random
Number of observed tasks	t=5	t=10	t=15
Avg F1 score (%)	Macro	Micro	Macro	Micro	Macro	Micro
Multitask	15.26	67.07	5.05	37.20	3.57	38.61
Fine-tuning	6.02	16.44	4.35	5.77	3.96	6.18
Fine-tuning + BERT	6.02	16.44	4.06	5.45	3.03	5.80
Fine-tuning + RMR	11.15	44.40	2.56	15.77	3.51	15.19
EWC	8.57	20.42	2.42	6.81	1.95	7.27
GEM	13.04	30.95	3.07	12.51	2.70	15.07
Ours	12.61	49.75	6.96	47.30	5.13	44.62
Table 2: Experimental results. RMR: random memory replay. The best results are in bold.
Number of observed tasks	t=5	t=10	t=15
Avg F1 score (%)	Macro	Micro	Macro	Micro	Macro	Micro
Full Model	12.61	49.75	6.96	47.30	5.13	44.62
w/o DKLmem	15.00	58.64	4.21	36.36	3.72	40.87
w/o VRL	11.05	35.03	4.53	13.69	3.65	11.28
w/o LB-SOINN	13.01	50.99	6.15	44.42	5.59	30.91
Table 3: Ablation study. w/o  $D_{KLmem}$ :  $D_{KLmem}$  in the equation 6 is removed. w/o VRL: VRL is replaced by the model used in the fine-tuning setting, i.e., fine-tuning + LB-SOINN memory replay. w/o LB-SOINN: LB-SOINN memory replay is replaced by random memory replay, i.e., VRL + RMR. The best results are in bold.
samples from each task.  $m$  is a predefined size of the episodic memory. We set  $m = 100$  in our experiments, so each task can add 100 tweets to the memory. By the end of the 15 tasks, the total memory of GEM contains 1500 tweets.
Multitask Learning: The tasks are trained simultaneously. We mix the training data from multiple tasks to train the model. This setting does not follow the lifelong learning setting where the tasks are trained sequentially. We add this setting in our experiments to show the potential room for improvement concerning each lifelong learning method.
We do not compare our method with Support Vector Machine (Suykens and Vandewalle, 1999) or Logistic Regression, because they require the number of classes to be fixed and to be known in advance, which is unrealistic in our tasks. We also do not compare our method with Qian et al. (2018a) since the latter also has this requirement, as mentioned in section 3. Adapting their method for the lifelong learning setting requires modifying both the model architecture and the training algorithm, which is beyond the scope of this paper.
In all our experiments, we use 1-layer bi-LSTM as encoders except the fine-tuning + BERT setting and we use cosine distance to measure similarity. The input of the group encoder is the concatenation of the group name and its hate ideology. We use 1-layer bidirectional GRU (Cho et al., 2014) as the decoder in VRL. The hidden size of the encoders and the decoders is 64. The latent variable size in VRL is 128. We use 300-dimensional randomly ini
tialized word embeddings. All the neural networks are optimized by Adam optimizer with the learning rate 1e-4. The batch size is 64. The loss margin  $m = 0.5$ . The maximum number of training epochs for each task is set to 20. For LB-SOINN,  $\lambda = 1000$ ,  $\eta = 1.04$ . The memory size is limited to 1000 tweets for all the methods using a memory module except GEM. We do not set episodic memory size for each task as GEM because for lifelong hate speech classification, the number of tasks keeps increasing in the real world, and assuming unlimited total memory is unrealistic.
The experimental results are shown in Table 2. We report the performance of each method after the model finishes training on the first 5 tasks, first 10 tasks, and all the 15 tasks. The average macro-F1 score is much lower than the average micro-F1 score due to the imbalanced data of each task. The large performance gap between the multitask training and fine-tuning shows that there exists severe catastrophic forgetting and that the low average F1 scores in the fine-tuning setting are not due to the model capacity. Replacing the bi-LSTM encoder with the pre-trained BERT encoder does not improve the performance. This reconfirms that the low scores result from catastrophic forgetting, not model capacity. Actually fine-tuning and fine-tuning with BERT achieves the same average F1 scores at  $t = 5$  because both models completely forget the previous tasks after converging on the fifth
task, so both models achieve the same F1 scores on the testing data of the fifth task while achieving 0 scores on the previous four tasks. Due to the large model capacity of BERT, fine-tuning with BERT tends to overfit on the training data more seriously, leading to slight performance decline at  $t = 10$  and  $t = 15$  compared to using bi-LSTM encoders. Since model capacity is not the key factor to solve catastrophic forgetting, we simply use bi-LSTM as encoders in our model instead of BERT, considering the computational cost.
Adding RMR to the fine-tuning setting achieves significant performance improvement, even better than EWC or GEM. This is related to the characteristic of our tasks mentioned at the end of section 3. EWC remembers previous tasks by slowing down the update of the model parameters important to them, which is more suitable for the sequence of tasks that are similar to each other. However, significant changes in vocabulary, topic, or input data distribution are very common in our sequence of tasks, making memory replay more efficient than EWC. The performance of GEM during the second half of the training is close to that of fine-tuning with RMR, but there exists a gap in the first half. The reason is that GEM sets an episodic memory for each task, of which the size is 100 in our experiments, so before the 10th task in the sequence, the size of the total memory available for GEM is less than that of the memory module used in the fine-tuning with RMR setting.
Although RMR improves the performance, the average F1 scores still drop quickly when the number of tasks increases. In the late stage of sequential training, each task can only keep dozens of samples in the memory and the model is not able to generalize well based on the memory. Our method solves this problem by combining VRL and LB-SOINN memory replay. The performance of our model is better and more stable than the other methods when the number of tasks increases. Our method achieves higher scores than multitask training in the last four columns of Table 3 because learning on one task is easier than learning on a mix of tasks simultaneously. Every model in our sequential training experiments can easily achieve high F1 scores on the current task, making a large contribution to the average F1 scores. However, when doing multitask training, the model loses this benefit.
To investigate the effect of our method, we conduct the ablation study as shown in Table 3. Re
Figure 3: The testing results of the first 5 tasks in the sequence when our model is trained on the first 10 tasks.
moving  $D_{KLmem}$  from the final loss function in equation 6 does not lower the performance when the number of observed tasks is small  $(t = 5)$  because each task can store a few hundreds of samples in the memory at the early stage of sequential training, which is sufficient for the model to learn the previous tasks. However, when the number of tasks increases,  $D_{KLmem}$  shows its effect on alleviating catastrophic forgetting.
Fine-tuning+LB-SOINN (Table 3) does not perform as well as fine-tuning+RMR (Table 2), while VRL+LB-SOINN (i.e., full model) performs better than VRL+RMR (Table 3). The reason lies in the input for LB-SOINN. Compared to the hidden representations spread evenly in the hidden space, the hidden representations which are well-organized in different group clusters make it easier for LB-SOINN to learn a reasonable topology structure of the training samples. VRL achieves this by explicitly pushing the hidden representation of tweets to follow a learned multivariate Gaussian distribution unique to each group. On the other hand, directly using the hidden state of the tweet encoder does not exhibit such kind of characteristics. VRL not only distills task knowledge but also provides an appropriate input for LB-SOINN, as stated in section 4.
Although our model achieves significant improvement over the baseline methods, we observe that our method does not perform well on the first task. As shown in Figure 3, there exists a large gap between the performance on the first task and the other tasks, and the micro-F1 score on the first task quickly drops to almost 0 when the number of observed tasks increases. We find the same results after we change the order of tasks in the sequence, so this is not the result of the task difficulty but is
the result of our method. We find this problem is due to the reconstruction loss, which is the first part in equation 4. The model observes a very limited number of tweets when training on the first task, making it difficult to learn the language model and reconstruct the tweet. As a result, the tweet representation learned on the first task may not contain the information we require, resulting in a large performance gap. When the number of observed tasks increases, this problem goes away quickly. We anticipate pre-training the VAE in our model (the left branch in Figure 2) on a large Twitter corpus can alleviate this problem at the beginning of training.
In this paper, we introduce the lifelong hate speech classification task and propose to use the VRL and LB-SOINN memory module to alleviate catastrophic forgetting. Our proposed method has the potential to benefit other lifelong learning tasks where the similarity between the contiguous tasks can be low. We intend to make our implementation freely available to facilitate more application and investigation of our method in the future.
Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759-760. International World Wide Web Conferences Steering Committee.
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022.
Pew Research Center. 2017. Online harassment 2017.
Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar. A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. arXiv preprint arXiv:1703.04009.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 13122-13131.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. 2017. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734.
Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135.
Lei Gao, Alexis Kuppersmith, and Ruihong Huang. 2017. Recognizing explicit and implicit hate speech using a weakly supervised two-path bootstrapping approach. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 774-782.
Jennifer Golbeck, Zahra Ashktorab, Rashad O Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A Geller, Quint Gergory, Rajesh Kumar Gnanasekaran, et al. 2017. A large labeled corpus for online harassment research. In Proceedings of the 2017 ACM on Web Science Conference, pages 229-233. ACM.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526.
Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. 2018. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262-2268. IEEE.
Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional LSTM model and inner-attention. arXiv preprint arXiv:1605.09090.
David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6467-6476.
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. 1995. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. *Psychological review*, 102(3):419.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier.
Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In Proceedings of the 25th international conference on world wide web, pages 145-153. International World Wide Web Conferences Steering Committee.
Jose L Part and Oliver Lemon. 2016. Incremental online learning of object classes using a combination of self-organizing incremental neural networks and deep convolutional neural networks. In Workshop on Bio-inspired Social Robot Learning in Home Scenarios (IROS), Daejeon, Korea.
Jose L Part and Oliver Lemon. 2017. Incremental online learning of objects for robots operating in real environments. In 2017 Joint IEEE International Conference on Development and Learning and Epi-genetic Robotics (ICDL-EpiRob), pages 304-310. IEEE.
Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. 2018a. Hierarchical cvae for fine-grained hate speech classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3550-3559.
Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. 2018b. Leveraging intra-user and inter-user representation learning for automated hate speech detection. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 118-123.
Roger Ratcliff. 1990. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. *Psychological review*, 97(2):285.
Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pages 3738-3748.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. CoRR, abs/1606.04671.
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020. LAMOL: language modeling for lifelong language learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Johan AK Suykens and Joos Vandewalle. 1999. Least squares support vector machine classifiers. Neural processing letters, 9(3):293-300.
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sentence embedding alignment for lifelong relation extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 796-806.
William Warner and Julia Hirschberg. 2012. Detecting Hate Speech on the World Wide Web. In ACL'12: Proceedings of the 2nd Workshop on Language in Social Media, pages 19-26. Association for Computational Linguistics.
Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88-93.
Rui Xia, Jie Jiang, and Huihui He. 2017. Distantly supervised lifelong learning for large-scale social media sentiment analysis. IEEE Transactions on Affective Computing, 8(4):480-491.
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3987-3995.
Hongwei Zhang, Xiong Xiao, and Osamu Hasegawa. 2013. A load-balancing self-organizing incremental neural network. IEEE Transactions on Neural Networks and Learning Systems, 25(6):1096-1105.
Haoti Zhong, Hao Li, Anna Cinzia Squicciarini, Sarah Michele Rajtmajer, Christopher Griffin, David J Miller, and Cornelia Caragea. 2016. Content-Driven Detection of Cyberbullying on the Instagram Social Network. In *IJCAI'16: Proceedings of the 25th International Joint Conference on Artificial Intelligence*, pages 3952–3958.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Short Papers), volume 2, pages 207-212.