SlowGuess/NAACL_2022_OCR huggingface dataset:
Michael Matena and Colin Raffel. 2021. Merging models with fisher-weighted averaging. ArXiv, abs/2111.09832.
Martin Müller, Marcel Salathé, and Per Egil Kummervold. 2020. Covid-twitter-bert: A natural language processing model to analyse Covid-19 content on twitter. ArXiv, abs/2005.07503.
Sheshera Mysore, Zachary Jensen, Edward Kim, Kevin Huang, Haw-Shiuan Chang, Emma Strubell, Jeffrey Flanigan, Andrew McCallum, and Elsa Olivetti. 2019. The materials science procedural text corpus: Annotating materials synthesis procedures with shallow semantic structures. In Proceedings of the 13th Linguistic Annotation Workshop, pages 56-64, Florence, Italy. Association for Computational Linguistics.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, Online. Association for Computational Linguistics.
Elsa A Olivetti, Jacqueline M Cole, Edward Kim, Olga Kononova, Gerbrand Ceder, Thomas Yong-Jin Han, and Anna M Hiszpanski. 2020. Data-driven materials research enabled by natural language processing and information extraction. Applied Physics Reviews, 7(4):041317.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rückle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Linguistics.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. icarl: Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5533-5542. IEEE Computer Society.
Shruti Rijhwani and Daniel Preotiac-Pietro. 2020. Temporally-informed analysis of named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7605-7617, Online. Association for Computational Linguistics.
Anthony V. Robins. 1995. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci., 7:123-146.
Paul Röttger and J. Pierrehumbert. 2021. Temporal adaptation of bert and performance on downstream document classification: Insights from social media. Findings of EMNLP.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress & compress: A scalable framework for continual learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm Mässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4535-4544. PMLR.
Yangyang Shi, Martha Larson, and Catholijn M. Jonker. 2015. Recurrent neural network language model adaptation with curriculum learning. Comput. Speech Lang., 33:136-154.
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020. LAMOL: language modeling for lifelong language learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323-4332, Hong Kong, China. Association for Computational Linguistics.
Jens Vindahl. 2016. Chemprot-3.0: a global chemical biology diseases mapping.
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sentence embedding alignment for lifelong relation extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 796-806, Minneapolis, Minnesota. Association for Computational Linguistics.
Zirui Wang, Sanket Vaibhav Mehta, Barnabas Poczos, and Jaime Carbonell. 2020. Efficient meta lifelong-learning with limited memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 535-548, Online. Association for Computational Linguistics.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. 2022. Pretrained language model in continual learning: A comparative study. In International Conference on Learning Representations.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104, Online. Association for Computational Linguistics.
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, and Furu Wei. 2021. Adapt-and-distill: Developing small, fast and effective pretrained language models for domains. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 460-470, Online. Association for Computational Linguistics.
(b) SciERC
Figure 6: Performance of downstream models with various number of training examples. The models are fine-tuned from the final pretrained model  $(f_4)$ .
A Detailed Experiment Settings
We use a linearly decreasing learning rate initialized with 5e-4 on the research paper stream and 3e-4 on the tweet stream. On the research paper stream, we train the model for 8,000 steps in the first task, and 4,000 steps in the subsequent tasks. On the tweet stream, we train the model for 8,000 steps in all tasks. We hold out 128,000 sentences from each corpus to evaluate MLM performance. As the size of pretraining corpora is large, during training, each training example is visited only once. We use the masked language modeling perplexity over held-out validation sets of the pretraining corpora as the metrics for hyperparameter tuning. Common hyperparameters such as learning rate and batch sizes are tuned with Task-specific models with the first task. Hyperparameters that are specific to continual learning algorithms, such as the scale of the distillation loss, is tuned using the first two domains in the stream according to the MLM performance over validation sets. The weight of the distillation term  $\alpha$  is set as 1.0 for logit distillation and 0.1 for other distillation algorithms. By default, we replay or perform distillation with a mini-batch of examples from the replay memory every 10 training steps in ER and Distillation-based CL approaches. We use the huggingface transformers library https://github.com/huggingface/transformers for implementation.
B Low-Resource Fine-Tuning
Figure 6 summarizes the performance of fine-tuned models from the final model checkpoint  $(t = 4)$
Task	2014	2016	2018	2020
Hashtag Prediction
RoBERTa-base	56.65±0.6	45.50±2.1	48.08±1.0	56.42±0.2
Sequential PT	59.00±0.1	54.28±0.3	56.79±0.5	59.85±0.4
ER	59.00±0.1	54.90±0.2	56.93±0.1	59.56±1.7
Adapter	58.76±0.7	52.55±1.5	54.34±1.7	59.01±1.0
Logit-KD	60.93±0.5	55.96±0.2	58.21±0.5	60.52±0.2
Rep-KD	60.47±0.1	51.77±2.6	55.79±1.4	59.80±0.2
Contrast-KD	60.72±0.6	55.85±0.0	57.94±0.4	59.54±0.3
SEED-KD	58.82±0.4	54.55±0.5	56.87±0.2	59.71±0.2
SEED-Logit-KD	61.28±0.2	55.59±0.5	57.75±0.4	60.74±0.6
Task-Specific (2014)	61.62±0.3	55.38±0.6	56.16±0.6	59.59±0.3
Task-Specific (Latest)	59.91±0.3	55.47±1.0	56.61±0.4	59.87±0.6
MTL	60.51±0.3	55.16±1.6	57.89±0.4	59.95±0.3
Emoji Prediction
RoBERTa-base	28.73±0.2	26.86±0.2	25.71±0.1	24.42±0.2
Sequential PT	32.69±0.2	30.55±0.3	29.30±0.1	27.69±0.1
ER	32.88±0.2	30.52±0.2	29.50±0.1	27.75±0.1
Adapter	32.15±0.2	29.85±0.0	28.72±0.0	26.80±0.3
Logit-KD	33.08±0.3	30.88±0.1	29.77±0.1	27.80±0.1
Rep-KD	32.71±0.2	30.51±0.2	29.45±0.1	27.27±0.2
Contrast-KD	32.90±0.1	31.01±0.1	29.48±0.2	27.72±0.3
SEED-KD	32.91±0.1	30.84±0.3	30.12±0.1	27.66±0.1
SEED-Logit-KD	33.28±0.1	31.17±0.1	29.98±0.1	27.84±0.2
Task-Specific (2014)	33.37±0.2	30.54±0.3	28.94±0.0	26.98±0.2
Task-Specific (Latest)	32.31±0.0	29.83±0.5	29.06±0.2	27.19±0.1
MTL	32.78±0.1	30.54±0.0	29.52±0.2	27.47±0.0
Table 6: Full performance on Twitter Hashtag prediction and Emoji prediction, fine-tuned from the pre-trained model in the final time step.
using different amount of downstream training examples. We see on Chemprot and SciERC, the benefit of Sequential Pretraining over RoBERTa-base is more significant in low-resource fine-tuning setups. Whenever Sequential Pretraining outperforms RoBERTa-base, we notice Logit-KD could further improve over Sequential Pretraining.
C Full Results over the Tweet Stream
Tables 6 and 7 summarize full results over the Tweet stream. Compared to the table 5 in the main text, we add downstream performance over data from years 2014 and 2016  $(D_{1}, D_{2})$ , and temporal generalization from year 2014 to 2020  $(D_{1} \rightarrow D_{4})$ .
The research paper stream consists of full text of 6.6M, 12.1M, 7.8M, and 7.5M research papers from the S2ORC (Lo et al., 2020) dataset. We evaluate downstream fine-tuning performance on two in-domain datasets for each research area: Chemprot relation exaction dataset (Vindahl, 2016) and RCT abstract sentence role labeling dataset (Dernoncourt and Lee, 2017) for the bio-medical domain; ACL-ARC citation intent classification dataset (Jurgens et al., 2018) and SciERC relation extraction dataset (Luan et al., 2018) for the
Task	2014 → 2020	2016 → 2020	2018 → 2020
Crossyear Hashtag Prediction
RoBERTa-base	39.31±2.7	42.23±2.7	37.19±2.1
Sequential PT	44.00±1.1	49.87±1.8	46.63±0.9
ER	43.31±0.2	50.72±0.6	46.27±0.4
Adapter	42.61±0.5	48.00±1.6	42.63±0.9
Logit-KD	44.26±0.9	50.92±0.8	46.84±1.0
Rep-KD	42.48±0.2	50.38±1.5	42.23±0.2
Contrast-KD	45.22±0.1	52.14±1.1	47.47±0.8
SEED-KD	43.39±0.4	49.62±1.0	46.37±0.8
SEED-Logit-KD	45.35±0.6	51.56±0.7	47.74±0.3
Task-Specific (2014)	44.34±0.6	49.26±0.7	45.09±0.7
Task-Specific (2020)	43.44±0.5	49.41±1.1	44.34±0.4
- 4x steps	44.34±0.6	51.78±0.7	44.69±0.7
MTL	44.04±0.3	50.37±0.3	44.31±0.0
Crossyear Emoji Prediction
RoBERTa-base	12.02±0.4	13.24±0.2	18.67±0.1
Sequential PT	14.20±0.2	16.08±1.4	21.06±0.9
ER	14.36±0.4	16.82±0.3	21.57±0.1
Adapter	13.53±0.2	15.68±0.3	20.64±0.1
Logit-KD	14.20±0.3	16.28±1.1	21.29±1.0
Rep-KD	13.89±0.1	16.03±0.3	20.86±0.2
Contrast-KD	14.42±0.3	17.52±0.1	21.43±0.1
SEED-KD	14.36±0.1	16.97±0.4	21.88±0.3
SEED-Logit-KD	14.36±0.1	16.97±0.3	21.62±0.1
Task-Specific (2014)	13.39±0.2	15.14±0.2	20.79±0.3
Task-Specific (2020)	13.00±0.2	14.48±0.3	19.30±0.2
- 4x steps	12.90±0.4	14.85±0.3	19.83±0.2
MTL	14.07±0.2	16.64±0.2	20.94±0.7
Table 7: Temporal generalization performance on Twitter Hashtag prediction datasets fine-tuned from the final pretrained model. Year  $1\rightarrow$  Year 2 indicates the hashtag prediction model is fine-tuned on data in year Year 1, and evaluated on test data in Year 2.
computer science domain; relation extraction over Synthesis procedures (Mysore et al., 2019) and named entity recognition over material science papers (MNER) (Olivetti et al., 2020) for material science domain; keyphrase classification and hyponym classification after filtering out physics papers for the physics domain (Augenstein et al., 2017). We report micro-averaged F1 on Chemprot, RCT, MNER datasets following the evaluation metrics in the original work, and report macroaveraged F1 on all other datasets. We use the official data splits for all datasets except for RCT, where we employ a low-resource training setup following Gururangan et al. (2020).
The pretraining corpora for the tweet stream consist of 25M tweets in each year. For downstream tasks, we use a separate set of 1M tweets from each year to construct multi-label hashtag prediction (Gong and Zhang, 2016) datasets and single-label emoji prediction datasets (Barbieri et al., 2018). We replace user names to special tokens. For Hashtag prediction, the label space consists of tweets containing 200 most frequent hashtags in each year. We independently sample 500 tweets per label (hashtag) as training, validation and test
sets, which results  $10k$  examples in each of the data splits. For emoji prediction, we construct 20-way single-label emoji prediction datasets for each year following Barbieri et al. (2018) with the 1M held out tweets. We sample 5,000 tweets per emoji in each split, resulting in balanced datasets of the same size as the hashtag prediction datasets.
E Details of Continual Learning Algorithms
E.1 Contrastive Distillation
During continual pretraining, in addition to the language model pretraining objective, we add a unsupervised contrastive learning objective, namely the SimCSE (Gao et al., 2021) objective, so that the similarity in the sentence representation better reflects the semantic similarity in the sentence. We use the  $l^2$ -normalized representation of the start-of-sequence token at the final layer as the sentence representation, noted as  $h$ . Then, we distill the intra-batch representational similarity from the previous model  $f_{t-1}$  to the current model  $f_t$ . Given a mini-batch of  $N$  examples  $x$ , we compute the representational dot-product similarity matrix between normalized sentence representations  $h$  between each pair of examples with  $f_{t-1}$  and  $f_t$ , noted as  $\mathbf{B}^{t-1}$  and  $\mathbf{B}^t$ , where each element  $\mathbf{B}_{ij}$  is,
$$
\mathbf {B} _ {i j} = \frac {\exp \left(\boldsymbol {h} _ {i} \cdot \boldsymbol {h} _ {j} / \tau\right)}{\sum_ {k = 1 . . N} \exp \left(\boldsymbol {h} _ {i} \cdot \boldsymbol {h} _ {k} / \tau\right)} \tag {1}
$$
where  $\tau$  is a temperature hyperparameter. We specify a temperature  $\tau_{t} = 0.05$  for the teacher model  $f_{t - 1}$  and a temperature  $\tau_{s}$  for the student model  $f_{t} = 0.01$ . We compute the cross-entropy between  $\mathbf{B}^{t - 1}$  and  $\mathbf{B}^t$  as the distillation loss,
$$
\ell_ {\text {d i s t i l l}} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {N} \mathbf {B} _ {i j} ^ {t - 1} \log \mathbf {B} _ {i j} ^ {t} \tag {2}
$$
E.2 SEED Distillation
SEED distillation proposed by (Fang et al., 2021) has a similar spirit as the contrastive distillation with differences in the examples used for computing similarity matrices computes. The algorithm distills representational similarity between the batch and a large set of other examples, maintained in an example queue  $Q$ . As the number of target examples  $K$  can be much larger than the batch size, it allows distillation of richer information by regularizing similarities. During pretraining, the method maintains a fixed-size queue
$Q$  to cache examples from the current domain  $D_{t}$ . Given a mini-batch of training examples  $\pmb{x}$ , it computes cosine similarity between each pair of examples within the batch  $\pmb{x}$  and  $Q$  with  $f_{t-1}$  and  $f_{t}$ , resulting in two similarity matrices  $\mathbf{B}^{t-1}$ ,  $\mathbf{P}^y \in \mathbb{R}^{|B| \times |Q|}$ . Similar to the contrastive distillation, the distillation loss is the cross-entropy between two similarity matrices  $\mathbf{B}^{t-1}$  and  $\mathbf{B}^t$  computed in the same way as Eq. 2.
F Analysis and Controlled Experiments of Computational Costs
Computational cost is a crucial matter for online continual learning systems. In this section, we analyze the computational costs of continual learning algorithms and perform controlled experiments of computational costs.
We quantify computational costs with the total number of forward  $(C_f)$  and backward  $(C_b)$  computations  $(C = C_f + C_b)$  over the PTLMs, which is easy to control; in practice, we find the wall clock time of training was approximately linear to  $C$ . We summarize the number of forward and backward passes and the wall clock time of training in Table 4. In the visit of  $b$  batches from the training stream, Sequential PT performs  $b$  forward and backward passes respectively over the PTLM, resulting in  $C = 2b$ . Experience replay further replays 1 batch of examples every  $k$  steps over the training stream, which results in  $C = (2 + 2/k)b$ . In our main experiments,  $r$  is set to 10 (Sec. 3.3). Logit-Distill and Rep-Distill require one additional forward pass over a frozen PTLM to compute the target of distillation, resulting in  $C = (3 + 3/k)b$ . Distillation algorithms that perform contrastive learning with SimCSE (i.e. SEED-Distill and SEED-Logit-Distill) additionally require one forward and backward pass using the same batch of examples with different dropout masks. Therefore, for SEED-Logit-Distill,  $C = (5 + 5/k)b$ .
To control the number of forward and backward passes, we present approaches to compensate the lower computation costs compared to Distillation algorithms and one approach to shrink the computational cost of distillation algorithms: (1) for Sequential PT, we train the models for 1.2 times more steps so that  $C = 2.4b$ , noted as Sequential  $\mathrm{PT}_{b' = 1.2b}$ ; (2) for ER, we increase the replay frequency  $k$  to 5 from the default setup 10, so that  $C = 2.4b$ . We also decrease the cost of Logit-KD and SEED-Logit-KD by reducing the frequency
of distillation from every 1 batch to every  $r' = 10$  steps, while still replaying and distilling knowledge over 1 batch of memory examples every 10 training steps. This results in  $C_f = (1 + 2/k + 1/k')b$  and  $C_b = (1 + 1/k)b$ , where  $C = 2.4b$  when both  $r$  and  $r'$  are 10. The approach is referred to as Sparse Logit-KD. Finally, for SEED-Logit-KD, we remove the SimCSE loss from training and perform sparse distillation similar to Sparse-Logit-KD, which also results in  $C = 2.4b$ .
The performance of the models is presented in Table 9. We notice that at the end of pretraining, increasing the number of training steps in Sequential PT by 1.2 times does not lead to performance boost on the latest domain  $(D_4)$ , while the performance over tasks from earlier domains (Chemprot, ACL-ARC, SciERC) slightly dropped, possibly due to increased forgetting. For ER, we notice replaying only slightly more frequently  $(\mathrm{ER}_{k=5})$  than the default setup  $(k=10)$  greatly increased the perplexity of MLM, implying significantly increased overfitting to the memory; while the performance differences of downstream tasks compared to the default ER is mixed. When we decrease the replay frequency of distillation, the performance on Logit-KD and SEED-KD also decreased and does not outperform ER.
The results show additional computation costs can be necessary for continual learning algorithms such as Logit-KD and SEED-Logit-KD. However, the results also show that there is no simple tradeoff between computational cost and performance. We have seen that it is not always beneficial to increase the number of training steps over the emerging data, as it increases forgetting in earlier domains. Similarly, increasing the frequency of replay may lead to significant overfitting to the replay memory. Investigating into more effective continual learning algorithms, despite increased computation costs, allows us to obtain performance improvement that cannot be simply traded with more computation with arbitrary continual learning algorithms. We leave more thorough studies into this topic as future work.
G Experiments with RoBERTa-large
We present additional experiments on RoBERTa-large. Figure 7 and Table 8 summarizes the results of selected continual learning algorithms and baselines. On Chemprot, RCT-Sample and SciERC, Logit-KD achieves best performance with the last
(d) SciERC
Figure 7: Performance evolution of downstream models with RoBERTa-large as the base model. Models are fine-tuned from checkpoints of lifelong pretrained LMs at different time steps  $t$ . For Chemprot and RCT-Sample from  $D_{1}$ , we use  $t \in \{1,2,3,4\}$  while for ACL-ARC and SciERC from  $D_{2}$ ,  $t \in \{2,3,4\}$ . Methods achieving the best performance at the end of training ( $t = 4$ ) is highlighted.
Domain	Biomedical	Computer Science
Dataset	Chemprot	RCT-Sample	ACL-ARC	SciERC
RoBERTa-large	84.39±0.7	80.76±0.7	72.20±3.2	83.02±0.8
Naive	85.43±0.6	81.10±0.5	73.44±2.0	82.88±0.7
ER	85.42±0.2	81.30±0.4	71.51±2.5	83.22±0.5
Logit-KD	86.18±0.7	81.93±0.7	72.10±2.0	83.23±0.6
Task-Specific	85.99±0.3	82.02±0.6	76.07±1.0	82.91±0.9
Table 8: Results on the Research Paper stream with RoBERTa-large as the base model.
pretraining checkpoint. On ACL-ARC, however, we notice that Sequential PT achieves the best performance, while all continual learning algorithms (ER, Logit-KD) achieves lower F1 at the initial time step  $(t = 2$ , Figure 7(c)). This implies continual learning algorithms may hurt model's performance in capturing new knowledge, despite its potential to reduce forgetting.
H Experiments with BERT on Tweet Stream After 2019
In this section, we present an additional set of experiments on BERT-base (Devlin et al., 2019) model, which is originally pretrained with Wikipedia articles before 2019, with Tweets only after 2019. The training corpora  $D_{1..4}$  consist of tweets from the first half of 2019, the second half of 2019, the first half of 2020, and the second half of 2020 respectively. We accordingly construct hashtag prediction and cross-year hashtag prediction datasets. The performance of downstream tasks fine-tuned from the final pretrained model is presented in Table 10. We see Sequential PT clearly outperforms BERT-base which is not continually pretrained, and that
Logit-KD generally improves hashtag prediction performance compared to Sequential PT except on the first half of 2019. We hypothesize the small temporal gap between  $D_{1..4}$  makes improvements less significant than our main experiment setup. We present temporal generalization performance in cross-year hashtag prediction tasks in Table 11. Similarly, Logit-KD improves over Sequential PT in two out of three cross-year hashtag prediction setups.
I Analysis of Data Streams
In this section, we provide further analysis about the created research paper stream and the tweet stream. We measure cosine distances  $d_v$  of vocabulary distributions between each pair of different domains  $(D_{1..4})$  and summarize the results in Figure 8. The results indicate that the Tweet stream has a magnitude smaller vocabulary distribution gap between domains, which is in the scale of  $1e^{-5}$ , compared to the research paper stream, which is in the scale of  $1e^{-2}$ . On the Tweet stream, we see the differences of vocabulary distributions align with the temporal gap between domains. On the research paper stream, we find some domains to be more similar than others. For example, Biomedical  $(D_1)$  and Material Science domains  $(D_3)$  have larger similarity in their vocabulary distributions, which explains general downstream performance increase on  $D_1$  after the model is pretrained on  $D_3$  (Fig. 4 (a,b)).
The differences in vocabulary distribution explain inconsistency in results between two data
Task	D1 - Biomedical	D2 - Computer Science	D3 - Materials Science	D4 - Physics
Dataset	Chemprot	RCT-Sample	MLM	ACL-ARC	SciERC	MLM	MNER	Synthesis	MLM	Keyphrase	Hyponym	MLM
Sequential Pretraining	82.09±0.5	79.60±0.5	1.654	72.73±2.9	81.43±0.8	1.807	83.99±0.3	92.10±1.0	1.590	67.57±1.0	74.68±4.4	1.381
Sequential Pretrainingb'=1.2b	81.68±0.5	79.80±0.4	1.656	70.57±3.0	80.89±1.2	1.793	83.65±0.3	92.16±0.7	1.578	67.61±1.4	75.03±4.1	1.379
ER	82.73±0.3	79.98±0.3	1.737	72.50±1.0	81.64±1.1	1.857	83.99±0.4	92.65±0.4	1.621	66.11±1.1	72.82±4.3	1.391
ERk=5	83.00±0.1	79.79±0.4	1.913	69.85±2.6	82.30±1.2	2.049	84.03±0.2	91.60±0.6	1.721	65.55±0.4	75.64±3.2	1.418
Logit-KD-Sparse	82.80±0.4	79.80±0.5	1.476	73.31±2.0	81.19±0.8	1.744	83.84±0.4	92.29±0.7	1.472	66.65±0.7	77.27±7.1	1.385
SEED-KD-Sparse	82.51±0.4	79.52±0.5	1.474	73.70±3.4	81.92±0.8	1.741	83.96±0.3	92.20±1.0	1.480	64.75±1.1	71.29±3.6	1.381
Table 9: Performance of distillation algorithms in the setup of controlled computational costs.
Task	2019-1	2019-2	2020-1	2020-2
Hashtag Prediction
BERT-base	46.38±0.4	48.05±0.8	41.67±1.0	69.00±0.5
Sequential PT	50.46±0.1	52.70±0.7	46.49±1.0	71.63±0.7
ER	49.90±0.4	52.33±0.6	46.84±0.3	71.67±0.4
Logit-KD	50.19±0.9	53.70±0.4	47.64±0.4	72.44±0.5
SEED-Logit-KD	50.79±0.8	52.84±0.5	46.04±0.4	72.24±0.6