SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 10: Hashtag prediction performance of continually pretrained BERT models over tweets after 2019.
Task	2019-1→2019-2	2019-1→2020-1	2019-1→2020-2
Hashtag Prediction
BERT-base	40.19±0.3	41.00±0.6	40.85±0.8
Sequential PT	43.30±0.7	48.60±2.1	44.07±0.8
ER	42.96±0.9	46.07±1.6	44.26±0.7
Logit-KD	43.35±1.6	46.91±0.5	45.03±0.2
SEED-Logit-KD	43.56±0.4	45.77±0.7	43.76±0.5
Table 11: Temporal generalization performance of Hash-tag prediction models fine-tuned from continually pretrained BERT models over tweets after 2019.
streams, specifically, whether lifelong pretraining improves downstream model performance on the latest domain, as we mentioned in Sec. 4.3. Other than this, our main findings, such as the effect of distillation-based CL algorithms on reducing forgetting, are consistent over two datasets with such significant differences in their changes of vocabulary distribution. We believe it implies the conclusions in this paper should be reliable in diverse data streams.
J Ethnic Risks
We would like to note that, in practice, continually pretrained models over real-world data streams would require identification and removal of biased contents from pretraining corpora, which may affect the prediction of downstream models. As PTLMs are continuously updated, the bias in earlier pretraining may have a profound negative impact. In future works, it is preferable to develop algorithms to "forget" certain biased knowledge from language models. We further note that any data released in this paper, especially the tweet stream, should only be used for research purposes.
(a) Research Paper Stream
(b) Tweet Stream
Figure 8: Cosine distance of vocabulary distributions between each pair of datasets in two data streams.