A Systematic Analysis on the Temporal Generalization of Language Models in Social Media (2024)

Abstract: In machine learning, temporal shifts occur when there are differences between training and test splits in terms of time. For streaming data such as news or social media, models are commonly trained on a fixed corpus from a certain period of time, and they can become obsolete due to the dynamism and evolving nature of online content. This paper focuses on temporal shifts in social media and, in particular, Twitter. We propose a unified evaluation scheme to assess the performance of language models (LMs) under temporal shift on standard social media tasks. LMs are tested on five diverse social media NLP tasks under different temporal settings, which revealed two important findings: (i) the decrease in performance under temporal shift is consistent across different models for entity-focused tasks such as named entity recognition or disambiguation, and hate speech detection, but not significant in the other tasks analysed (i.e., topic and sentiment classification); and (ii) continuous pre-training on the test period does not improve the temporal adaptability of LMs.

heading: 1 Introduction
text: Modern natural language processing (NLP) is centered on language models (LMs) (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019; Min et al., 2023). The versatility of LMs has enabled many real world applications, including chatbot1, text-guided image generation (Aditya et al., 2021), and text-to-speech (Paul K. et al., 2023). One of the well-known issues of LMs, however, is that the capabilities of LMs can not be fully analyzed due to their blackbox nature. To overcome such limitations to understand LMs’ true capability, methodologies and datasets to inspect LMs have been proposed in the context of model probing study, which
*Work done while at Cardiff NLP 1https://openai.com/blog/chatgpt
uncovered various features such as syntax (Hewitt and Manning, 2019; Goldberg, 2019), factual knowledge (Petroni et al., 2019; Ushio et al., 2021), semantics (Ettinger, 2020; Tenney et al., 2019), and emergent ability (Jason et al., 2022).
Besides such studies of LM probing, there is another line of research that focuses on the adaptability of LMs under settings incurring changing conditions, including temporal shifts (Lazaridou et al., 2021; Loureiro et al., 2022a). In this paper, we refer to temporal shifts when discussing settings in which the time period of the test set is different from that of the training set (with the test set period being generally after, reassembling real-world settings.). These settings have been empirically known to lead a non-trivial decrease in performance on some tasks (Liska et al., 2022; Jungo et al., 2022). Needless to say, temporal shifts are more important in more dynamic streaming data with frequent meaning changes and evolving entities, such as social media (Antypas et al., 2022; Ushio et al., 2022).
In this paper, we focus on temporal shifts on Twitter, one of the major social media platforms, and propose a unified evaluation scheme to assess the adaptability of LMs toward temporal shift on Twitter. In particular, we are interested in answering the following two research questions:
• RQ1. Are temporal shifts in social media detrimental for LM performance in NLP tasks?
• RQ2. If so, what are the causes of this temporal shift and can it be mitigated (by e.g. using LMs pre-trained on recent data)?
For the evaluation, we selected five diverse social media NLP tasks for which there are datasets with temporal information available: hate speech detection, topic classification, sentiment classification, named entity disambiguation (NED), and
52
named entity recognition (NER) ranging over different time periods. The temporal shifts considered are relatively short compared to those studied in other sources of streaming data such as news and scientific papers. We test both LMs specialized on social media and other general-purpose trained on encyclopedic and web-crawled corpus.
Our study shows that tasks driven by named entities or events (i.e., hate speech, NED, and NER) present consistent decrease across model under temporal shift settings, while it is less prominent in the other tasks. Crucially, our results show that the decrease caused by temporal shift cannot be mitigated by considering a more recent corpus to the pre-training dataset. Finally, qualitative analysis highlights that the common mistakes made by LMs are indeed instances that require to understand the named entities in the tweet. All the datasets and the scripts to reproduce our experiments are made publicly available online2.

heading: 2 Related Work
text: LMs on Social Media. Major LMs are commonly pre-trained on encyclopedic and webcrawled corpora (Lewis et al., 2020; Raffel et al., 2020; Aakanksha et al., 2023; Rohan et al., 2023; Hugo et al., 2023b,a; Tom B. et al., 2020), while the adaptation of such LMs to social media has led new LMs pre-trained on corpus curated over social media (Nguyen et al., 2020; Loureiro et al., 2022a; DeLucia et al., 2022; Barbieri et al., 2022), which present better performance on social media NLP tasks than standard LMs (Barbieri et al., 2020; Antypas et al., 2023). However, such studies on NLP tasks in social media mainly focus on static datasets without temporal shift. A few of them associate timestamps to the dataset and provide basic temporal analysis (Antypas et al., 2022; Ushio et al., 2022), but these are limited to a single task. Finally, related to the temporal aspect of this work, shortterm meaning shift has also been studied in the context of social media and LMs (Loureiro et al., 2022b).
Temporal Generalization. Importantly, this work aligns to the research on the temporal or diachronic generalization of LMs. In this area, however, most previous works focus on relatively long term (over 10 years) (Lazaridou et al., 2021) or formal source of text such as news and scientific
2https://huggingface.co/datasets/ tweettemposhift/tweet_temporal_shift
papers (Liska et al., 2022; Jungo et al., 2022). In the context of short-term temporal analysis, there are three studies that are most similar to ours. Luu et al. (2022) analyse the temporal performance degradation of LMs in NLP tasks in relatively short time periods. While social media is included as one of the domains, the evaluation is limited to the classification task and to general-domain models. Agarwal and Nenkova (2022) performed a similar general analysis for different tasks, while also analysing the effect of self-labeling as a mitigation to temporal misalignment, which we do not analyse in this work. The main difference between these works in ours is our focus on social media, where we carry out a targeted comprehensive analysis on short-term temporal effects. When it comes to social media, temporal shifts are especially relevant given the real-time nature of the platform and their focus on current events. In the context of Italian Twitter, Florio et al. (2020) analysed the temporal sensitivity of models for hate speech detection, which is one of the tasks included in this paper.
Temporal-aware LMs. To enhance adaptability of LMs for temporal shift, there are a few works that explicitly ingests the temporal information to the model by specific attention mechanism (Rosin and Radinsky, 2022), augmenting the input with timestamp (Rosin et al., 2022), joint modeling of temporal information (Dhingra et al., 2022), and self-labeling (Agarwal and Nenkova, 2022). In this paper, we do not include any temporal-aware LMs, because we are interested in analysing the adaptability of plain LMs to temporal shifts.

heading: 3 Experimental Setting
text: In this section, we describe our experimental setting to investigate the effect of temporal shifts in LMs.

heading: 3.1 Evaluation Methodology
text: Let us define Dtrain and Dtest as the training and test splits of a dataset D for a single downstream task (e.g. sentiment classification), where each dataset contains pairs of a text input and associated labels. Importantly, Dtrain is taken from the period prior to Dtest without any temporal overlap. Given such dataset with temporal split, we consider the following two settings of out-of-time (OOT) and in-time (IT).
Out-of-Time (OOT). In the first setting, we simply train the models on Dtrain and evaluate them
on Dtest. Noticeably, models have no access to the instances from the test period at the training phase in this setting, so we refer the setting as out-of-time (OOT) as an analogy to the out-of-domain (OOD).
In-Time (IT). As a comparison to OOT, we consider the second experimental setting, which is designed to assess the effect of training instances from the test period. The test set is randomly split into non-overlapped four chunks (Dtest = ⋃4 i=1Ditest) for cross validation, where models trained on Dtrain
⋃Dtest\Ditest are evaluated on Ditest. For each chunk of the test set, we downsample the IT training set to the same size as Dtrain with three random seeds and report the averaged metrics over the runs. To be precise, we consider a function Fs(D) that randomly samples |Dtrain| instances from D, and we independently train models on Fs(Dtrain
⋃Dtest\Ditest) for s = 0, 1, 2. In contrast to OOT, we refer this setting as in-time (IT) setting.
Figure 1 presents an example overview of the differences between IT and OOT settings from the perspective of data sampling periods (data from 2018 to 2022 in the example).

heading: 3.2 Tasks & Datasets
text: We consider the following five diverse social media NLP tasks: hate speech detection, topic classification, sentiment classification, named entity disambiguation (NED), and named entity recognition (NER). For each task, we employ a public dataset for English and leverage its original temporal splits, unless there is overlap between the periods of training and test sets.
Hate Speech Detection. Hate speech detection in Twitter consists of identifying whether a tweet contains hateful content. We use the dataset proposed by Waseem and Hovy (2016) framed as binary classification as the dataset to create the training and test splits based on the timestamp. The first half is used for training and the rest for test split. The training split is further randomly split into 2:8 for validation:training. We use accuracy to evaluate the hate speech detection models.
Topic Classification. Topic classification is a task that consists of associating an input text with one or more labels from a fixed label set. For this evaluation, we rely on TweetTopic (Antypas et al., 2022), a multi-label topic classification dataset with 19 topics such as sports or music. As evaluation metric, we use micro-F1 score to measure the performance of topic classification models.
Sentiment Analysis. Sentiment analysis is a standard social media task consisting of associating each post with its sentiment. In particular, we use the dataset from the task 2: LongEvalClassification from CLEF-2023 (Alkhalifa et al., 2023) in which the task is framed as binary classification with positive and negative labels. The original training split contains around 50k instances while 1k test split, which is highly imbalance and the effect of the IT sample can be very limited. To balance the training and test splits, we randomly sample 2.5k instances from each label, amounting 5k new training instance. We use accuracy to evaluate the sentiment classification models.
Named Entity Disambiguation (NED). NED is a a binary classification that consists of identifying if the meaning of a given target entity in context is the same as the one provided. We use the TweetNERD (Mishra et al., 2022) dataset and reformulated into NED following SuperTweetEval (Dimosthenis et al., 2023). Then, we create the train, validation, and test splits in the same way as the hate speech detection. We use accuracy to evaluate the NED models.
Named Entity Recognition (NER). NER is a sequence labelling task to predict a single namedentity type on each token on the input text. We rely on TweetNER7 (Ushio et al., 2022), a NER dataset on Twitter that contains seven named entity types. We use span F1 score to evaluate NER models.

heading: 3.2.1 Data Statistics
text: Table 1 shows the size and the period of the training and the test sets for each dataset, and Figure 2 displays the number of tweets per quarter for each task. Topic classification and NER use the same tweets, which are sampled uniformly from each month, while NED and hate speech detection have the majority of the tweets in the latest quarter. Sentiment analysis covers the longest period in the dataset that spans over four years. Figures 3 and 4 show the comparisons of the label distribution of the binary (i.e., hate speech, sentiment classification, and NED) and multi-classification tasks (i.e., NER and topic classification), respectively. As can be observed, hate speech detection has fewer positive labels in OOT than in IT, while the other two tasks have the same ratio of the positive labels between OOT and IT. The same pattern can be observed for
topic classification and NED, for which the label distribution does not substantially change.

heading: 3.3 Models
text: We investigate an established general-purpose LM, RoBERTa (Liu et al., 2019) as well as other LMs pre-trained on tweets including BERTweet (Nguyen et al., 2020), TimeLM (Loureiro et al., 2022a), and BERNICE (DeLucia et al., 2022). For RoBERTa and BERTweet, we consider the base and the large models, referred as RoBERTa (B), RoBERTa (L), BERTweet (B), and BERTweet (L). For TimeLM, we consider the base models trained on the tweets up to 2019, 2020, 2021 and 2022, referred as TimeLM2019 (B), TimeLM2020 (B), TimeLM2021 (B) and TimeLM2022 (B), and the large model trained upto 2022, referred as
TimeLM2022 (L). The end date of the pre-trained corpus for each model is 2019-02 (RoBERTa), 2019-08 (BERTweet), 2019-12 (TimeLM2019), 2020-12 (TimeLM2020), 2021-12 (TimeLM2021 and BERNICE), and 2022-12 (TimeLM2022). All the model weights are taken from the transformers model hub (Wolf et al., 2020) and Table 2 shows the details of models we used in the paper.3 Table 3 shows the overlap between the period of the pre-trained corpus and the test set for each task, which will be relevant for the analysis on the effect of pre-training in Section 5.1. These models are then fine-tuned in the datasets presented in the previous section, in both OOT and IT settings. For model fine tuning, we run hyperparameter search with Optuna (Akiba et al., 2019) with the default search space.

heading: 4 Results
text: Figures 5 to 9 show the comparisons of IT and OOT in hate speech detection, NED, NER, topic classification and sentiment analysis. As can be observed, hate speech detection, NED and NER present inconsistencies in both settings, decreasing the performance from IT to OOT. In contrast, this cannot be observed for both sentiment analysis, and especially topic classification. The average decrease of OOT performance for each of the tasks is 4.5, 2.4, 1.7, 0.8 and -0.1 for hate speech detection, NER, NED topic classification and sentiment analysis.
One of the main differences of those two groups of tasks (i.e. hate/NED/NER v.s. topic/sentiment) entity-centric or event-driven nature of the former. NER and NED are clearly related to named entities. Hate speech detection does not relate to named entities explicitly, but since the tweets for hate speech detection are collected by querying specific events, they are often about events or celebrities which peak around the sampled timestamp (Gómez et al., 2023). On the other hand, events or named entities are not as important in sentiment analysis, as the sentiment can be estimated from the context in most cases. Topic classification depends on the topic, with some of them related to entities (e.g. those related to celebrities or TV) and others not (e.g., daily life, family or food), but in the main clearly identifiable by the context. Through the lens of entity relevancy, this result may suggest that the temporal shift can be caused by named entities, which includes meaning drift of existing named
3Note that for this analysis we are not interested in the performance of zero-shot LLMs such as GPT-4, but rather on the effect of fine-tuned LMs.
entities or emerging new named entities. Topic classification can be seen as a mixture of entityrelated instances and not, which results in not fully consistent gain from OOT, but still significant in the average.

heading: 5 Analysis
text: This section focuses on the second research question (RQ2) and analyses the main causes behind temporal shift performance degradation of LMs.

heading: 5.1 Effect of Pre-Training
text: A possible direction to mitigate the temporal shift is to pre-train the LMs on the text from the test period, which does not require any labeling. Figure 10 visualizes the performance and relative IT improvement of LMs with/without pre-training corpus covering the test period of each task for topic classifi-
cation/NED/NER4. At a glance, we cannot observe see any relationship between the pre-training corpus and the performance. The averaged relative gains of the metrics from OOT within the LMs pretrained on the test period and the others are 2.0 and 0.6 (topic classification), 3.5 and 3.8 (NER), and 2.1 and 1.9 (NED) respectively. Therefore, all models are affected by the temporal shift irrespective of the pre-training corpus date. This implies that the temporal shift cannot be robustly resolved by only adding data from the test period to the pre-training corpus, a conclusion that was also reached by Luu et al. (2022).

heading: 5.2 Effect of Label Distribution
text: In supervised machine learning label distribution, the distribution of the binary label over the test instances, shifts can affect a model’s performance. In
4The test periods of hate speech detection and sentiment classification are covered by all the LMs we considered in the experiment.
this section, we analyse this potential effect when it comes to temporal shifts. For this, we rely on hate speech detection, which presents the largest decrease in performance from IT to OOT, with a different label distribution between training and test (see Figure 3). For the other tasks, the label distribution appears to be largely similar. To separate the effect of label distributional shift between IT and OOT from the temporal shift, we conduct a controlled experiment by balancing the label distribution of each IT training split to be the same as OOT training split. This is achieved by undersampling the size of the training set. Table 4 shows the results, where the average relative gain is still positive, although it becomes less dominant in balanced experiment. This highlights how label distribution may change over time and this itself have an effect in model performance. A similar finding was already discussed by Luu et al. (2022).

heading: 5.3 Qualitative Analysis
text: In this analysis, we have a closer look on the test instances that are incorrect in OOT, turning to be correct in IT. To be precise, we sort the test instance in a single task based on the number of models where the error in OOT setting has been corrected in IT setting over all the random seeds. In other words, given a test instance, we check whether a model prediction is incorrect in OOT, but correct in the IT setting. This particular instance is counted as a correction. In total, we have 10 models with 3 independent runs with different random seed to construct the training data, so 30 would be the maximum number of corrections. For sentiment classification, hate speech detection, topic classification and NED, we simply count instance-level corrections. Given the complex nature of NER evaluation, we decided to only focus on the entity type
predictions for this analysis.
Table 5 shows the top instances in terms of IT corrections for each of the task. We can observed the marked differences across tasks, with NED and hate speech detection including instances which were corrected 100% in the OOT setting. In fact,
there are respectively 44 and 15 instances for which this is the case in these two tasks. Similarly for NER, the number of corrections is high. This is correlated with the main results of the paper (see Section 4) which showed clear improvements for these tasks in the IT setting, but not for sentiment and and topic classification.
Finally, Table 6 shows some of these instances for NED and hate speech detection. In the case of NED, the tweets relate to two new TV series that were on air at test time (Japanese Kieta Hatsukoi in the first example and Indian Bade Achhe Lagte Hain in the second, both from 2021). This is similar to the hate speech detection in which the examples
belong to the My Kitchen Rules TV show. This highlights the event-driven nature of social media, and the importance of acquiring the background context for the specific task.

heading: 6 Conclusion
text: We proposed an evaluation method to assess the adaptability of LMs for temporal shifts on social media with five diverse downstream tasks including sentiment classification, NER, NED, hate speech detection, and topic classification. We have tested diverse LMs trained on Twitter under different temporal settings. The experimental results indicate that the adaptability gets consistently worse on entity or event-driven tasks (NED, NER, and hate speech detection) while the effect is limited in the other tasks. This conclusion was similar to previous work in more general domains, which observed a variation across different types of task when it comes to temporal degradation (Luu et al., 2022; Agarwal and Nenkova, 2022). Finally, our analysis shows that pre-training on a corpus from the test period is not enough to solve the temporal shift issue, with performance still being degraded in comparison to models fine-tuned on the labeled dataset from the test period.
Limitations
Regardless of some similarities between Twitter and other streaming data such as news and other social media platforms being real-time and trenddriven, they can have different characteristics, and the results of our study may apply to Twitter exclusively. For our evaluation we rely on a single dataset for each of the tasks. Of course, these datasets are not a faithfully representation of the task and may contain their own biases. Therefore, even for the same task, the findings in this paper may differ if using a different dataset.
Ethical Statement
The datasets we used in the experiments are all from Twitter. Data has been anonymized (only information about legacy-verified users is kept) so that they do not contain any personal identifiable information (PII). We do not gather information from individual accounts but rely on aggregated information and metrics only. Please note that the text may contain sensitive content due to the nature of social media and the task, in particular hate speech detection.

heading: Acknowledgements
text: Jose Camacho-Collados is supported by a UKRI Future Leaders Fellowship.

heading: Future Work
text: Jose Camacho-Collados is supported by a UKRI Future Leaders Fellowship