Characterizing and Measuring Linguistic Dataset Drift
Year: 2023

Abstract:
NLP models often degrade in performance when real world data distributions differ markedly from training data. However, existing dataset drift metrics in NLP have generally not considered specific dimensions of linguistic drift that affect model performance, and they have not been validated in their ability to predict model performance at the individual example level, where such metrics are often used in practice. In this paper, we propose three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift. These dimensions correspond to content word frequency divergences, syntactic divergences, and meaning changes not captured by word frequencies (e.g. lexical semantic change). We propose interpretable metrics for all three drift dimensions, and we modify past performance prediction methods to predict model performance at both the example and dataset level for English sentiment classification and natural language inference. We find that our drift metrics are more effective than previous metrics at predicting out-of-domain model accuracies (mean 16.8% root mean square error decrease), particularly when compared to popular fine-tuned embedding distances (mean 47.7% error decrease). Fine-tuned embedding distances are much more effective at ranking individual examples by expected performance, but decomposing into vocabulary, structural, and semantic drift produces the best example rankings of all considered model-agnostic drift metrics (mean 6.7% ROC AUC increase).

heading: 1 Introduction
text: Dataset drift, when test data distributions differ from a model’s training data, can have detrimental effects on NLP model performance (Broscheit et al., 2022; Do et al., 2021; Koh et al., 2021). In real world scenarios, models are regularly monitored for potential performance degradations by
∗ Work done during an internship at AWS AI Labs. † Corresponding author.
comparing incoming test data with the training data (Elango et al., 2022; Nigenda et al., 2022). For these scenarios, researchers have proposed a variety of linguistic dataset drift metrics that aim to predict NLP model performance degradations between training and test domains (Elsahar and Gallé, 2019; Ramesh Kashyap et al., 2021).
However, previous drift metrics and performance predictions suffer from several limitations. First, previous metrics have generally been designed as holistic measures of linguistic dataset drift, despite the fact that different NLP tasks and models might be sensitive to different dimensions of linguistic drift. Second, previous research has focused on drift metrics at the dataset level rather than the individual example level. Not only does this require multiple labeled evaluation domain datasets to make out-of-domain performance predictions (regressions require multiple dataset-level drift values to fit to; Elsahar and Gallé, 2019; Ramesh Kashyap et al., 2021), but drift metrics are often used in practice to predict model performance when individual real world examples are streamed in real time (Elango et al., 2022). We seek to overcome both of these limitations by proposing and evaluating specific dimensions of linguistic drift, predicting out-of-domain model performance at both the individual example level and the dataset level.
Specifically, we propose three dimensions of linguistic dataset drift along with corresponding drift metrics: vocabulary, structural, and semantic drift. Because these dimensions capture distinct features that could have different effects on the performance of an NLP model, we hypothesize that decomposing into these three dimensions will allow NLP performance prediction models to better predict model performance on novel data. Indeed, when compared to previous model-agnostic drift metrics predicting performance on English sentiment classification and natural language inference (NLI), our metrics produce both improved predictions
8953
of dataset-level accuracies and improved rankings of individual examples by expected performance, for both in-domain and out-of-domain data (mean 16.8% accuracy root mean square error decrease, mean 6.7% ROC area under the curve increase). Although we find that previously-proposed finetuned embedding distances (Elango et al., 2022) are far more effective at ranking individual examples by expected performance, those distances are extremely ineffective at predicting actual model accuracies. We conclude that decomposing linguistic drift into vocabulary, structural, and semantic drift is an effective approach for predicting out-ofdomain model accuracy, and for ranking individual examples when model-agnostic metrics are desired.

heading: 2 Related Work
text: Past work has quantified the drift between NLP datasets using distances between token frequency distributions or TF-IDF vectors (Bäck, 2019; Ramesh Kashyap et al., 2021; Sato et al., 2022), language model embedding distances (Feldhans et al., 2021; Yamshchikov et al., 2021), or the ability of domain classifiers to discriminate between datasets (Dredze et al., 2010; Elsahar and Gallé, 2019; Ruder et al., 2017). Notably, Ramesh Kashyap et al. (2021) find that these metrics can predict performance degradations when an NLP model is transferred from a training dataset Dtrain to an outof-domain evaluation dataset Deval.
However, existing metrics have generally been designed as holistic measures of linguistic drift, failing to capture specific dimensions that might affect NLP model performance in different ways. Furthermore, the traditional setup for evaluating drift metrics (Elsahar and Gallé, 2019; Ramesh Kashyap et al., 2021) only allows for dataset-level drift metrics that predict overall model accuracy on out-of-domain datasets. In practice, when real world examples are streamed during test time, it is desirable to predict model performance for individual examples using example-level drift metrics (i.e. drift between an example x and a training dataset Dtrain; Elango et al., 2022; Nigenda et al., 2022). In our work, we modify the setup from Ramesh Kashyap et al. (2021) to predict performance for individual examples (Section 4), using logistic regressions fitted to example-level drift metrics. In contrast to Ramesh Kashyap et al. (2021), we can fit our regressions to predict outof-domain performance even when only a single
in-domain evaluation dataset is available.

heading: 3 Dimensions of Linguistic Drift
text: As described above, previous measures of dataset drift in NLP suffer from (1) lack of specificity and (2) lack of validation at the example level, where such metrics are often used in practice. First, we address the lack of specificity by proposing three dimensions of linguistic dataset drift: vocabulary, structural, and semantic drift. As in previous work, we primarily focus on domain drift, i.e. divergence in the input probabilities P (x) rather than the joint probabilities over inputs and labels P (x, y). For each of our proposed drift dimensions, we propose a metric that quantifies the drift between an evaluation example x and a training dataset Dtrain, allowing us to use our metrics to predict examplelevel model performance. We evaluate our metrics empirically in Section 4.

heading: 3.1 Vocabulary Drift
text: We define vocabulary drift as the divergence between content word frequencies in two text samples. Content words are defined as open class words that generally contain substantial semantic content (e.g. nouns, verbs, adjectives, and adverbs), contrasted with function words that primarily convey grammatical relationships (e.g. prepositions, conjunctions, and pronouns; Bell et al., 2009; Segalowitz and Lane, 2000). By restricting our vocabulary drift definition to content word distributions, we capture vocabulary differences between two text datasets without the confounds of structural features. For example, “The student ate the sandwich” and “A sandwich was eaten by a student” would have low vocabulary drift after excluding function words. Notably, our definition of vocabulary drift is designed to include drift in word choice, regardless of the semantic similarity between chosen words; for example, “The dog was happy” and “The beagle was ecstatic” would have high vocabulary drift due to differing word choice, despite their high semantic similarity. This property is useful because NLP models are often sensitive to changes in word choice even if datasets are semantically similar (Hu et al., 2019; Misra et al., 2020).
Formally, to quantify the vocabulary drift between an evaluation example x and a training dataset Dtrain, we compute the cross-entropy be-
tween content word frequencies in x and Dtrain as:
1 |xcontent| ∑
w∈xcontent log(Ptrain_content(w)). (1)
Here, xcontent is the set of content words in example x, and Ptrain_content(w) is the frequency (restricted to content words) of word w in the training dataset. Our vocabulary drift metric is equal to the logperplexity (training loss) of a unigram language model restricted to content words, trained on Dtrain and evaluated on x. We annotate content words using the spaCy tokenizer and part-of-speech (POS) tagger (Honnibal et al., 2017), defining content words as those with an open class Universal POS tag (nouns, verbs, adjectives, adverbs, and interjections; Nivre et al., 2020) and excluding stop words in spaCy.

heading: 3.2 Structural Drift
text: In contrast to vocabulary drift, structural drift captures divergences between the syntactic structures in two text samples. For example, “Yesterday, I was surprised by a dog” and “Usually, she is recognized by the audience” would have low structural drift despite high vocabulary drift. Previous work in discourse analysis has attempted to quantify structural similarity separately from semantic similarity in natural conversations, although their metrics are not directly applicable to NLP datasets due to computational limitations (Boghrati et al., 2018).1 Structural drift has also been studied in machine translation, primarily considering structural divergence between parallel text in different languages (Dave et al., 2004; Deng and Xue, 2017; Dorr, 1990; Saboor and Khan, 2010); in our work, we focus on divergences between nonparallel monolingual text.
We quantify the structural drift between an example x and Dtrain using the cross-entropy between the true POS tag sequence for x and the predictions of a POS 5-gram model trained on POS tag sequences in Dtrain. This metric captures the divergence between syntactic structures in x and Dtrain using 5-gram sequences, abstracting away from semantic content and vocabulary by considering only the POS tag for each word (Axelrod et al., 2015; Nerbonne and Wiersma, 2006). Formally,
1The CASSIM structural similarity metric in Boghrati et al. (2018) is based on tree-edit distances between all sentence pairs, which is slow to compute even for relatively small NLP datasets.
we compute:
1
|x|
|x|∑
i=1
log(Ptrain(tagi|tagi−1, ..., tagi−4)). (2)
We pad the beginning of the POS tag sequence with [SEP] tokens, and we only annotate examples with structural drift if they contain at least two non[SEP] tokens. As with our vocabulary drift metric, we annotate POS tags using the spaCy tokenizer and POS tagger.

heading: 3.3 Semantic Drift
text: Finally, we consider semantic drift, defined as any divergence in semantic meaning between two text samples. Semantic drift is closely related to both vocabulary and structural drift; the words and syntactic structures used in a sentence are closely tied to the meaning of that sentence, particularly under compositional assumptions of language (Szabó, 2022). However, there are notable cases where semantic drift is independent from vocabulary and structural drift. For example, “I saw the doctor” and “I took a trip to the hospital” have high vocabulary and structural drift under our definitions, despite similar semantic meaning. Conversely, some sentences have different meanings or connotations across time and contexts, despite remaining identical in both vocabulary and structure (e.g. the word
“sick” in “That salamander is sick!” can mean very cool or physically ill depending on the context).
Many of these semantic similarities and differences can be quantified using contextualized embeddings from modern language models (Briakou and Carpuat, 2020; Devlin et al., 2019; Liu et al., 2020; Sun et al., 2022), which we include in our drift metric experiments (Section 4). However, when identifying individual dimensions of linguistic drift, we seek to identify dimensions that are both interpretable and relatively independent from one another, to better isolate specific dimensions that impact NLP model performance. Language model embeddings reflect vocabulary and structural properties of sentences as well as semantic properties (Hewitt and Manning, 2019; Tenney et al., 2019), and thus they are less effective for pinpointing interpretable effects that are specific to semantic drift.
Lexical Semantic Change. Instead, we consider lexical semantic change, in which a word’s meaning changes between two datasets while its sur-
face form remains the same (Gulordava and Baroni, 2011; Kulkarni et al., 2015; Sagi et al., 2009; Tahmasebi et al., 2021). Past work has quantified a token’s lexical semantic change LSCD1↔D2(w) using the mean pairwise cosine distance between contextualized RoBERTa embeddings for that token in two different datasets D1 and D2 (Giulianelli et al., 2020; Laicher et al., 2021). Motivated by this metric, we quantify the lexical semantic change between an evaluation example x and a training dataset Dtrain using the mean lexical semantic change between x and Dtrain for all content tokens w shared between x and Dtrain:
1 |xcontent| ∑
w∈xcontent LSCx↔Dtrain(w). (3)
Here, LSCx↔Dtrain(w) is the mean pairwise cosine distance between embeddings for w in example x and dataset Dtrain, using a non-fine-tuned RoBERTa model. Again, we define content tokens as tokens that are annotated with an open class POS tag anywhere in the Universal Dependencies English dataset, excluding stop words (Nivre et al., 2020).2 While this lexical semantic change metric is still based on contextualized embeddings, matching embeddings based on token surface forms allows us to minimize effects of vocabulary and structural drift, as compared to matching each example representation with all other example representations regardless of surface form. Of course, lexical semantic change is just one type of semantic drift; future work might consider other types of semantic drift that are independent from vocabulary and structural drift.

heading: 4 Experiments
text: Previous work has evaluated drift metrics by assessing their ability to predict out-of-domain model performance at the dataset-level using dataset-level metrics (e.g. Ramesh Kashyap et al., 2021; Section 2). We extend this work by predicting individual example-level performance (probabilities of getting individual examples correct) along with datasetlevel accuracies, using drift metrics between each
2We exclude non-content tokens for lexical semantic change because non-content token embeddings (e.g. for function words and punctuation) are more likely to encode structural drift information rather than lexical semantic change. Contextualized token embeddings are computed as the mean of the token representations in the last two RoBERTa layers before fine-tuning (Elango et al., 2022).
example x and the training dataset Dtrain. Using these example-level metrics instead of datasetlevel metrics allows us to fit regressions predicting model performance using only a set of examples (e.g. using only the in-domain evaluation set), rather than a set of multiple evaluation datasets covering different domains. Thus, our approach can be used in common real world scenarios where labeled data is available only in one domain. In our experiments, we compare previous drift metrics with our proposed metrics for vocabulary, structural, and semantic drift, evaluating whether decomposing linguistic drift into these three dimensions improves NLP model performance predictions. 3

heading: 4.1 Datasets
text: We evaluate cross-domain transfer performance for language models fine-tuned on sentiment classification (split by product category or review year) and natural language inference (NLI, split by source domain). Because these tasks output one prediction per sequence, they allow us to directly evaluate sequence-level (i.e. example-level) drift metrics.
Amazon Reviews (product categories). For sentiment classification, we consider the Amazon reviews dataset, containing customer-written product reviews for 43 different product categories (Amazon, 2017). As in Blitzer et al. (2007), we label 1- and 2-star reviews as negative, and 4- and 5- star reviews as positive. We sample up to 100K polarity-balanced reviews from each product category, considering each category as a domain. For each product category, we use a 70/20/10% split for training, evaluation and test datasets.
Amazon Reviews (temporal split). Next, we consider the same Amazon reviews dataset for sentiment classification, but we define domains by review date rather than by product category. We generate a category-balanced and polarity-balanced sample for each year between 2001 and 2015 (inclusive) by sampling up to 5K polarity-balanced reviews from each product category for each year, sampling the same number of reviews each year for any given category. The resulting dataset has 33K training examples, 5K evaluation examples, and 5K test examples for each year, similar to Agarwal and Nenkova (2022), but balanced for product category and polarity.
3Code is available at https://github.com/amazon-science/ characterizing-measuring-linguistic-drift.
MultiNLI. Finally, we consider the MNLI dataset for natural language inference (NLI), covering five training domains and ten evaluation domains, including government documents, pop culture articles, and transcribed telephone conversations (Williams et al., 2018). Each training domain has approximately 77K training examples, and each evaluation domain has approximately 2K evaluation examples.

heading: 4.2 Models
text: We fine-tune a RoBERTa base-size model M for each training domain for each task, using batch size 32, learning rate 2e-5, and four epochs through the training data (Liu et al., 2019). Because there are only five training domains for MNLI, we run five fine-tuning runs per MNLI training domain. Full fine-tuning details and hyperparameters are listed in Appendix A.1. We evaluate each model on each evaluation domain; to simulate realistic scenarios for temporal data, we evaluate only on future years for models trained on temporal splits.

heading: 4.3 Drift metrics
text: We consider drift metrics between individual evaluation examples x and training datasets Dtrain. First, we consider our vocabulary, structural, and semantic drift metrics from Section 3. Initial motivations and theoretical examples of how these three dimensions differ are described in Section 3, but the dimensions are not perfectly independent. Empirically, Pearson correlations between our vocabulary, structural, and semantic drift metrics range from 0.10 to 0.50 across the different tasks. For comparison, we also consider drift metrics from past work: token frequency divergences and embedding cosine distances. With the exception of the finetuned embedding distances, all of our metrics are model-agnostic, meaning they are not dependent on the internals of the fine-tuned model.
Token frequency divergences. We compute the Jensen-Shannon (JS) divergence between the token frequency distribution for each example x and each training dataset Dtrain. This divergence has been shown to correlate with out-of-domain model performance when computed at the dataset-level (i.e. between an entire evaluation set Deval and the training set Dtrain; Ramesh Kashyap et al., 2021), and it has been recommended as a metric for training dataset selection (Ruder et al., 2017).
However, because example-level token frequency distributions are quite sparse (Ruder et al., 2017), we also consider the cross-entropy between each example frequency distribution and each training frequency distribution (i.e. the loss of a unigram language model). The resulting token frequency cross-entropy is equivalent to our vocabulary drift metric, but using the RoBERTa tokenizer and without the restriction to content words.
Embedding cosine distances. We compute embeddings for training and evaluation examples x by taking the mean over all tokens in x and the last two RoBERTa layers, either before or after task fine-tuning (i.e. pre-trained or fine-tuned; Elango et al., 2022). We note that the pre-trained RoBERTa model is still the same model that is fine-tuned for each task, potentially leading to overly optimistic results for the pre-trained embedding cosine distances; this caveat also holds for our semantic drift metric, which relies on pre-trained embeddings. For the embedding cosine distance drift metrics, we compute the mean cosine distance between the embedding for evaluation example x and each example in the training dataset Dtrain (Nigenda et al., 2022).4
4For efficiency, we compute mean pairwise cosine distances using the method described in Appendix A.2.

heading: 4.4 Predicting Model Performance
text: For each drift metric (or set of drift metrics) and each model M trained on dataset Dtrain, we fit a logistic regression predicting whether M will get example x correct (i.e. a “positive” example), based on the drift metric(s) between x and Dtrain. The regression input is the considered drift metric(s) from x to Dtrain, and the label is 1 if M predicts x correctly, and 0 otherwise.5 We fit the logistic regression for all x in the in-domain evaluation dataset, mimicking a scenario where labeled evaluation data is only available in the same domain as training. This allows us to test whether regressions fitted only to in-domain examples can extrapolate to out-of-domain examples.
We evaluate the logistic regressions on both indomain and out-of-domain evaluation examples. Each regression produces a predicted probability of “positive” (getting an example correct) for each example.6 For dataset-level accuracy predictions, we compute the mean predicted “positive” probability over all examples in each evaluation dataset Deval, equal to the expected value of model accuracy on Deval based on the example-level probabilities.

heading: 4.5 Evaluating Performance Predictions
text: We use ROC curves to evaluate example-level performance predictions, both in-domain and out-ofdomain, and we use root mean square errors (RMSEs) to evaluate out-of-domain dataset-level accuracy predictions.
ROC AUC. For each logistic regression, predicting positive examples (correct model predictions) from a given drift metric and for a given model, we compute the area under the ROC curve for indomain and out-of-domain examples. An ROC curve plots recall (proportion of true positives identified) over the false positive rate for different probability thresholds. In our case, a higher ROC AUC indicates that the input drift metric can generally predict more true positives (examples the model gets correct) for a given false positive rate. However, ROC curves are dependent only on the rankings of examples by predicted positive probabilities (Tang et al., 2010); the raw probabilities of correct
5In cases where we input multiple drift metrics into the logistic regression, we exclude interaction terms; interaction terms generally resulted in worse out-of-domain performance predictions, based on both ROC AUCs and RMSEs.
6For in-domain evaluation example predictions, we use 5-fold cross-validation, fitting regressions to only 80% of the in-domain evaluation dataset per fold.
model predictions do not affect the ROC AUC as long as the example ranking is preserved. From this perspective, a higher ROC AUC indicates that evaluation examples are ranked roughly in order of expected performance; examples with higher predicted probabilities are more likely to be predicted correctly by the model. For each drift metric, we compute the mean ROC AUC over all trained models M, for in-domain and out-of-domain examples. RMSE. Because ROC AUCs depend only on the ranking of evaluation examples, they do not capture whether the predicted positive probabilities (probabilities of correct predictions) are actually reflective of model accuracies. For example, a given drift metric can achieve a high ROC AUC by ranking evaluation examples accurately, even if the mean probability (expected model accuracy) is far from the true model accuracy for Deval (e.g. Figure 1).
Thus, for each drift metric, we also compute the RMSE comparing expected model accuracy (mean positive probability over all examples in Deval) to actual model accuracy on Deval. We compute RMSEs over all models M and their corresponding out-of-domain datasets Deval. We report RMSEs as percentages of a baseline RMSE that predicts out-of-domain accuracy on Deval to be the same as the in-domain evaluation accuracy (i.e. predicting no out-of-domain performance drop). Our reported RMSE percentages indicate the percentage of accuracy prediction error that remains when using a given drift metric, relative to the baseline.
To summarize, we compute the predicted accuracy RMSE and the mean ROC AUC for each drift metric and for each task. ROC AUC measures how well a drift metric ranks the evaluation examples (examples with higher “positive” probabilities should be more likely to be predicted correctly by the model), while RMSE measures how well the drift metric predicts actual model accuracy (mean probabilities should be close to the true model accuracy). An ideal drift metric should have high ROC AUC and low RMSE.

heading: 5 Results
text: The mean accuracy change (±standard deviation; in raw accuracy percentage difference) from indomain to out-of-domain evaluation is −1.04 ± 1.02% for sentiment classification across product categories, −0.20± 0.57% for sentiment classification across years, and −1.83± 2.91% for MNLI across source domains. Notably, in many cases, ac-
curacy improves for out-of-domain evaluation (e.g. MNLI fiction → government). Results predicting out-of-domain evaluation accuracies (RMSEs) and example-level performance (ROC AUCs) from different drift metrics are reported in Table 1.

heading: 5.1 Ranking Examples (ROC AUC)
text: Mean ROC AUCs for different drift metrics are reported in Table 1, for both in-domain and out-ofdomain evaluation examples. Recall that a higher ROC AUC indicates that higher scoring examples (as ranked by the logistic regression) are more likely to be predicted correctly by the model.
Decomposing drift improves rankings. Using vocabulary, structural, and semantic drift as input features into the logistic regressions results in higher ROC AUCs than any of the previous model-agnostic drift metrics, for all three multidomain datasets and for both in-domain and out-ofdomain examples (top section of Table 1). Across the three datasets, this decomposed drift improves ROC AUCs by an average of 0.039 for in-domain examples and 0.033 for out-of-domain examples when compared to the best model-agnostic drift
metric from previous work. To ensure that this is not simply a result of including three different metrics in the regression, we also consider the combination of all three model-agnostic metrics from previous work (“combined previous model-agnostic” in Table 1: token frequency JS-divergence, token frequency crossentropy, and pre-trained embedding cosine distance). For all three datasets, the combination of previous metrics still results in worse ROC AUCs than the combination of vocabulary, structural, and semantic drift, for both in-domain and outof-domain examples. This indicates that decomposing into vocabulary, structural, and semantic drift results in better rankings of individual examples by expected performance than previous modelagnostic drift metrics.
Fine-tuned embeddings lead to the best rankings. However, fine-tuned (model-dependent) embedding cosine distances result in by far the best rankings of examples by expected performance (higher ROC AUCs). Indeed, this is the recommended drift metric when examples need to be ranked relative to one another or relative to some
threshold (e.g. when there is some threshold drift value to flag examples; Elango et al., 2022; Nigenda et al., 2022); our results validate this approach. Notably, the fine-tuned embedding distances produce quality rankings even for out-ofdomain examples, despite work suggesting that fine-tuning affects the in-domain representation space differently from the out-of-domain representation space in language models (Merchant et al., 2020). Our results indicate that despite these differences between the in-domain and out-of-domain fine-tuned spaces, the fine-tuned embedding distances can still be used to rank both in-domain and out-of-domain examples by expected performance.
That said, fine-tuned embedding distances require access to the internal representations of a given model; model-agnostic metrics are still useful in cases where only model outputs can be observed, or when the same drift metric needs to apply to multiple models. For these use cases, our decomposed vocabulary, structural, and semantic drift metrics outperform previous model-agnostic metrics. Furthermore, as we observe in the next section, our decomposed drift metrics result in drastically better out-of-domain accuracy predictions than fine-tuned embedding distances, despite worse rankings of individual examples.

heading: 5.2 Predicting Model Accuracy (RMSE)
text: As described in Section 4.5 and shown in Figure 1, a given drift metric can produce quality rankings of examples even if the raw predicted accuracies are far from the true model accuracies. Thus, as reported in Table 1, we evaluate RMSEs using different drift metrics to predict model accuracies for out-of-domain evaluation datasets.7
Decomposed drift has the best accuracy predictions. Decomposing into vocabulary, structural, and semantic drift results in better dataset-level accuracy predictions (lower RMSEs) than any previous drift metric(s), for all three multi-domain datasets. Accuracy predictions based on individual dimensions vary (e.g. individual dimensions are sometimes better than including all three dimensions), but predicting out-of-domain accuracy from all three dimensions results in reliably low errors compared to previous metrics. Across the three datasets, our decomposed drift results in an aver-
7We only consider accuracy prediction RMSEs for out-ofdomain datasets because sufficiently sized in-domain datasets have very low variation in model accuracy.
age decrease of 16.8% in accuracy prediction error (RMSE) when compared to the best metric from previous work.
Fine-tuned embedding distances have poor accuracy predictions. The fine-tuned embedding distances result in worse out-of-domain accuracy predictions (higher RMSEs) than our decomposed vocabulary, structural, and semantic drift for all three multi-domain datasets. Notably, they have by far the worst out-of-domain accuracy predictions of any drift metric for MNLI and sentiment classification split temporally. Across all three datasets, finetuned embedding distances result in an average of 2.03x more error than our decomposed vocabulary, structural, and semantic drift. This contrasts with fine-tuned embedding distances’ ability to rank individual examples by expected performance better than any other metric(s). This suggests that despite maintaining relative distances that are predictive of relative model performance for individual examples (high ROC AUCs), fine-tuning adjusts the example embeddings such that raw distances are not predictive of raw out-of-domain accuracies (high RMSEs). Concretely, the logistic regressions fit to fine-tuned embedding distances yield examplelevel probabilities that are highly predictive of relative model performance between out-of-domain examples, but quite far from the actual expected probabilities of getting each example correct. In practice, this suggests that fine-tuned embedding distances should be used in scenarios where the relative performance of evaluation examples is important (e.g. establishing drift threshold values), but they should not be used to predict actual out-ofdomain model accuracies.

heading: 6 Discussion
text: We find that decomposing linguistic dataset drift into our proposed vocabulary, structural, and semantic drift metrics leads to improved out-ofdomain dataset-level accuracy predictions for sentiment classification and NLI. Furthermore, our decomposed drift metrics produce better rankings of individual examples by expected performance than previous model-agnostic drift metrics (e.g. token frequency divergences and pre-trained embedding distances), both in-domain and out-of-domain. Although fine-tuned embedding distances produce by far the best example rankings, they also produce egregiously incorrect out-of-domain model accuracy predictions. Our results suggest that fine-tuned
embedding distances should still be used in cases where examples need to be ranked by expected performance (e.g. relative to a cutoff value, as in Elango et al., 2022). Vocabulary, structural, and semantic drift should be used in cases where either (1) the internal states of a model are unavailable, which is increasingly common as models are accessed through external APIs, (2) the same metric values need to be applied across multiple models (i.e. model-agnostic metrics), or (3) raw model accuracy predictions are desired.
Our work also opens up future directions of research studying specific effects of linguistic dataset drift on NLP model performance. First, future work might assess whether there are systematic effects of particular drift dimensions on specific tasks or model architectures. Second, it might consider new types of linguistic drift, potentially extending beyond domain drift (drift in P (x)) to consider concept drift P (y|x) in NLP (Webb et al., 2016). Finally, future work might investigate methods of quantifying drift in natural language generation, where the outputs y are linguistic data. Our work lays the groundwork for these future investigations.

heading: 7 Conclusion
text: We propose three dimensions of linguistic dataset drift—vocabulary, structural, and semantic drift— and we modify previous performance prediction methods to predict NLP model performance at the individual example level along with the dataset level. We validate existing drift metrics for particular use cases (e.g. fine-tuned embedding distances for example ranking), and we highlight complementary use cases where our decomposed drift metrics outperform previous metrics (e.g. when predicting model accuracies or when using model-agnostic metrics). Our work lays the foundation for future research into specific and interpretable dimensions of linguistic dataset drift, improving our ability to predict NLP model performance on real world data.
Limitations
Our work has several limitations. First, our experiments are limited by the multi-domain datasets available for sequence classification tasks, limiting both our task coverage (sentiment classification and NLI) and domain type coverage (product categories, temporal splits, and text source domains). Future work can evaluate our drift metrics on token classification tasks or even sequence-to-sequence
tasks by predicting sequence-level performance (e.g. proportions of correct tokens, or examplelevel BLEU scores; Papineni et al., 2002) from our example-level drift metrics. Past work has already considered dataset-level drift metrics and performance predictions for token classification tasks such as named entity recognition (NER) and part-of-speech (POS) tagging (Ramesh Kashyap et al., 2021; Rijhwani and Preotiuc-Pietro, 2020), and example-level drift metrics have been used in machine translation for training data example selection (Axelrod et al., 2011; Wang et al., 2017). We hope that future work will evaluate example-level drift metrics in their ability to predict NLP model performance on this wider variety of tasks.
Second, we only consider simple logistic regressions to predict whether individual examples will be predicted correctly by different models. More complex classifiers (e.g. XGBoost; Chen and Guestrin, 2016) might improve performance predictions, particularly if more drift metrics are included as inputs, or if raw example features are included (e.g. sequence length; Ye et al., 2021). Our three dimensions of linguistic drift (vocabulary, structural, and semantic drift) represent just one way of decomposing linguistic dataset drift into distinct dimensions. We hope that future work will explore novel dimensions of linguistic drift, identifying new ways of integrating different drift metrics into NLP model performance predictions across tasks and domains.

heading: A Appendix
text: A.1 Model fine-tuning details
For each sentiment classification and NLI training domain in Section 4, we fine-tune a RoBERTa basesize model using the hyperparameters in Table 2 and the pre-trained RoBERTa model from Hugging Face, containing approximately 123M parameters (Liu et al., 2019; Wolf et al., 2020). Because there are only five training domains for MNLI, we run five fine-tuning runs per MNLI training domain; otherwise, we run one fine-tuning run per domain (43 domains for sentiment classification split by product category, 15 domains for sentiment classification split by review year). All models are finetuned using one Tesla V100 GPU, taking about two hours per model.
A.2 Efficient cosine distance computations In Section 4.3, we compute the mean cosine distance between each evaluation example embedding x and all training example embeddings from Dtrain. Each example embedding is computed by taking the mean over all tokens in the example and the last two RoBERTa layers (before or after fine-tuning, as specified; Elango et al., 2022). Mean embedding cosine distances are also computed for individual tokens when quantifying lexical semantic change in Section 3.3. To avoid saving the embedding for each example in Dtrain and computing each cosine distance individually, we note that the mean pairwise cosine similarity between a set of vectors U and V is:
Mean u∈U,v∈V
(cos(u, v)) = 1 |U ||V | ∑
u∈U,v∈V
⟨u, v⟩ ||u|| · ||v||
= 1 |U ||V | ∑
u∈U
∑
v∈V
〈 u
||u|| , v ||v||
〉
= 1 |U ||V | ∑
u∈U
〈 u
||u|| , ∑
v∈V
v
||v||
〉
= 1
|U ||V |
〈∑
u∈U
u ||u|| , ∑
v∈V
v
||v||
〉
=
〈 1
|U | ∑
u∈U
u ||u|| , 1 |V | ∑
v∈V
v
||v||
〉
= 〈 Mean u∈U ( u ||u|| ) ,Mean v∈V ( v ||v|| )〉
In other words, we only need to compute the dot product between the mean normed vector for U and V . For our uses, when computing the mean cosine distance between an example embedding x and all training example embeddings from Dtrain, we need only compute one minus the dot product between the normed x and the mean normed embedding over all examples in Dtrain. This way, we only need to store one vector (the mean normed embedding) for the entire training set, rather than one vector per training example.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section (unnumbered).
7 A2. Did you discuss any potential risks of your work? Our work analyzes existing models (RoBERTa) on existing datasets, predicting their performance for out-of-domain data. Rather than introducing new models and risks, we hope that these results can be used to reduce the potential risks of applying existing models in cases where they might perform poorly.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section 1 (Introduction).
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Datasets and models in Section 4 (Experiments): Amazon Reviews dataset, MNLI dataset, and Hugging Face RoBERTa model.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 (Experiments).
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The datasets and models used are publicly available with citation.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 (Experiments).
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets used are standard datasets, used as provided in their publicly available form.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4 (Experiments). More details can be found in the cited work corresponding to each dataset used.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4 (Experiments).
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4 (Experiments).

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 (Experiments), Appendix A.1 (Model fine-tuning details).
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. No hyperparameter search was used. RoBERTa hyperparameters are included in Appendix A.1 (Model fine-tuning details).

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 (Experiments), Section 5 (Results), Appendix A.1 (Model fine-tuning details).

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 (Experiments), Appendix A.1 (Model fine-tuning details). Used spaCy and the Hugging Face library.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Our work lays the foundation for future research into specific and interpretable dimensions of linguistic dataset drift, improving our ability to predict NLP model performance on real world data. Limitations
Our work has several limitations. First, our experiments are limited by the multi-domain datasets available for sequence classification tasks, limiting both our task coverage (sentiment classification and NLI) and domain type coverage (product categories, temporal splits, and text source domains). Future work can evaluate our drift metrics on token classification tasks or even sequence-to-sequence
tasks by predicting sequence-level performance (e. g. proportions of correct tokens, or examplelevel BLEU scores; Papineni et al. , 2002) from our example-level drift metrics. Past work has already considered dataset-level drift metrics and performance predictions for token classification tasks such as named entity recognition (NER) and part-of-speech (POS) tagging (Ramesh Kashyap et al. , 2021; Rijhwani and Preotiuc-Pietro, 2020), and example-level drift metrics have been used in machine translation for training data example selection (Axelrod et al. , 2011; Wang et al. , 2017). We hope that future work will evaluate example-level drift metrics in their ability to predict NLP model performance on this wider variety of tasks. Second, we only consider simple logistic regressions to predict whether individual examples will be predicted correctly by different models. More complex classifiers (e. g. XGBoost; Chen and Guestrin, 2016) might improve performance predictions, particularly if more drift metrics are included as inputs, or if raw example features are included (e. g. sequence length; Ye et al. , 2021). Our three dimensions of linguistic drift (vocabulary, structural, and semantic drift) represent just one way of decomposing linguistic dataset drift into distinct dimensions. We hope that future work will explore novel dimensions of linguistic drift, identifying new ways of integrating different drift metrics into NLP model performance predictions across tasks and domains