Measuring the Instability of Fine-Tuning
Year: 2023

Abstract:
Fine-tuning pre-trained language models on downstream tasks with varying random seeds has been shown to be unstable, especially on small datasets. Many previous studies have investigated this instability and proposed methods to mitigate it. However, most studies only used the standard deviation of performance scores (SD) as their measure, which is a narrow characterization of instability. In this paper, we analyze SD and six other measures quantifying instability at different levels of granularity. Moreover, we propose a systematic framework to evaluate the validity of these measures. Finally, we analyze the consistency and difference between different measures by reassessing existing instability mitigation methods. We hope our results will inform the development of better measurements of fine-tuning instability.1

heading: 1 Introduction
text: Since the introduction of BERT (Devlin et al., 2019), the pre-train-then-fine-tune paradigm has achieved state-of-the-art performance across many NLP benchmarks (Sun et al., 2021; Fedus et al., 2022; Chi et al., 2022). However, despite its wide success, the fine-tuning process, especially when fine-tuning large models on small datasets, is shown to be unstable: fine-tuning a given model with varying random seeds can lead to different performance results (Lee et al., 2020; Dodge et al., 2020; Mosbach et al., 2021; Hua et al., 2021). This instability makes the investigation of better architectures and instability mitigation methods (IMMs) challenging (Zhang et al., 2021).
Many previous studies have investigated finetuning instability (Dodge et al., 2020; Lee et al., 2020; Mosbach et al., 2021; Zhang et al., 2021). In these studies, the most prevalent instability measure is the standard deviation of performance (SD,
1Our implementation is available at https://github. com/nlpsoc/instability_measurement.
e.g. the standard deviation of F1-scores). However, as we discuss in §3 and §6, SD can only offer very limited assessments. For example, classifiers can obtain the same accuracy score (i.e. zero SD) even when they neither make the same predictions on each example (prediction instability) nor have the same hidden representations (representation instability). Therefore, it is important to also use other measures that can address the weaknesses of SD.
However, it is difficult to decide which measures to use: because instability is an abstract concept, it is hard to examine to which extent a measure indeed quantifies what it intends to measure. This property is called validity in measurement theory (William M. K., 2023). For example, using the average accuracy of models as an instability measure would have low validity, because how accurate these models make predictions does not reflect their stability.
To better assess the instability of fine-tuning pretrained language models (PLMs), we study more measures concerning instability at different granularity levels (Summers and Dinneen, 2021; Khurana et al., 2021; Raghu et al., 2017; Kornblith et al., 2019; Ding et al., 2021) and develop a framework to assess their validity. We focus on BERT and RoBERTa for their popularity, but our framework can also be applied to other PLMs. Concretely, • First, we discuss six other instability measures
at different granularity levels in addition to SD, and categorize them into prediction measures and representation measures based on the type of instability they focus on (§3). • Second, we propose a framework to systematically assess two types of validity of these measures, without relying on labelled data (§5). • Third, we investigate the consistency and differences between different measures by reassessing the effectiveness of existing IMMs, analyzing their correlations (§6.1), and performing bootstrap analyses (§6.2). We find that measures at
6209
different granularity levels do not always produce consistent instability scores with each other and tend to differ more when the models are more stable. Moreover, based on our observations, we offer two suggestions for future studies: (1) use multiple instability measures, especially when models are more stable; (2) use only one prediction and one representation measure when limited computational resources are available (§6.3).

heading: 2 Background

heading: 2.1 Instability of Fine-tuning
text: The seminal work of BERT by Devlin et al. (2019) has already shown that fine-tuning PLMs is unstable regarding the choices of random seeds. This observation was further confirmed by other studies on more PLMs, including RoBERTa (Liu et al., 2019; Lan et al., 2020; Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020; Mosbach et al., 2021; Zhang et al., 2021; Sellam et al., 2022). Most of these studies used SD to measure the instability.
Different explanations have been proposed to account for the instability of fine-tuning PLMs on small datasets, including catastrophic forgetting (Lee et al., 2020)2, the lack of Adam bias correction (Mosbach et al., 2021; Zhang et al., 2021), too few training steps (Mosbach et al., 2021), and taskspecific top layers (Zhang et al., 2021).

heading: 2.2 Instability Mitigation Methods (IMMs)
text: Various IMMs have been used to mitigate the instability of fine-tuning PLMs. Following Zhang et al. (2021), we focus on four methods for their popularity. Nevertheless, we acknowledge the existence of other methods, including entropy regularization and co-distillation (Bhojanapalli et al., 2021), and component-wise gradient norm clipping (Yang and Ma, 2022).
Mixout (Lee et al., 2020) is a generalized version of Dropout (Srivastava et al., 2014). It randomly replaces the outputs of neurons with the ones produced by the pre-trained weights by a probability p. In this way, it can mitigate the catastrophic forgetting of pre-trained knowledge which potentially stabilizes fine-tuning.
2Although several studies assumed that catastrophic forgetting causes instability (Kirkpatrick et al., 2017; Schwarz et al., 2018; Lee et al., 2020), Mosbach et al. (2021) argued against it.
WDpre (Li et al., 2018) is a variant of weight decay: after each optimization step, each model weight w will move a step size of λw towards the pre-trained weights, where λ is a hyper-parameter. WDpre also aims to improve the fine-tuning instability by mitigating catastrophic forgetting.
Layer-wise Learning Rate Decay (Howard and Ruder, 2018, LLRD) assigns decreasing learning rates from the topmost layer to the bottom layer by a constant hyper-parameter discounting factor η. Howard and Ruder (2018) empirically show that models trained using LLRD are more stable, by retaining more generalizable pre-trained knowledge in bottom layers, while forgetting specialized pre-train knowledge in top layers.
Re-init (Zhang et al., 2021) stabilizes fine-tuning by re-initializing the top k layers of PLMs. The underlying intuition is similar to LLRD: top layers of PLMs contain more pre-train task specific knowledge, and transferring it may hurt stability.

heading: 3 Instability Measures
text: Despite its wide usage, SD only provides a narrow view of the instability of models. For example, consider fine-tuning two pre-trained models on the same classification task. If one of them makes correct predictions only on half of the test data, while the other model makes correct predictions only on the other half, these two models will both have a 0.5 accuracy score and therefore no instability would be measured using SD. However, they actually make different predictions on each data point (i.e. prediction instability). Moreover, even if these two models achieve the same accuracy by making identical predictions, due to the overparameterization of PLMs (Roeder et al., 2021), they can have different sets of hidden representations (i.e. representation instability).
To better assess these two types of instability, we study six other instability measures at different granularity levels in addition to SD. Furthermore, according to the instability types that these measures intend to quantify, we categorize these measures into two types: prediction measures (§3.1) and representation measures (§3.2). All these instability measures have a continuous output range 0–1, with higher values indicating lower stability.
It is worth noting that similar categorizations have been used before. For example, Csiszárik et al. (2021) categorized measures as functional and
representational. However, they used functional similarity to refer to the function compositions that different components of the models realize. Also, Summers and Dinneen (2021) categorized measures as performance variability and representation diversity. However, they used performance variability to specifically refer to SD and used representation diversity to refer to all other measures at different granularity levels that we study here.
Notation Formally, suppose we have a dataset consisting of n data points. We fine-tune m BERT models {M1,M2, . . . ,Mm}, with the same settings except for m different random seeds. We use pki and ŷ k i to denote the class probability and the prediction of Mi on the k-th test sample. Assume the l-th layer of Mi consists of e neurons, we use M li ∈ Rn×e to denote this layer’s centered representation, w.r.t. all n data points (all representation measures discussed below require us to center the representations). Representation measures involve computing the distances between the representations derived from the same layer of two different models. We use dli,j to represent the distance between M li and M l j .

heading: 3.1 Prediction Measures
text: We refer to measures that assess the prediction instability of models as prediction measures. In other words, prediction measures only assess the output of the models (i.e. logits and predictions). In this paper, we study three prediction measures besides SD: pairwise disagreement, Fleiss’ Kappa, and pairwise Jensen-Shannon divergence (pairwise JSD). Among these three measures, both pairwise disagreement and Fleiss’ Kappa quantify the instability of the discrete predictions of models, and therefore are at the same granularity level. Pairwise JSD looks at continuous class probabilities and is thus more fine-grained. Nevertheless, they are all more fine-grained than SD, which only considers the overall performance.
Pairwise Disagreement Following Summers and Dinneen (2021), we measure the models’ instability by averaging the pairwise disagreement among models’ predictions. Formally,
Ipwd = 2 nm(m− 1) m∑
i=1
m∑
j=i+1
n∑
k=1
1(ŷki 6= ŷkj ),
where 1 is the indicator function. We note that our definition of pairwise disagreement relates
closely to churn and jitter proposed and used by Milani Fard et al. (2016) and Bhojanapalli et al. (2021); Liu et al. (2022).
Fleiss’ Kappa Similar to Khurana et al. (2021), we adopt Fleiss’ Kappa, which is a popular measure for inter-rater consistency (Fleiss, 1971), to measure the consistency among different models’ predictions. Because Fleiss’ Kappa is negatively correlated with models’ instability and ranges from 0 to 1, we use its difference with one as the output, to stay consistent with other measures. Formally,
Iκ = 1− pa − p 1− p ,
where pa is a term evaluating the consistency of models’ predictions on each test sample, and p is an error correction term (Details in Appendix B).
Pairwise JSD The previous two measures only look at discrete labels, while continuous class probabilities contain richer information about a model’s predictions. Therefore, we average the pairwise JSD of models’ class probabilities to obtain a finergrained evaluation of instability. Formally,
IJSD = 2 nm(m− 1) m∑
i=1
m∑
j=i+1
n∑
k=1
JSD(pki ‖pkj ),
where JSD(·‖?) is the JSD between · and ?.

heading: 3.2 Representation Measures
text: We refer to measures that assess the instability of models based on their hidden representations as representation measures. Here, we study three representation measures: singular vector canonical correlation analysis (SVCCA, Raghu et al., 2017), orthogonal Procrustes distance (OP, Schönemann, 1966), and linear centered kernel alignment (Linear-CKA, Kornblith et al., 2019). Because all representation measures look at the hidden representations of models, they are at the same granularity level, which is more fine-grained than prediction measures.
All these three measures are originally developed to compute the distance between a pair of representations (although Linear-CKA is also used by Summers and Dinneen (2021) to study model instability). With these measures, we are able to analyze the behavior of neural networks, going beyond the model predictions alone (Kornblith et al., 2019). To evaluate the instability of all m models
regarding a specific layer l, I l, we average the distance d of each possible pair of models. Formally,
I l = 2m(m−1) ∑m i=1 ∑m j=i+1 d l i,j .
We next describe how to calculate d for each representation measure. We respectively denote the instability score of each measure after aggregating d as ISVCCA, ICKA, and IOP. SVCCA (Raghu et al., 2017) is developed based on canonical correlation analysis (CCA, Hardoon et al., 2004). For two representations M li and M l j , CCA finds e orthogonal bases so that their correlations after projection are maximized.
Formally, for 1 ≤ k ≤ e,
ρk = max wki ,w k j
corr ( M liw k i ,M l jw k i ) ,
s.t. ∀k1 < k2, M liwk1i ⊥M liwk2i and M ljwk1j ⊥M ljwk2j ,
where wki ,w k j ∈ Rp1 . After obtaining ρ, we use the mean correlation coefficient to transform ρ into a scalar dissimilarity measure. Formally,
dCCA = 1− 1e ∑e k=1 ρk.
Raghu et al. (2017) find that meaningful information usually distributes in a lower-dimensional subspace of the neural representations. To avoid overfitting on noise, SVCCA first uses singularvalue decomposition to find the most important subspace directions of the representations.3 The representations are then projected onto these directions, followed by CCA. We again calculate the mean ρ as the dSV CCA.
OP (Ding et al., 2021) consists of computing the minimum Frobenius norm of the difference between M li and M l j , after M l i being transformed by an orthogonal transformation. Formally,
min R ‖M lj −M liR‖2F, s.t. R>R = I.
Schönemann (1966) provides a closed-form solution of this problem. To constrain the output range to be between zero and one, we normalize the representations with their Frobenius norms. Formally,
dOP(M l i ,M l j) = 1− ‖M l>i M lj‖∗ ‖M l>i M li‖F‖M l>j M lj‖F ,
where ‖ · ‖∗ is the nuclear norm. 3Following Raghu et al. (2017), we keep directions that explain 99% of the representations.
Linear-CKA measures the representation distance by the similarity between representations’ inter-sample similarity 〈M l>i M li ,M l>j M lj〉 (Kornblith et al., 2019). After normalizing the representations with Frobenius norms, we obtain a similarity score between zero and one. We then use its difference with one as the distance measure. Formally,
dCKA(M l i ,M l j) = 1− ‖M l>i M lj‖2F ‖M l>i M li‖F‖M l>j M lj‖F .

heading: 4 Experimental Setup
text: We study the instability of fine-tuning BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) empirically. Following Lee et al. (2020) and Zhang et al. (2021), we perform our experiments on three small datasets of the GLUE benchmark (Wang et al., 2018): RTE, MRPC (Dolan and Brockett, 2005), and CoLA (Warstadt et al., 2019), because models trained on small datasets are observed to be less stable (Zhang et al., 2021).4
Unless specified, we fine-tune BERT-large and RoBERTa-large models from HuggingFace Transformers (Wolf et al., 2020), with a 16 batch size, a 0.1 Dropout rate, and a 2 × 10−5 learning rate, using de-biased Adam, as well as a linear learning rate warm-up during the first 10% steps followed by a linear decay, following Zhang et al. (2021). Consistent with Mosbach et al. (2021), we train the models for five epochs with 20 random seeds.
Consistent with Zhang et al. (2021), we divide the validation data into two equally sized parts, respectively as new validation and test data, because we have no access to the GLUE test datasets. Moreover, we keep the checkpoint with the highest validation performance and obtain all our results on the test set. More details are provided in Appendix A.

heading: 5 Assessing the Validity of Instability Measures
text: It is not trivial to assess the validity of instability measures, because there is no clear ground truth. Nevertheless, we can still perform validity assessments by building on approaches from measurement theory. Here, we propose a framework to assess two important types of validity (William M. K., 2023), by computing their correlations with
4To study whether our findings also generalize to larger datasets, we include a pilot study on SST-2 (8× larger than CoLA) in Appendix C. As expected, we observe higher stability. Furthermore, the behaviors of measures are consistent with those observed on smaller datasets.
each other (convergent validity, §5.1) and observing their responses to different inputs (concurrent validity, §5.2). Except for SVCCA, all other measures show good validity in our tests, and hence they are suitable for examining fine-tuning instability.5 Although there are other types of validity (e.g. face, content, discriminative, and predictive validity), we select these two types because of their relevance to our study and our lack of labelled test data. Ding et al. (2021) also provided a framework to evaluate the sensitivity (i.e. responding to important changes) and specificity (i.e. ignoring changes that do not matter) of the representation similarity metrics (i.e. dCKA, dOP, dSVCCA). However, their framework was not build on validity theory and they did not consider prediction measures.

heading: 5.1 Convergent Validity
text: In measurement theory, convergent validity refers to validity established by correlations with measures that are theoretically related (Gravetter and Forzano, 2018). In other words, if two measures aim to quantify the same underlying concept, their measurements should have a high correlation. It is worth noting that convergent validity usually should be evaluated against established and validated measures (Jackman, 2008). However, in our case, none of the measures have been validated before. Therefore, low convergent validity may have different causes: for example, it can be that only one of the measures is invalid, or that these measures quantify different aspects of the concept.
5We note that passing our tests does not necessarily imply that a measure is perfectly valid: it is also possible that our validity tests/datasets/PLMs are not comprehensive (i.e. monomethod and mono-operation biases, William M. K., 2023). Moreover, as aforementioned in §3, different measures may concern different aspects of instability and should usually be used together. We offer a more extensive discussion in §6.
For representation measures, we have an instability score for each hidden layer. We therefore assess their convergent validity by computing Pearson’s r between instability scores that different measures assign to different layers of the same group of models (e.g. BERT fine-tuned on RTE with different random seeds). We show the results on BERT in Figure 1a. All three representation measures correlate highly (> 0.77) with each other, which suggests a good convergent validity. For prediction measures, we only have a single scalar output on each dataset/PLM combination. It is thus not practical to estimate their convergent validity directly because the sample size (i.e. the number of dataset/PLM combinations) is too small. In §6, we offer a detailed discussion and observe that they actually show good convergent validity.

heading: 5.2 Concurrent Validity
text: In measurement theory, concurrent validity refers to the “ability to distinguish between groups that it should theoretically be able to distinguish between” (William M. K., 2023). We therefore test concurrent validity based on the following assumption: a valid instability measure should not only be able to distinguish groups of models with substantial instability differences, but also be unable to distinguish groups of models with trivial instability differences. Concretely, we treat substantial instability differences as the differences between successful/failed fine-tuning runs, and define trivial instability differences as different i.i.d. test datasets.6 We accordingly present two analyses.
Differences between successful and failed runs Previous studies have identified failed fine-tuning
6Our tests are inspired by the concurrent validity definition, rather than strictly following it. See the limitations section.
runs where the training fails to converge (Dodge et al., 2020; Mosbach et al., 2021).7 In particular, Mosbach et al. (2021) observe that failed runs suffer from vanishing gradients. Because all runs start from the pre-trained weights, and the vanishing gradient makes the models update less intensively, this observation leads to the following assumption: compared with successful runs, failed runs bear lower representation instability. In this analysis, we use this assumption to evaluate the concurrent validity of representation measures, by testing whether they are able to distinguish failed from successful runs. Because of this former observation only applies to hidden representations, in this analysis we exclude prediction measures.
Specifically, we train our models using the same 20 random seeds and keep the last checkpoint for each seed. We adopt larger learning rates: 5×10−5 for BERT and 3 × 10−5 for RoBERTa, because failed runs occur more frequently with larger learning rates (Mosbach et al., 2021). For each group of models, we obtain 9–13 failed runs out of 20 runs.
We show our results for BERT on RTE in Figure 1b and observe similar patterns on other PLMs/datasets (see Appendix E). Linear-CKA and OP indeed indicate a lower instability in failed runs. This observation is consistent with our expectation, suggesting the concurrent validity of these two measures. However, SVCCA fails to distinguish successful and failed runs based on the representations in the bottom layers, and therefore fails this test. One plausible explanation is that because lower layers of models tend to update less intensively during fine-tuning (the nature of back-propagation), they are likely to be more stable, and SVCCA may ignore these smaller differences.
Differences among test datasets Because we aim to quantify the instability of models themselves, one desideratum of a valid measure is to be independent of the specific data samples used to obtain the predictions and representations of models, as these data samples are not inherent components of these models. Concretely, we expect a valid measure to produce similar outputs for the same group of models when the instability scores are computed using different i.i.d. datasets.
To evaluate the input invariance of the measures,
7Following Dodge et al. (2020) and Mosbach et al. (2021), we define failed runs as runs whose accuracies at the end of the training ≤ a majority classifier (i.e. a classifier that consistently predicts the majority label regardless of the inputs).
we create four sub-samples with half the test dataset size for each task, by uniformly sampling without replacement. We then compute the instability scores using both prediction and representation measures on all samples, and show the results for BERT on MRPC in Figure 1c (we include results for RoBERTa and on MRPC/CoLA in Appendix E). We observe that the variance among different samples is very small, suggesting that all these measures show good concurrent validity in this test.8

heading: 6 The Need to Use Different Measures
text: In §5, all measures discussed in §3 except for SVCCA showed good validity in our tests, thus they are capable of measuring fine-tuning instability. However, the following question remains: when do we need which instability measures? In this section, we explore this question via two studies. First, we reassess the effectiveness of existing IMMs by comparing the results when using different measures (§6.1). Second, we further analyze the relationship between different measures using bootstrapping analyses (§6.2). We observe that measures at different granularity levels show better consistency when the models are less stable and vice versa. Moreover, based on our findings, we provide two concrete suggestions for selecting instability measures for future studies (§6.3).

heading: 6.1 Reassessing IMMs
text: To study the relationships between different measures, we reassess existing IMMs from §2.2 and compare their instability scores. We include all measures discussed in §3, except for SVCCA because it did not show good validity in our tests.
Experimental setup For each IMM, we train 10 models using different random seeds, with the same hyper-parameters in §4. We also adopt the IMM hyper-parameters from Zhang et al. (2021): Mixout p = 0.1, WDpreλ = 0.01, LLRD η = 0.95, and Re-init k = 5. We compare the IMMs against a baseline (Standard), which does not use any IMM.9
8To investigate the impact of sample sizes, we also include the results of sample 10% of the test datasets in Appendix D. Despite that the differences between different sub-sampled datasets are larger compared to 50%, we still observe good concurrent validity, especially in lower layers.
9Note that our Standard baseline is different from the one in Zhang et al. (2021). Ours uses a longer training time (five epochs) and a standard weight decay regularization, because these two choices can be seen as hyper-parameter selections. As observed by Mosbach et al. (2021), these modifications make our baseline stronger.
Measures of different granularity levels are not always consistent with each other We show the results of prediction measures and ICKA on BERT in Table 1 and Figure 2 (other results show similar trends, see Appendix E). We observe that different measures are not always consistent with each other. For example, when using BERT on MRPC (Table 1), SD shows Mixout to be the most stable training scheme. However, the other three prediction measures and ICKA (the top layer, Figure 2b) rank Mixout to be the (second) least stable one.
To better quantify the inconsistency, we calculate Kendall’s τ between the rankings of IMMs based on different measures, on each dataset/PLM. We include the full results in Appendix E and make two observations. First, measures of similar granularity level tend to be consistent with each other. For example, ICKA ∼ IOP (both representation measures) and Ipwd ∼ Iκ (both based on discrete predictions) show good consistency (i.e. τ ≥ 0.8) on each combination of models and datasets. Also, Ipwd and Iκ show better consistency with Ijsd (τ ≥ 0.6) than with SD (τ = −0.2 for BERT on MRPC). Second, the consistency among measures differs across datasets and models. For example, all measures correlate well for BERT on RTE, with a minimum τ ≈ 0.6. In contrast, the correlations derived from MRPC are much smaller, with closeto-zero τ values between SD and other measures.
Most IMMs are not always effective Our results also show that most IMMs are not always effective: they sometimes fail to improve stability compared to the Standard baseline, which is consistent with the observations of Zhang et al. (2021). In fact, Re-init is the only IMM that consistently improves over Standard according to all measures. Also, for BERT on RTE, Standard is the third most stable training method according to all prediction (Table 1) and representation (Figure 2) measures. Generally, models trained with WDpre and Mixout are less stable compared to models trained with LLRD and Re-init. Because both WDpre and Mixout aim to stabilize fine-tuning by resolving catastrophic forgetting, these results suggest that catastrophic forgetting may not be the actual or sole cause of instability, which is consistent with the observations of Mosbach et al. (2021).

heading: 6.2 Bootstrapping Analyses
text: In §6.1, we computed Kendall’s τ between the rankings of different IMMs obtained using different instability measures. However, because we only have five groups of models (i.e. each group consists of 10 models trained with the same IMM/Standard baseline but different random seeds), the results we obtain may be less accurate. To mitigate this issue, in this section we focus on generating more groups for a specific IMM-dataset combination.
Unfortunately, generating more groups is extremely expensive, as each group consists of 10 models trained with different random seeds.10
To avoid the high training cost, we instead use bootstrapping to generate more groups of models. Concretely, for each group of 10 models (i.e. 10 different random seeds), we sample 10 models with replacement for 1000 times to obtain 1000 groups of models. We then compute the Pearson’s r between each pair of measures using these groups. We apply the representation measures on the topmost layer to make the results more comparable with the prediction measures.
We show the results for the BERT Standard baseline on MRPC and CoLA in Figure 3 and observe similar trends on other datasets/models (see Appendix E). We make two observation. First, consistent with §6.1, we observe that measures at closer granularity levels have higher correlations with each other. For example, SD has correlations of a decreasing strength with other measures on MRPC: from the most similar and coarse-grained Fleiss’ Kappa and pairwise disagreement, to the pairwise JSD in between, and finally to the furthest and the most fine-grained representation measures OP and Linear-CKA. Also, the correlations between the two representation measures (ICKA and IOP ) are much higher than that between them and other measures. Second, we also observe that correlations obtained from different combinations of dataset/IMM are different from each other, which is expected.
The second observation points to another question: when will different measures be more consistent with each other? Intuitively, they will be
10For example, the minimum recommended sample size for Pearson’s r = 0.5 is 99 (Bonett and Wright, 2000). In our case, this means training 990 models (i.e. 99 groups).
more consistent when the differences between models are large. In this case, both coarse-grained and fine-grained measures can detect the instability. In contrast, when the differences between models are small, only fine-grained measures can capture these nuances. In other words, instability measures are more consistent when the models are less stable, and vice versa.
To quantitatively check this intuition, on each PLM (i.e. BERT and RoBERTa), using the bootstrapping results on each dataset and IMM, we compute the Pearson’s r between 1) the average correlations between each measure and other measures and 2) SD values.11 We observe strong correlations on both BERT (r = 0.734, Figure 3c) and RoBERTa (r = 0.653, Appendix E), confirming our intuition.

heading: 6.3 Implications
text: In §6.1 and §6.2, we investigated the consistency and differences between different instability measures. Based on our observations, we provide two practical suggestions for future studies.
First, we observed that measures are not always consistent with each other, despite their good validity in §5. This observation suggests that different measures focus on different aspects of instability and therefore should be used together in future studies. Moreover, we observed that different measures tend to be less consistent with each other when the models themselves are more stable. This observa-
11Although sharing the same range -1–1, correlations between different measures usually have different scales of values. In other words, some measures are more consistent with other measures, and thus have larger correlations. To balance the weights of different measures, we standardize the correlations for each measure according to the its average correlations with other measures on different datasets/IMMs.
tion further demonstrates the necessity of adopting multiple measures when the instability assessed by one of the measures is low, and that using any measure alone may produce inaccurate conclusions.
Second, we observed measures at similar granularity levels to be more consistent. One can therefore start with SD, and sequentially add more finegrained measures when previous measures indicate low stability. Because computing fine-grained instability is often slow, only one prediction measure and one representation measure can be used when limited computational resources are available.

heading: 7 Conclusion
text: In this paper, we study measures that quantify the instability of fine-tuning PLMs. In addition to the most commonly used measure, SD, we study six other measures at different granularity levels and propose a framework to evaluate their validity. Using this framework, we show that all these measures except SVCCA have good validity. Moreover, by reassessing existing IMMs, we show that different instability measures are not always consistent, and that they are more consistent when the models are less stable. Finally, based on our observations, we offer two suggestions for selecting instability measures in future studies.
Limitations
Our study leaves room for future work. First, we would like to highlight the difficulty of applying the validity assessment framework from measurement theory to instability measures. For example, in §5.1, our low convergent validity scores may have different interpretations because there are no well-established instability measures. Further, in §5.2, because no previous studies have built theoretical foundations of factors that impact the prediction and representation instability, both our tests do not rigorously follow the concurrent validity definition: our first test of successful and failed runs is based on an assumption derived from observations of Mosbach et al. (2021) rather than theory, and our second test of differences among test datasets examines the consistency between theoretically indistinguishable groups instead of the differences between theoretically distinguishable groups.
Second, we only experimented with a limited number of tasks, instability measures, PLMs, and validity types. Future work can use our framework to further validate the generalizability of our
observations. For example, to apply our validity testing framework to larger datasets, to include other measures (e.g. functional similarity measures, Csiszárik et al., 2021 and jitter, Liu et al., 2022), to study generative PLMs (e.g. T5, Raffel et al., 2020 and OPT, Zhang et al., 2022), and to test other types and validity (e.g. discriminative and predictive validity).
Third, we focused on general text classification tasks in this paper. One promising direction is to investigate which measures to use for specific settings. For example, to extend our framework to more recent generative models (e.g. BART, Lewis et al., 2020 and GPT-3, (Brown et al., 2020)). However, in this case, because our prediction measures in §3 are only useful for classification, new prediction measures should be developed, and our tests should be adjusted accordingly.

heading: Acknowledgements
text: This work is part of the research programme Veni with project number VI.Veni.192.130, which is (partly) financed by the Dutch Research Council (NWO).

heading: A Experimental Setup
text: Running Environment All models are trained using a single NVIDIA RTX 6000 graphics card, with Python 3.7, PyTorch 1.10.1 (Paszke et al., 2019), Hugging Face Transformers 4.14.1 (Wolf et al., 2020), and CUDA 10.2. The total training time is approximately 70 GPU hours. We calculate the results of the instability measures on Intel Xeon E5-2699 CPUs, with Python 3.7 and Numpy 1.19.5 (Harris et al., 2020), taking approximately 24 CPU hours.
Validation/Test Split The GLUE benchmark is one of the most popular benchmarks in NLP research (Wang et al., 2018), which consists of 11 different tasks, including RTE, MRPC, and CoLA. We download and process the datasets using Hugging Face Datasets (Lhoest et al., 2021). Following Zhang et al. (2021), we split the original validation dataset into two parts of (almost) equal sizes, because we have no access to the test data. We then use one part as the new validation data to select checkpoints with the best performance, and we use the other part as the new test data to compute all instability measures. We provide the statistics of the splits in Table 2.

heading: B Details of Fleiss’ Kappa
text: Consider a k-class classification task, m different models, and a test dataset size of n. We denote the number of models which predict the i-th data point as the j-th class as xij . Clearly, we have∑k
j=1 xij = m, because each of themmodels will make a prediction on xi.
We estimate the proportion of pairs of models that agree on the i-th data point by
pi =
∑k j=1C (xij , 2)
C(m, 2) =
∑k j=1 x 2 ij −m
m(m− 1) ,
where C means the combination. We can then
calculate the mean value of pi as
pa= 1
n
n∑
i=1
pi
= 1
mn(m− 1)
  n∑
i=1
k∑
j=1
x2ij −mn
  .
Moreover, we estimate the error term as
p = k∑
j=1
( 1
nm
n∑
i=1
xij) 2.
After obtaining pa and p , we can calculate Fleiss’ Kappa as
C Impact of Dataset Size
To better understand the impact of using only small train datasets, we perform a preliminary study on SST-2 (Socher et al., 2013), which consists of 67,000 training samples, around eight times larger than the size of CoLA (8,000). We use the same hyper-parameter settings as in Section 4, namely a 16 batch size, a 0.1 Dropout rate, a 2 × 10−5 learning rate, using 20 different random seeds and de-biased Adam, without IMMs.
We computed the instability scores using different prediction measures. We also computed the correlations between representation measures and performed bootstrapping analyses. We show the results in Table 3 and Figure 4. We make three observations. First, as expected, we observe lower instability from models trained on SST-2 compared with models trained on the three small datasets we used in the main text. Second, consistent with our observations in Section 6, we observe that the correlations between different measures on SST-2 are
lower, because the models are more stable. Third, also consistent with our observations in Section 6, Figure 4 shows that measures at similar granularity levels are more consistent with each other. Our results on SST-2 suggest that our previous observations are generalizable to larger datasets.
D Impact of Subsample Size
To investigate the impact of sample sizes regarding the differences among different i.i.d. datasets, we also experimented with sampling only 10% of the test samples. We show the results in Figure 5–6. Sampling only 10% of the test samples does bring larger variances (compared with sampling 50%), but results on different samples are mostly still consistent, especially in the lower layers.

heading: E Additional Results
text: Figures are on the next page. In Figures 7 – 13, the Y-axis refers to the instability scores computed by different measures.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Finally, based on our observations, we offer two suggestions for selecting instability measures in future studies. Limitations
Our study leaves room for future work. First, we would like to highlight the difficulty of applying the validity assessment framework from measurement theory to instability measures. For example, in §5. 1, our low convergent validity scores may have different interpretations because there are no well-established instability measures. Further, in §5. 2, because no previous studies have built theoretical foundations of factors that impact the prediction and representation instability, both our tests do not rigorously follow the concurrent validity definition: our first test of successful and failed runs is based on an assumption derived from observations of Mosbach et al. (2021) rather than theory, and our second test of differences among test datasets examines the consistency between theoretically indistinguishable groups instead of the differences between theoretically distinguishable groups. Second, we only experimented with a limited number of tasks, instability measures, PLMs, and validity types. Future work can use our framework to further validate the generalizability of our
observations. For example, to apply our validity testing framework to larger datasets, to include other measures (e. g. functional similarity measures, Csiszárik et al. , 2021 and jitter, Liu et al. , 2022), to study generative PLMs (e. g. T5, Raffel et al. , 2020 and OPT, Zhang et al. , 2022), and to test other types and validity (e. g. discriminative and predictive validity). Third, we focused on general text classification tasks in this paper. One promising direction is to investigate which measures to use for specific settings. For example, to extend our framework to more recent generative models (e. g. BART, Lewis et al. , 2020 and GPT-3, (Brown et al. , 2020)). However, in this case, because our prediction measures in §3 are only useful for classification, new prediction measures should be developed, and our tests should be adjusted accordingly