Two-Stage Fine-Tuning for Improved Bias and Variance for Large Pretrained Language Models
Year: 2023

Abstract:
The bias-variance tradeoff is the idea that learning methods need to balance model complexity with data size to minimize both under-fitting and over-fitting. Recent empirical work and theoretical analyses with over-parameterized neural networks challenge the classic bias-variance trade-off notion suggesting that no such trade-off holds: as the width of the network grows, bias monotonically decreases while variance initially increases followed by a decrease. In this work, we first provide a variance decomposition-based justification criteria to examine whether large pretrained neural models in a fine-tuning setting are generalizable enough to have low bias and variance. We then perform theoretical and empirical analysis using ensemble methods explicitly designed to decrease variance due to optimization. This results in essentially a two-stage fine-tuning algorithm that first ratchets down bias and variance iteratively, and then uses a selected fixed-bias model to further reduce variance due to optimization by ensembling. We also analyze the nature of variance change with the ensemble size in lowand high-resource classes. Empirical results show that this two-stage method obtains strong results on SuperGLUE tasks and clinical information extraction tasks. Code and settings are available: https://github.com/christa60/ bias-var-fine-tuning-plms.git

heading: 1 Introduction
text: Transformer-based neural language models, such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019), have achieved state-of-the-art (SOTA) performance for a variety of natural language processing (NLP) tasks through the process of fine-tuning. Given an NLP task, the process often involves searching for optimal pretrained models and hyperparameters
∗ co-first authors
while continuing to train the pretrained model on a domain-specific dataset, with the aim of building generalizable and robust fine-tuned models.
Based on the classic notion of the bias-variance tradeoff (Geman et al., 1992), where increasing model capacity decreases bias but increases variance (leading to a U-shaped test error curve), large pretrained models (LPMs) should have large variance and overfit domain-specific data which is often sparsely labeled and extremely imbalanced for classification. However, recent empirical work and theoretical analysis of neural networks challenge this classic bias-variance trade-off notion (Neal et al., 2018; Yang et al., 2020). It has been suggested that no such trade-off holds: as the width/depth of the network grows, bias monotonically decreases while variance initially increases followed by a decrease. This is why transformerbased LPMs often achieve better performance compared to less complex models like long short-term memory (LSTM)-based models or feature-rich methods (e.g., support vector machines). In the context of the new bias-variance paradigm, these LPMs are seemingly complex enough to have low bias and variance, however, so far there has been no method to justify whether those SOTA models are generalizable and robust in solving a variety of downstream tasks. In this paper, we (1) show that many SOTA models are very sensitive to data and training randomness, and (2) provide a variance decomposition-based justification method.
We also aim to improve model performance, reducing the generalization error of LPMs by reducing their bias and variance. Recent findings in Yang et al. (2020) show that the generalization error mainly comes from bias. Bias can be reduced by modifying the model architecture, e.g., making the neural networks wider and deeper as in transformer-based LPMs. However, pretraining new or larger language models can be challenging due to the technical and computational resource
15746
requirements afforded by only a few institutions – a topic outside the scope of this paper. We focus on the problem of reducing variance of neural models to further boost model performance, given a fixed bias (i.e., a fixed pretrained model). Ensemble methods have been successful in boosting predictive performance of single learners (Ren et al. (2016) presents a comprehensive review) and thus are a promising venue to explore.
We propose a two-stage fine-tuning framework that first justifies the generalization status of a selected pretrained model through a concrete metric, and then uses the fixed-bias model to further reduce variance due to optimization through ensembling techniques. To the best of our knowledge, we are the first to provide such a metric and perform theoretical and empirical analysis using ensembles for improved bias and variance for LPMs. We conduct experiments on the SuperGLUE tasks as well as on information extraction from clinical text as it is a domain of high significance and presents data challenges due to the limitations of sharing patientrelated data. We believe our proposal is of interest to any unstructured domain where neural models are used. Specifically we make the following contributions:
• We propose a two-stage fine-tuning algorithm for improving bias and variance in the new bias-variance paradigm;
• We provide a variance decomposition-based strategy to examine whether LPMs in finetuning settings are generalizable enough to have low bias and variance;
• We perform theoretical and empirical analyses using ensembles explicitly designed to decrease variance due to optimization while keeping bias unchanged;
• We analyze the nature of variance change due to ensembling in low- and high-resource classes in classification tasks;
• We conduct comprehensive experiments and show that the proposed two-stage method obtains strong results on SuperGLUE tasks and two clinical NLP tasks.

heading: 2 Preliminaries
text: In this section we present the bias-variance decomposition for squared loss in the new paradigm studied in Neal et al. (2018) and Yang et al. (2020).
We also present a further decomposition of variance. We denote f as a supervised learning task such that f : X → Y , based on a training dataset S of m i.i.d. samples drawn from a joint distribution D of (X ,Y). The learning target is to minimize the mean squared error E(f) = E(x,y) [ ∥ y − f(x) ∥2 ] , where (x, y) ∼ D.
We consider the predictor fθ as a random variable depending on the random variable S for training dataset and the random variable O for optimization randomness, where θ = A(S,O) ∈ Rp represents the weights of neural networks produced by the learning algorithm A. p denotes the dimension of θ. The notations and their descriptions are shown in Table 3 in the Appendix.

heading: 2.1 Bias-variance decomposition
text: In the context of classification of C classes, where y ∈ RC is represented as a one-hot vector and fθ(x) ∈ RC denotes an output probability vector by the predictor, the risk R of the learning algorithm can be decomposed into three sources of errors (Geman et al., 1992):
R = Enoise + Ebias + Evar (1) The first term is the irreducible noise and comes from the intrinsic error of data independent of the predictor. The second is a bias term:
Ebias = E(x,y) [ ∥ Eθ[fθ(x)]− E[y|x] ∥2 ] (2)
The third is a variance term
Evar = ExVar(fθ(x)) = Ex [ Eθ [ ∥ fθ(x)− Eθ[fθ(x)] ∥2 ]] (3)
and can be further decomposed into the variance due to optimization Varopt and the variance due to sampling Varsamp (Neal et al., 2018):
Var(fθ(x)) = Varopt + Varsamp = ES [VarO(fθ(x)|S)] + VarS(EO[fθ(x)|S])
(4)
where we denote the expectation of the decomposed variance as EvarO = Ex[Varopt] and EvarS = Ex[Varsamp].

heading: 2.2 Theoretical findings from variance decomposition
text: Assuming the learning task is to learn a linear mapping y = θTx+ϵ where ϵ denotes the noise random
variable with E[ϵ] = 0 and Var(ϵ) = σ2ϵ , and the input vector is x ∈ Rp where p is the input or parameter dimensionality.
In this context, the over-parameterized setting is when p > m and the under-parameterized setting is when p ≤ m.
The theoretical findings in Neal et al. (2018) prove that Evar grows with p in the under-parameterized case, while in the overparameterized case, the variance does not grow with p but scales with the dimension of the data:
ExVar(fθ(x)) = { p mσ 2 ϵ for p ≤ m
r mσ 2 ϵ for p > m
(5)
where r = rank(X) and X ∈ Rm×p denotes the data matrix whose ith row is the training point xTi . Furthermore, EvarO vanishes as p increases under the linear squared regression assumption and EvarS depends on critical parameter dimensionality d(p).

heading: 2.3 Empirical findings from variance decomposition
text: Finding-I: as shown in the left panel of Figure 11, Ebias decreases quickly and levels off once sufficiently over-parameterized, while Evar is unimodal contrary to the classic theory. Finding-II: in the right panel of Figure 1, EvarO is significant and higher than EvarS in the under-parameterized regime. The two variances cross at a certain p once sufficiently over-parameterized. However, empirical p and m of the over-parameterized setting is not strictly following the theoretical findings in section 2.2. Finding-III: in multi-layer models where p is the width and q is the depth, given a fixed p, Ebias decreases while Evar increases as q increases (Yang et al., 2020). These empirical findings hold for multiple datasets in the original papers.
1The two underlying subplots are from Netzer et al. (2011) on the SVHN dataset

heading: 3 Two-Stage Fine-Tuning for Improved Bias and Variance

heading: 3.1 Overview
text: The prevailing fine-tuning methods first build or select an LPM and then fine-tune its parameters on the downstream datasets. The SOTA setting for the LPM and its best hyperparameters for fine-tuning are chosen based on evaluation results, such as precision (P), recall (R), F1 and accuracy scores, using grid-search or random-search. Given a fine-tuning task with a fixed training dataset, there is an upper limit to the learning ability of an LPM which is hard to measure by traditional evaluation methods. For a selected LPM, it is usually hard to decide when to stop searching for hyperparameters. Different from the prevailing fine-tuning setting, we propose a two-stage fine-tuning method. We first provide a variance decomposition-based justification method to roughly measure the generalization ability of a pretrained model w.r.t. the upper limit of its learning ability. In Stage-I, the SOTA setting is built by ratcheting down bias and variance in an iterative way. The searching loop stops until an acceptable performance appears or no more improvement is observed. In Stage-II, given the SOTA setting built in Stage-I, the variance due to optimization is reduced by ensembling techniques. Algorithm 1 outlines the procedure of the proposed two-stage fine-tuning method. The details of each stage are presented below.

heading: 3.2 Stage-I: Justification of generalization ability
text: Based on the preliminaries in Section 2.3, assuming an algorithm A(S,O) is fixed, the Ebias, Evar, EvarO , and EvarS changes as p, q, and m change. Taking the crossing point (EvarO = EvarS ) as a dividing point, we define the generalization ability Gp as:
Gp = {
Phase-I for EvarO > EvarS Phase-II for EvarO ≤ EvarS
(6)
where Phase-I implies large bias and variance leading to large generalization error. Phase-II implies small bias and variance leading to small generalization error which may be good enough w.r.t. the upper limit of the learning ability of A. Justification criteria: After each evaluation, if Gp is in Phase-I, it is necessary to explore more hyperparameter settings or new pretrained models until Gp
is in Phase-II or an acceptable performance (e.g., P, R, F1) is achieved given the limited computing resources available in practice. Then fine-tuning can move to Stage-II. If Gp is in Phase-II, the current setting may be generalizable enough for the given learning task so that the searching can be stopped. Stage-II can be applied but is not necessary. We note that similar to Finding-II in Section 2.3, Gp cannot be determined directly based on p, q, and m. This breakdown provides a high-level guideline for evaluating the generalization of LPMs in an empirical way.

heading: 3.3 Stage-II: Ensembling to reduce variance
text: Ensembles have been proven to be able to improve prediction accuracy and consistency of any learning models in Bonab and Can (2019); Wang et al. (2020). Bagging-based ensembles which are commonly used in various learning tasks have been proven to be able to reduce Evar while keeping Ebias unchanged. However, no theoretical analysis has been discussed in the context of the variance decomposition paradigm. In Stage-II, we focus on bagging-based ensembles to further improve the model performance by reducing EvarO while keeping EvarS unchanged. Applying Stage-II can either move a model from Phase-I to Phase-II though ensembling, i.e., reducing EvarO until EvarO ≤ EvarS ; or further improve a model’s generalization ability from Phase-II by reducing EvarO .
We perform empirical analysis in Section 4 and theoretical analysis in Section 5 to investigate why and how bagging-based ensembles can guarantee such improvements in this context. We also analyze the nature of variance change with the ensemble size in low- and high-resource classes in classification tasks. Boosting ensembles have a more complex behaviour thus are out of scope for this paper.

heading: 4 Experiments

heading: 4.1 Data and models
text: We conduct experiments on the SuperGLUE tasks and two major clinical information extraction datasets. The data processing and statistics and hyperparameter settings are shown in Appendix Table 4 and Table 5. Their brief descriptions are:
• SuperGLUE (Wang et al., 2019) is a benchmark dataset designed for a more rigorous test of general-purpose language understanding
Algorithm 1: Pseudocode of the two-stage fine-tuning method.
Input: S: training dataset; A: optimization algorithm; N : number of ensemble single learners; F : majority voting for ensemble; Output: ζ: ensemble learner /* Stage-I */ 1 Gp ← Phase-I 2 E∗ ← 0 3 while (Gp = Phase-I) or (E∗ is not acceptable) do 4 Choose a pretrained model f and an initialization seed O. 5 θ ← A(S,O, f) 6 Compute EvarO and EvarS by Equation 4. 7 Gp ← Phase-I if EvarO > EvarS otherwise Phase-II 8 E∗ ← score(fθ) // P,R,or F1. 9 ζ := f /* Stage-II */
10 if (Gp ̸= Phase-II) or (E∗ is not satisfied) then 11 ξ ← ∅ // The set of ensemble components. 12 N is set to be ≥ EvarO/EvarS 13 while len(ξ) < N do 14 Resample training and validation sets Si from S. 15 Train f∗ on Si using snapshot learning and
save N trained learners f∗θ1 , . . . , f ∗ θN
where θi = A(Si, O∗, f∗). 16 Select l trained learners from the saved ones
by applying pruning algorithm (Wang et al., 2020).
17 ξ ∪ f∗θ1 , . . . , f∗θl 18 ζ := F(ξ)
systems after GLUE (Wang et al., 2018). We replicate the scores on dev set2, and select six tasks (BoolQ, CB, RTE, MultiRC, WiC, and COPA). The selected tasks cover question answering (QA), natural language inference, and word sense disambiguation. The SOTA setting is based on the setting in Liu et al. (2019) using roberta-large as the pretrained model.
• THYME (Temporal Histories of Your Medical Events) corpus (Styler IV et al., 2014) for temporal relation extraction, consisting of 594 de-identified clinical and pathology notes on colon cancer patients. We use the THYME+ version of the corpus (Wright-Bettner et al., 2020). There are 10 classes of extremely imbalanced class distribution. The SOTA setting is based on the setting in Lin et al. (2021) using PubmedBERTbase-MimicBig-EntityBERT as the pretrained model.
2https://github.com/pytorch/fairseq/ tree/master/examples/roberta
• 2012 i2b2 Temporal Relations (Sun et al., 2013) consists of 394 training de-identified reports, 477 test de-identified reports, and 877 unannotated reports. There are 3 classes of slightly imbalanced class distribution. The SOTA setting is based on the setting in Haq et al. (2021) using BioBERT-base-cased-v1.1 as the pretrained model.

heading: 4.2 Metrics and settings
text: We use an NVIDIA Titan RTX GPU cluster of 7 nodes for fine-tuning experiments through HuggingFace’s Transformer API (Wolf et al., 2020) version 4.13.0. We leverage the run_glue.py pytorch version as our fine-tuning script. Unless specified, default settings are used in our experiments. Due to differences in the fine-tuning script and some missing settings not provided by the original authors, we were unable to reproduce the exact SOTA scores but we achieved scores close to the SOTA ones. Our implementation are denoted as replicated-SOTA (RSOTA). We compare our implementation and reference SOTA scores in Appendix Table 7. We use the RSOTA settings as the starting point to conduct the experiments. We use Ebias, Evar, EvarO , and EvarS for Stage-I, and adopt classic evaluation metrics (P, R, and F1) for Stage-I and Stage-II. For the purposes of consistency, we report P, R, and F1 of SuperGLUE tasks for consistency with the reported results for the THYME and i2b2 tasks in the experiment results. Accuracy scores are reported in Appendix Table 7.

heading: 4.3 Experimental design
text: There are two stages in the proposed method. For Stage-I, we use 5 random seeds for the randomness over data samples S and 5 for the randomness over initialization O, resulting in a total of 25 fine-tuned models. The averages over data samples are performed by taking the training set S and creating 5 bootstrap replicate training/validation splits with the same class distribution. The bias expectation in Equation 2 is estimated as the averages over both S and O. The variance decomposition is estimated based on Equation 4. More specifically, EvarO is estimated as the averages over S of the variance over O, and EvarS is estimated as the variance over S of the averages over O. Furthermore, we also apply RoBERTa-base-uncased (RBU) as the pretrained model for each fine-tuning task using the RSOTA setting except for pretrained models. Their
descriptions are shown in Appendix Table 6. To replicate SOTA scores and obtain RSOTA settings for each task, we conduct hyperparameter searching in an iterative way. This process is considered as the experiment of Stage-I.
For Stage-II, any bagging-based ensemble algorithms are feasible. In our preliminary experiments (Wang et al., 2022), we have shown that the dynamic snapshot ensemble algorithm (Wang et al., 2020), which we call ENS in this paper, works better than vanilla bagging ensembles. ENS is a bagging-based ensemble explicitly designed to reduce variance over optimization-related hyperparameters in one framework, with the aim of building computationally efficient strategies to boost model performance on top of any given setting with a guarantee (i.e., simple bagging ensemble cannot guarantee an improvement). In our implementation, we employ ENS. The ensemble size is 5 and majority voting is used to generate ensemble predictions. To explore the ensemble impact on low- and high-resource classes, we compute and compare performance improvements of each class from the extremely imbalanced THYME dataset. To investigate the impact of ensemble size on improving the model performance of imbalanced classes, we also evaluate performance of individual classes using ENS of size 1 to 10. We compute 95% confidence intervals for these estimates using bootstrapping over 5 samples. More details are in Appendix B.

heading: 4.4 Justification results
text: Table 1 shows Ebias, Evar, EvarO , and EvarS computed on the datasets with different pretrained models. It is noted that in our experiment, we are not applying the algorithm for Stage-I to ratchet down bias and variance in an iterative manner. The goal of this table is to analyze both RBU and RSOTA models for the bias and variance trends discussed in Section 2. Interestingly, we observe that EvarO > EvarS for all datasets and models except for CB-RBU and COPA-RBU where models are not well trained given that F1 score is around 0.5 indicating random guess. This implies that the vast majority of the SOTA models we experimented with are in Phase-I (i.e., not generalizable enough for their tasks), which is contrary to our intuition that these transformer-based models are complex enough given the moderate sized labeled datasets. It is also observable that Ebias is much larger than Evar indicating that the model performance is dominated more by bias than by variance. For SuperGLUE tasks, with the same hyperparameter setting (i.e., A(S,O)), the RSOTA models (i.e., larger p and q than RBU models) achieve smaller Ebias but larger Evar than RBU models except for MultiRC. The change of Evar mainly comes from the change of EvarO . As Finding-I in Section 2.3 that Ebias decreases while Evar is unimodal, our observation implies that RBU models are before peak and long way toward Phase-II while RSOTA models get closer to Phase-II than RBU models. The exception is the result on the MultiRC dataset which is QA corpus listing a passage which consists of text, a question, and multiple answers to that question each with a true/false label. Although MultiRC represents a variety of question types (e.g. yes/no, factual, etc.), the passages are not annotated for question type. As explained in Appendix section B.1., we represented the QA pairs within each passage as text, question, answer, label instances and sampled from these instances. Using this instance-based sampling likely leads to samples not stratified by question types, therefore not necessarily representative. This probably explains the better mean F1 for MultiRC-RBU as compared to the mean F1 for MultiRC-RSOTA in Table 1 (different samples are created for each run). However, when we drill down to the best model F1 for MultiRC-RBU and MultiRC-RSOTA, the results are 78.9 and 85.4 F1-scores respectively, which supports the trend in Table 1. For the SuperGLUE tasks, the bias and variance of RBU models and RSOTA models are shown together to illustrate a trend (like Fig. 1) as p and q are the only variables. However for THYME and i2b2 tasks, similar trends could not be interpreted since the RSOTA models are pretrained with domain specific corpora while the RBU models are pretrained with general corpora. This implies that for fine-tuning tasks such as temporal relation extraction, other factors (e.g., domain corpora used to pretrain models) may have larger impact than the model complexity. Our observations are consistent with Finding-II that empirical p and m setting is not strictly following theoretical findings which are under linear squared regression assumption. This also indicates that p, q, and m cannot be used to measure Gp empirically. On the other hand,
our variance decomposition-based method does not rely on p, q, and m, therefore it provides the basis for a more generalized justification method.

heading: 4.5 Ensemble results
text: Table 2 presents P, R, and F1 scores of RSOTA and ENS methods on all datasets. Similar to prior studies (e.g. Zhang et al., 2020; Du et al., 2022), results for the SuperGLUE tasks are reported on the dev set. The accuracy scores for each task are presented in the Appendix Table 7. Compared to the RSOTA setting, the ENS method boosts performance on all datasets, with the largest gains of 49.3% and 15.2% relative F1 improvements on the low-resource CB and COPA datasets respectively. In Section 5, we analyze why ensembles work from the variance decomposition perspective, which provides insights into how ensembles help reduce EvarO and lead to better prediction accuracy.

heading: 4.6 Ensemble impact on low- and high-resource classes
text: We further investigate the improvements on lowresource datasets (e.g. CB and COPA). To eliminate all interference from p, q, A(S,O), pretrained models and only keep m as the variable, we tease apart the results of the extremely unbalanced THYME dataset and analyze the performance on each class. Its most frequent classes (i.e., high-resource classes) are CONTAINS (2895), OVERLAP (2359), and BEFORE (2241); and the least frequent classes (i.e., lowresource classes) are NOTED-ON (140), BEGINSON (160), and ENDS-ON (244). The initial F1 scores are: CONTAINS-0.776, OVERLAP0.539, and BEFORE-0.469; NOTED-ON-0.618, BEGINS-ON-0.608, and ENDS-ON-0.695. In Figure 2, we show absolute improvement and improvement percentage of F1 with various ensemble size N (compared with single learners i.e., N = 1). These values are computed based on the mean with 95% confidence interval over 5 random samples for each class and each ensemble size. It is observable that given a fixed N , the performance improvements by F1 scores on the low-resource classes – NOTED-ON (brown), BEGINS-ON (red), and ENDS-ON (orange) – are larger than the ones of high-resource classes. The difference becomes larger as N increases. The scales of improvement are not affected by the initial results; i.e., the larger improvements on low-resource classes are not due to lower initial F1 scores. This is an interesting ob-
servation and may introduce a new solution for improving performance of imbalanced datasets. More similar results on P and R are shown in in Appendix Fig. 3. We explore theoretical insights into these observations in Section 5.

heading: 5 Discussion and Theoretical Analysis

heading: 5.1 Basic statistics
text: Let X1, X2, · · · , XN be a random sample from a population with mean µ and variance σ2 and X̄ = 1N ∑N i=1Xi. Then the following two items hold. a: E[X̄] = E[Xi] = µ b: Var(X̄) = 1N Var(Xi) = 1 N σ 2

heading: 5.2 Ensemble in bias-variance decomposition
text: We work in the context of bagging-based ensembles, assuming the ensemble predictor f̄(x) is f̄(x) = 1N ∑N i=1 fi(x) is the averaging of N single learners trained with different samples of S and O. Based on the basic statistics, the Ebias of f̄(x) in Equation 2 is unchanged while the Evar of f̄(x) in Equation 3 decreases by 1N . Furthermore, we have:
EvarO = Ex [ ES [VarO(f̄θ(x)|S)] ]
= 1
N Ex [ES [VarO(fθ(x)|S)]]
(7)
and:
EvarS = Ex [ VarS(EO[f̄θ(x)|S]) ]
= Ex [VarS(EO[fθ(x)|S])] (8)
which indicates that EvarO reduces while EvarS keeps unchanged as the ensemble size N increases. EvarO vanishes when N is sufficiently large. The improvement of the variance by ensembling comes from the reduction of the variance due to optimization.
As mentioned in Section 2.2 that under linear squared regression assumption, EvarO vanishes as p increases and EvarS depends on critical parameter dimensionality d(p). In this paper, we also proved that EvarO vanishes as N increases. Given that pretraining LPMs with larger p and/or q is extremely difficult, increasing N is a much better way for improving performance of LPMs. This also proves the effectiveness of Stage-II in our proposed two-stage fine-tuning method. To ensure that a fine-tuned LPM can move from Phase-I to PhaseII, the ensemble size N in Stage-II should be set to a value that is larger or equal to EvarOEvarS .

heading: 5.3 Ensemble in low- and high-resource classes
text: One interesting experimental observation is that the improvement on low-resource classes is larger than that on high-resource classes. To further investigate the impact of the ensemble learners on the imbalanced datasets, we make the following analysis to Equation 5.
ExVar(f̄θ(x)) = { 1 N · p mσ 2 ϵ for p ≤ m
1 N · rmσ2ϵ for p > m
(9) where the impact of ensembles is represented as a function of ensemble size N , denoted as π(N) = 1 N ∈ [0, 1] that a smaller π(N) means a larger performance improvement. Given a fixed p, in overparameterized cases (p > m) where m is small, since the samples in S are i.i.d, thus X ∈ Rm×p is full rank so that r = rank(X) = m. The variance becomes 1N · σ2ϵ which does not change with m. The impact of ensemble solely depends on N thus is significant. On the other hand, in underparameterized cases (p ≤ m) where m is large, the variance is negligible as m becomes much larger than p, i.e., lim
m→∞ 1 N · p mσ 2 ϵ = 0, so that the vari-
ance becomes 0 regardless of π(N). This implies that the impact of ensemble can be ignored as m increases. In general, given a fixed p, for both cases the impact of ensemble is significant when m is small and insignificant as m becomes very large.
These theoretical findings explain why we observe larger performance improvement on lowresource than on high-resource classes using ensembles. Similar to the discussion in Section 4.4, empirically, it is hard to define low-resource and high-resource classes using m and p because our analysis is based on least squared linear regression
assumption which is simplified compared to conditions in real scenarios. Besides p and m, there are other factors that may have implicit but significant impact on model performance. This also explains why the improvement does not strictly follow the sorting of classes by their sample size. However, our findings show another advantage of using ensembles. The empirical impact of ensemble size on imbalanced classes has been examined and shown in Section 4.6 and Appendix C, which is consistent with the theoretical findings discussed in this section.

heading: 6 Related Works
text: In a fine-tuning setting, searching for an optimal setting of pretrained models and hyperparameters is challenging due to the high dimensionality of the search space, as well as the infinite values for each dimension. In previous works of fine-tuning tasks (Lee et al., 2020; Alsentzer et al., 2019; Beltagy et al., 2019; Lin et al., 2021), the SOTA models are single learners carefully selected and fine-tuned based on evaluation results, such as P, R, and F1 scores, using grid-search or random-search. To improve the stability of the pre-trained transformerbased language models, Mosbach et al. (2021) suggests using small learning rates with bias correction and increasing the number of iterations to avoid vanishing gradients. Prior efforts also highlight the comparable effect of weight initialization and training data order to the variance of model performance (Dodge et al., 2020).
Ensemble methods have been successful in boosting the predictive performance of single learners (Ren et al., 2016 present a comprehensive review; also see Wang et al., 2003; CireşAn et al., 2012; Xie et al., 2013; Huang et al., 2017) as well
as in estimating predictive uncertainty (Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017; Snoek et al., 2019). Among these studies, Bonab and Can (2019) and Wang et al. (2020) theoretically prove that ensembles can perform better than the average performance of component learners for prediction accuracy and consistency of learning models. Wang et al. (2022) empirically evaluates the application of ensemble methods to fine-tuned transformer-based models for clinical NLP tasks. The findings demonstrate that ensemble methods improve model performance, particularly when employing dynamic snapshot ensembling. Although it is common knowledge that ensembles can reduce variance thus reducing the generalization error, no previous work has discussed or measured this in the context of variance decomposition. Furthermore, no previous work has investigated the impact of ensembles on imbalanced datasets.

heading: 7 Conclusion
text: Different from the prevailing fine-tuning settings, we propose a two-stage fine-tuning method to improve the generalization ability of a fine-tuned LPM. We provide a variance decomposition-based justification method to empirically measure the generalization ability of the LPM w.r.t. the upper limit of its learning ability. In Stage-I, the RSOTA setting is built by ratcheting down bias and variance in an iterative way. In Stage-II, given the RSOTA setting, the fine-tuned LPM is guaranteed to be further generalized through ensembling techniques by reducing the variance due to optimization. The proposed justification method provides a concrete metric to track this process.
We provide empirical evidence by conducting experiments on the SuperGLUE tasks and two clinical datasets. Furthermore, we perform theoretical analysis on how ensembles improve variance due to optimization. We investigate the nature of variance change for the ensemble size in lowand high-resource classes in classification tasks. Different from previous theoretical analyses using only model complexity and data size which depends on least squared regression, our variance decomposition-based justification method in StageI does not rely on specific factors thus leading to a more generalizable measurement. The ENS further boosts performance without risk of computational cost and overfitting. Our analysis on imbalanced data reveals another advantage of ensemble algo-
rithms in improving model performance on lowresource classes.
As future work, we are interested in (1) rigorously proving variance decomposition-based justification criteria, (2) quantifying low- and highresource classes with specific features that interplay with ensemble size. If properly used, we believe the theoretical and empirical findings discussed in this paper can guide practitioners to fine-tune more generalizable models.
Limitations
As we stated under future work, one of the limitations is the variance decomposition-based proof. Our work is based on simplified settings, i.e., linear squared regression assumption. Post-ensemble variance is not evaluated due to the nature of the ENS ensemble algorithm. Extended experiments using vanilla bagging ensemble would enable analysis of post-ensemble variance. Further investigation into refining the two stages would help understand the performance of LPMs, e.g. those that are in Phase-I but before the peak in Figure 1. Our results for MultiRC are based on the instance sampling, however a better sampling technique should be based on stratified sampling based on the ratio of the question types in the MultiRC set. However, to achieve this, the MultiRC set needs to be annotated for question types, which is currently missing. Sampling techniques by themselves can become a research topic so that a further decrease of variance due to sampling can be achieved. Although we list these items as limitations, they are also topics for future research within the greater theme of understanding the new bias-prevalence paradigm for LPMs.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for feedback that improved the paper, the US National Institutes of Health (NIH) and the New Jersey Institute of Technology (NJIT) for providing funding. This research is supported by NJIT FY24 Faculty Seed Grant, NIH Grants U24CA248010, R01LM013486 and R01GM114355. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NJIT or NIH.

heading: A Notations
text: Table 3 shows major notations and their descriptions.

heading: B Experiments
text: B.1 Data description Table 4 shows the statistics of all datasets used in our experiments. Each downloaded SuperGLUE dataset includes train, val, and test sets in json format.2 The downloaded test set does not have goldstandard labels thus is not used in our experiment.
2https://super.gluebenchmark.com/ tasks
We split the train set into train (80%) and dev (20%) sets, and evaluate the model performance on val set. The i2b2 does not have a development (dev) set in the released data and we split the train set into train (80%) and dev (20%) sets. Random seed 42 is used to replicate the sampling process. For MultiRC, because each question can have more than one correct answer, we sampled the instances based on individual question-answer options in the train set for training and validation in our experiment.
B.2 Hyperparameter settings
Table 5 shows the details of hyperparameter settings. Unless otherwise specified, we use default values of the hyperparameters in Huggingface. We also summarize pretrained models used in our experiments in Table 6.
B.3 Replicated SOTA scores
To ensure that our experiments on the SuperGLUE tasks are reproducible, we followed the settings and replicated the SOTA accuracy scores reported in: https://github.com/ facebookresearch/fairseq/tree/ main/examples/roberta. We could not replicate the representation (special token extractions) and the model settings (unpublished pretrained model) for WSC task, thus it is omitted in our paper. In our experiments, we report the classic metrics of precision/recall/F1 for consistency with the reported results for the THYME and i2b2 tasks. Our accuracy scores for the SuperGlue tasks (shown in Table 7) are directly comparable and are consistent with those in Table 2 in the main paper.
B.4 Implementation details of ENS
ENS allows a pretrained model to be fine-tuned multiple times (i.e., multiple training runs) sequentially with different random seeds and data shuffling of train/validation splits. It uses a cyclic annealing schedule and cyclic snapshot strategy to periodically save the best model during each training run. Different from the simple bagging ensemble, after each training run, a dynamic pruning algorithm is applied to select a few single learners from the saved ones which can lead to better performance of the ensemble learner with theoretical guarantees. The sequential training runs stop when the accumulated number of selected single learners reaches a preset ensemble size. The total amount of training runs is a dynamic value rather than a preset value, which is determined by the snapshot strategy and pruning factor during the sequential training.
In our experiments, we implemented ENS on the top of RSOTA setting. The ensemble size is set as 5 and majority voting is used to generate ensemble predictions. We reuse RSOTA settings except that we set cosine with restarts as the learning rate scheduler and set the learning rate to restart every k epochs which, based on the RSOTA setting, allows the model to converge to a reasonable state before each restart. The total number of epochs for each training run is 5 × k and we save the top 4 models for pruning based on validation accuracy. The random seeds for initialization and data shuffling are [42, 52, 62, 72, 82]. The logic behind the above settings is to retain the benefits from RSOTA fine-tuning settings as much as possible. Code and settings to reproduce the results are avail-
able at https://github.com/christa60/ bias-var-fine-tuning.git.
B.5 Experimental design of bagging ensemble for investigating various ensemble sizes
To analyze the nature of the variance change with the ensemble size in low-resource classes (NOTED-ON, BEGINS-ON, END-ON relations in the THYME corpus) and high-resource (CONTAINS, OVERLAP, BEFORE relations in the THYME corpus) classes, we vary the ensemble size from 1 to 10 and then compute the P, R, and F1 scores for each class on THYME data.
We create 10 bootstrap replicate training sets by resampling training and dev datasets with the same size and class distribution. The random seeds for resampling are randomly chosen and then fixed. The various splits are denoted as [’split_r42’, ’split_r52’, ’split_r62’, ’split_r72’, ’split_r82’, ’split_r92’, ’split_r102’, ’split_r112’, ’split_r122’, ’split_r132’]. Given a random seed of initialization, we train N fine-tuned single learners. To compute 95% confidence intervals for these estimates, we use 5 random seeds of initialization, resulting in 5 ensemble models for each ensemble size. We vary the ensemble size N from 1 to 10 and have 100 ensemble models in total.

heading: C Section 4.6: Additional Results
text: We show the absolute and percentage improvement (compared with single learners i.e., N = 1) change over the ensemble size N using P and R in Figure 3. Together with Figure 2, the major observations are: (a) The absolute and percentage improvements of P, R, and F1 increase as N increases. (b) The precision improvements are more
pronounced than those of recall thus contributing the major part of the F1 improvements. This phenomenon is more pronounced for high-resource classes. (c) Given a fixed N , the improvements on low-resource classes are larger than those on high-resource classes across the three metrics. The difference becomes larger as N increases.
Discussion: Our experimental results are consistent with our theoretical findings in Section 5 that model performance keeps improving because variance due to optimization decreases as ensemble size increases. Furthermore, the impact of ensemble is more pronounced on low-resource classes than on high-resource classes.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.4 and 4.5
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: As future work, we are interested in (1) rigorously proving variance decomposition-based justification criteria, (2) quantifying low- and highresource classes with specific features that interplay with ensemble size. If properly used, we believe the theoretical and empirical findings discussed in this paper can guide practitioners to fine-tune more generalizable models. Limitations
As we stated under future work, one of the limitations is the variance decomposition-based proof. Our work is based on simplified settings, i. e. , linear squared regression assumption. Post-ensemble variance is not evaluated due to the nature of the ENS ensemble algorithm. Extended experiments using vanilla bagging ensemble would enable analysis of post-ensemble variance. Further investigation into refining the two stages would help understand the performance of LPMs, e. g. those that are in Phase-I but before the peak in Figure 1. Our results for MultiRC are based on the instance sampling, however a better sampling technique should be based on stratified sampling based on the ratio of the question types in the MultiRC set. However, to achieve this, the MultiRC set needs to be annotated for question types, which is currently missing. Sampling techniques by themselves can become a research topic so that a further decrease of variance due to sampling can be achieved. Although we list these items as limitations, they are also topics for future research within the greater theme of understanding the new bias-prevalence paradigm for LPMs