Bi-level Finetuning with Task-dependent Similarity Structure for Low-resource Training
Year: 2023

Abstract:
Training a large language model in lowresource settings is challenging since they are susceptible to overfitting with limited generalization abilities. Previous work addresses this issue by approaches such as tunable parameters reduction or data augmentation. However, they either limit the trained models’ expressiveness or rely on task-independent knowledge. In this paper, we propose the Bilevel Finetuning with Task-dependent Similarity Structure framework where all parameters, including the embeddings for unseen tokens, are finetuned with task-dependent information from the training data only. In this framework, a task-dependent similarity structure is learned in a data-driven fashion, which in turn is used to compose soft embeddings from conventional embeddings to be used in training to update all parameters. In order to learn the similarity structure and model parameters, we propose a bi-level optimization algorithm with two stages—search and finetune— to ensure successful learning. Results of experiments on several classification datasets in low-resource scenarios demonstrate that models trained with our method outperform strong baselines. Ablation experiments further support the effectiveness of different components in our framework. Code is available at https://github.com/Sai-Ashish/BFTSS.

heading: 1 Introduction
text: Finetuning pretrained large models in low-resource scenarios 1 face many challenges (Kann et al., 2020; Hedderich et al., 2021; Şahin, 2022). One of the challenges is the overfitting of the model when finetuned on the small training data. Different approaches have been proposed to tackle this problem and achieved great results. Some approaches restrict the number of parameters to be updated in
*Equal Contribution 1Our work is conducted in low resource scenarios that
comprise a few hundred instances of data.
the finetuning process to avoid overfitting to small amounts of data (Xu et al., 2021). However, parameter restriction while model finetuning may impact the model’s expressiveness.
Other methods such as data augmentation (Wei and Zou, 2019; Hoang et al., 2018) aim at increasing the training data size through synthesizing new training data examples to boost generalization ability. These methods rely on either external lexical resources and heuristics, which are limited in domain and language; or pretrained language models, the semantic similarity space of which is not taskdependent. For example, Apple may be replaced by Microsoft in synonym replacement based on some lexical resource, but the “replace-ability” really depends on whether the task is “separate tech companies from gas companies” or “find companies with their headquarters in Washington state”, the information of which pretrained language models or static lexical resources are not able to provide.
Our motivation is to combine strengths of both approaches. For generalization ability to unseen data, all parameters, especially embeddings of words not in the training data, should participate in the finetuning. At the same time, no external knowledge source should be required for such finetuning to ensure the method being scalable to different tasks. The ideal training framework for these goals should allow training signals to flow from tokens in training data to unseen tokens in a task-dependent way. Such a framework will ensure that the generalization ability of the trained model is strengthened through finetuning without the risk of overfitting quickly to a small amount of training data.
Our approach proposed in this paper, Bi-level Finetuning with Task-dependent Similarity Structure (BFTSS), aims to meet these goals. First, we propose a low-resource finetuning method where all parameters of the model, including the embeddings of unseen tokens, can be tuned directly through soft embeddings. The soft embeddings
8569
are constructed through the use of a similarity matrix with pairwise similarity scores between words, termed a similarity structure2 in this paper. Second, we propose a bi-level optimization algorithm to learn a task-dependent similarity structure with lowresource task data, where no extra data, knowledge source or task-dependent prior knowledge is required. Since the similarity structure is usually very large, two different methods are proposed to reduce its size and make the learning tractable. Finally, we conduct extensive experiments on different datasets and with different model sizes. Comparison to baseline models and ablated models shows the effectiveness of our approach, where the performance of the models trained in the proposed method surpasses all baselines by large margins.

heading: 2 Related work
text: Low-resource training has been a challenging but important task in natural language processing (Hedderich et al., 2021). Approaches have been proposed to tackle the issues encountered in lowresource training. Robust finetuning methods are applied in such training scenarios, such as approaches restricting tunable parameters (Houlsby et al., 2019; Lee et al., 2019; Chen et al., 2020; Xu et al., 2021) and noise-robust training methods (Jia et al., 2019; Onoe and Durrett, 2019; Jin et al., 2021). They alleviate the overfitting problem but introduce no new information beyond the training data into the model. Data augmentation on the token level (Wei and Zou, 2019; Raiman and Miller, 2017; Vania et al., 2019) as well as on the sentence level using syntax (Şahin and Steedman, 2018; Şahin, 2022), back-translation (Hoang et al., 2018; Xie et al., 2020) or generation models (Ding et al., 2020; Lowell et al., 2021; Liu et al., 2022; Zhou et al., 2022; Wang et al., 2022; Somayajula et al., 2022) are approaches which aims at introducing extra information into model training. Other similar methods rely on pseudo-labeling extra data (Mintz et al., 2009; Le and Titov, 2019; Lison et al., 2020) using task insights and heuristics. Most of them are designed for a specific task, or require external knowledge sources.
Bi-level optimization (BLO) has wide applications in machine learning. The neural architecture
2By using “similarity structure” instead of “similarity matrix”, we would like to emphasize that 1) the similarity scores are interrelated among words with underlying task-dependent structures, 2) a matrix is just one way of expressing the relations among words.
search proposed by Liu et al. (2018) uses BLO. It is also used in data selection (Shu et al., 2019; Wang et al., 2020; Ren et al., 2020) and meta-learning (Finn et al., 2017). Feurer et al. (2015) proposed a BLO-based optimization framework for hyperparameter tuning. Baydin et al. (2017) proposed BLO-based learning rate adaptation. Noisy label correction using BLO is proposed by Baydin et al. (2017). Among the papers mentioned above, the lower parameters are the model weights, and the upper parameters are the meta-variables, such as hyperparameters, architecture, training example weights, etc., to be optimized using BLO.

heading: 3 Bi-level Finetuning with Task-dependent Similarity Structure
text: The Bi-level Finetuning with Task-dependent Similarity Structure framework, as shown in Figure 1, centers around how to learn and utilize a taskdependent similarity structure S. The structure is first initialized and learned along with model parameters with bi-level optimization on task data, where soft embeddings of words are derived from the structure to propagate training signals into unseen words. After this first phase of training, named Search phase, the Finetune phase follows in which only model parameters are updated with the similarity structure fixed.

heading: 3.1 Motivation and Overview of BFTSS
text: In BFTSS, a task-dependent similarity structure is first learned with a small training data and then used to improve the performance of the language models. The motivation for the task-dependent similarity structure comes from the observation that only a few words appear in the training data in a low-resource scenario with their word embeddings updated in training. However, we want to pass more information about the unseen words to the model to train it. One way to do that is to identify a word’s similar words in the vocabulary and estimate gradients of them from the seen word’s gradient. This similarity structure is encoded as a similarity matrix S, with each row corresponding to a word3 in the vocabulary with size V . The row entries represent the task-dependent semantic proximity of a word with all other words in the vocabulary. Previous methods for data augmentation implicitly
3The smallest string units for a pretrained language model may be words, subwords or other symbols. We use word to refer to all items in the tokenization vocabulary.
assume that the similarity structure for a task is identical to a general similarity structure, which can be derived from pretrained models or gathered from external lexical semantic resources. However, similarity structures are task-specific with varying degrees of closeness to the general similarity structure, as shown in the Apple-Microsoft example. In our framework, they are trained with task data to ensure they are task-specific.
With a task-specific similarity structure, we are able to train all parameters through soft embeddings. Soft embeddings are defined as a linear combination of the embedding vectors weighted by the entries of the S matrix. Intuitively, it means that when a model with parameters W sees a word in the text input, it also sees all the related words weighted by the entries of the corresponding row in S. Thus the optimal model weights learned this way would be dependent on S, i.e., W ∗(S).
For training, the task-dependent similarity matrix S should not be learned by reducing the training loss on the model parameters W ∗(S), because this is similar to adding more parameters to an already huge language model. Therefore, the taskdependent similarity matrix S is learned using a bilevel optimization approach. Bi-level optimization is used because of the inter-dependency between the optimal model weights W ∗ and the optimal similarity matrix S∗. The learned optimal model weights W ∗(S) depend on S and S is learned in a way that further improves the performance of W ∗(S). This shows that both parameters influence and benefit from each other. With bi-level optimization, we are able to first estimate the W parameters by one-step gradient descent with S fixed on one portion of the training data, and learn the S parameter by using the learned W parameter on a different portion. This bi-level optimization phase of W and S is called the Search Phase.
Finally, with the learned S and W from the Search Phase, normal finetuning is conducted in
the Finetuning Phase for further tuning the W parameters on the entire training data. The learned S parameters is fixed throughout the phase.

heading: 3.2 Similarity structure initialization
text: The similarity structure is encoded in this work as a similarity matrix S ∈ RV×V , where V is the vocabulary size. Each row of the S matrix represents a word in the vocabulary. The row entries represent a word’s semantic proximity with all the other words in the vocabulary. One way to initialize this S matrix is to add the inner product of the pretrained language model’s embedding matrix E ∈ RH×V with itself (where H is the hidden dimension of the embedding layer of the language model) to the identity matrix:
S = αI + (1− α)f({ÊT Ê}d),
where Ê is the normalized E matrix where each column vector is a unit norm vector, d is the inverse temperature, I ∈ RV×V is the identity matrix, α is trade-off parameter, and f is the normalizing function that normalizes each row to sum to 1. The identity matrix is added to make the initialization a spiky distribution with the highest weight for the diagonal elements. The pretrained model’s embedding layer decides the weights of the off-diagonal elements. These language models are pretrained on a huge corpus of texts and the cosine distance between the embedding vectors depict the semantic proximity between the words. The inner-product matrix is raised to the power of inverse temperature and normalized to make it a smooth and light-tailed distribution. α controls how strong the pretrained embeddings influence the similarity values. By setting α to 1, we have Vanilla finetuning where the similarity terms are not considered. In this paper, we set both terms to have equal weights and omit it in the following sections.

heading: 3.3 Soft Embeddings
text: Soft embeddings are defined as the linear combination of all the related embedding vectors whose weights are determined by the similarity structure S. Formally, we define the soft embedding vector of a word as follows,
ê (t) i =
K∑
j=0
s (t) i,jE (t) j = e (t) i S (t){E(t)}T
where K is the number of related words used to calculate soft embeddings, E(t) ∈ RH×V and S(t) ∈ RV×V are the embedding matrix and similarity matrix at tth iteration. E(t)j is the embedding vector of jth word. s(t)i,j is the {i, j}th element of the S(t) matrix which describes how similar the j-th word is to the i-th word. e(t)i is the one-hot representation of the i-th word and ê(t)i is the soft embedding of it.
When the model weights are updated with backpropagation, the embeddings of all the similar words (determined by the entries of the similarity matrix S) are updated, propagating task knowledge into all parts of the model:
∇ei = K∑
j=0
s (t) ij ∇E (t) j .

heading: 3.4 Bi-level learning of a task-dependent similarity structure
text: Because the similarity structure needs to be trained with task data, a bi-level optimization-based approach is proposed in this work for learning such a task-dependent similarity structure. There are two stages in the bi-level learning process. In the first stage, the model weightsW is updated to minimize the loss on one dataset, searching for the optimal model weights W ∗(S) on that dataset. In the second stage, the task-dependent similarity matrix S is updated searching for S∗ that attains the minimum loss on a different dataset.4
4The two datasets used in the bi-level optimization are usually the training set and the validation set. However, because we are operating in low-resource scenarios, we split the small training dataset Dtrain into two halves to act as these two datasets. The first half DB-train is used to train the model in the first stage. The second half DB-val is used to train the optimal S parameter in the second stage. This procedure keeps the training data size the same among all baselines for fair comparison.

heading: 3.4.1 Training W
text: In the first stage, model parameters W are trained on BFTSS training set DB-train with the similarity matrix S fixed:
W ∗(S) = min W L(W,S,DB-train), (1)
where W is model parameters, S is the similarity matrix, and L is the task loss. The optimal model weights are learned on DB-train given a similarity matrix S. Hence we learn W ∗(S), which is dependent on S since W ∗ depends on the loss function L(·) which is a function of S. S is not updated in this stage because this would overfit the BFTSS training set; instead it will be updated in the second stage.

heading: 3.4.2 Training S
text: In the second stage, the optimal similarity matrix S is learned on BFTSS validation set DB-val given the optimal model weights W ∗(S) learned in the first stage on DB-train. The model trained in the first stage W ∗(S) is evaluated on DB-val and S is updated by minimizing the validation loss. The following optimization problem is solved at this stage:
min S
L(W ∗(S), S,DB-val). (2)
By performing both the stages iteratively with different parameters being fixed at each stage, we do not overfit on the any of the two dataset DB-train and DB-val.

heading: 3.4.3 A bi-level optimization framework
text: Combining both the stages, we have the following bi-level optimization framework:
minS L(W ∗(S), S,DB-val) s.t. W ∗(S) = minW L(W,S,DB-train) (3) The algorithm consists of two learning stages. From the bottom, the optimization problem corresponds to the learning stage 3.4.1 and then 3.4.2. These two stages are conducted end-to-end. The solution obtained in the first stageW ∗(S) is a function of S. We solve for S by minimizing the validation loss in the second stage. The S learned in the second stage changes the training loss in the first stage, which changes the solution W ∗(S).

heading: 3.5 The full optimization algorithm
text: The full optimization algorithm, shown in Algo.1 includes two distinct phases. 1) Search phase: In this phase, optimal similarity matrix S∗ is estimated by an iterative algorithm to solve the Bi-level optimization problem in Equation 3. The algorithm learns S′ ≈ S∗. 2) Finetune Phase: In this phase, we finetune the language model on the wholeDtrain for optimal model weights W ∗ with a fixed S′.
For optimization in the search phase, we develop a gradient-based optimization algorithm to solve the problem defined in Equation 3 (Liu et al., 2018). W is approximated using the one-step gradient descent.
W ∗(S) ≈W ′ =W − ηw∇WL(W,S,DB-train) (4)
W ′ is plugged into the second-level objective function. The gradient with respect to the S matrix is calculated to update S:
S∗ ≈ S′ = S − ηs∇SL(W ′, S,DB-val). (5)
Gradient of the loss function with respect to S is calculated using the chain rule. W ′ is an implicit function of S.
∇SL(W ′, S,DB-val) = ∇SL(W − ηw∇WL(W,S,DB-train), S,DB-val) = ∇SL(W ′, S,DB-val)− ηw× ∇2S,WL(W,S,DB-train)∇W ′L(W ′, S,DB-val)
Solving for S′ involves an expensive matrix-vector product, whose computational complexity can be reduced by a finite difference approximation:
∇2S,WL(W,S,DB-train)∇W ′L(W ′, S,DB-val) = ∇SL(W+, S,DB-train)−∇SL(W−, S,DB-train)
2 ,
(6)
where
W± =W ± ∇W ′L(W ′, S,DB-val),
= 0.01
‖∇W ′L(W ′, S,DB-val)‖2 .
This procedure is carried out iteratively until convergence, at which time the Finetune phase starts. With the trained S′ from the concluded first phase, the whole model is further finetuned for optimal weights W (S′) on the entire training data in Dtrain with S′ fixed. This allows the model parameters to be tuned on the unseen DB-val as well as DB-train.
Algorithm 1 Optimization algorithm Split training dataset Dtrain into two halves, {DB-train,DB-val}.
# Search phase while not converged do
Update model weights W using Eq.(4) on DB-train
Update similarity matrix S using Eq.(5) on DB-val end while
# Finetune phase With the learned S′, learn for optimal W on Dtrain until convergence.

heading: 3.6 Dimensionality reduction of S
text: The dimension of S is V × V which is generally difficult to optimize when V is very large. In this section, we discuss ways to reduce the dimensionality of S, making it substantially more convenient to optimize. We propose two different ways to reduce the dimension of S from V × V to V ×K where K V .
BFTSS Top-K : After the S matrix initialization as stated in Section 3.2, we choose the K words with the highest similarity scores and their corresponding indices from each row in the S matrix. The entries corresponding to the top-k words in each row of the S matrix are updated, thus reducing the dimension from V ×V to V ×Ktop-K where Ktop-K V .
BFTSS U-V : From Section 3.2, we initialize S as follows,
S = I + f({ÊT Ê}d) = I + Ŝ
where Ŝ = f({ÊT Ê}d). S is a full rank matrix because of the added identity matrix, but Ŝ may not be a full-rank matrix.5 Ŝ can be decomposed into a product of two lower-rank matrices. Thus to efficiently reduce the dimension of the S matrix, we apply rank reduction on Ŝ. We use PARAFAC (Bro, 1997) to decompose Ŝ into two factors U and V ∈ RV×KU-V of rank KU-V V such that,
S = I + f({ÊT Ê}d) = I + Ŝ ≈ I + U × V T
5There is an interdependency between similar words making their corresponding rows dependent.
Reconstruction of S matrix is not needed to perform soft embedding operations. The following operation is performed instead,
ê (t) i = e (t) i × S(t) × {E(t)}
T
= {E(t)i } T + h(((e (t) i × U)× V T )× {E(t)} T ),
(7)
where h is the Top-K operation which selects similar words dynamically contrary to the static selection in BFTSS Top-K. Multiplication is performed in a specific order to avoid reconstruction of S matrix everytime for soft embeddings. (e
(t) i × U) ∈ RKU-V is a KU-V-dimensional vector. V ∈ RV×KU-V . Thus, it boils down to product of a KU-V-dimensional vector with KU-V × V dimensional matrix (V T ). It is then multiplied with the embedding matrix to get a H dimensional soft embedding vector. Thus the computational complexity is of the same order as BFTSS Top-K approach.

heading: 4 Experiments

heading: 4.1 Datasets
text: We perform experiments on several datasets from GLUE (Warstadt et al., 2018; Wang et al., 2019). The GLUE datasets span a wide range of tasks such as linguistic acceptability (CoLA), semantic textual similarity (STS-B), paraphrase (QQP), natural language inference (RTE, QNLI, MNLI), and sentiment classification (SST-2). To simulate a low-resource finetuning scenario, 100, 300, and 1k examples are sampled from the original training dataset for training. The models are evaluated on the original development set following Xu et al. (2021).

heading: 4.2 Baselines
text: Many different baselines are compared with the proposed method in this paper. Vanilla finetuning is the classic finetuning method where the whole training dataset is used for training. RecAdam (Chen et al., 2020) is an advanced version of weight decay with time-varying coefficients for the crossentropy loss term and the regularization loss term. Child-D and Child-F (Xu et al., 2021) are methods where a mask is applied to the gradients to restrict the number of parameters being updated to avoid overfitting. Top-K-layer finetuning (Houlsby et al., 2019) only updates the top K layers, and Mixout (Lee et al., 2019) randomly replaced the updated parameters with the pretrained parameters. Finally EDA (Wei and Zou, 2019) is a popular data
augmentation method with a dependency on the knowledge resource WordNet (Fellbaum, 1998).
All baselines are evaluated with the pretrained BERT-base and BERT-large models. They are finetuned on the subsampled training datasets. The averaged results on the original development set over ten random seeds are reported following Xu et al. (2021). For BFTSS Top-K and BFTSS U-V, top 50 words and U-V dimension of 100 worked the best among other choices. More information about the hyperparameter tuning process can be found in the appendix.

heading: 4.3 Main Results
text: Table 1 shows the results of the average scores6 for the proposed and baseline methods. Models trained with our methods are most accurate compared to the baselines over all the sampled data sizes for both BERT-base and BERT-large models, often by large margins. This improvement indicates that our approach to using bi-level optimization to learn a task-dependent similarity structure for finetuning without any external knowledge is very effective in boosting the model’s performance over baselines. Among baseline methods, Mixout and Top-K-layer Tuning perform better than other baselines. However, there is still a substantial performance gap between these methods and our proposed methods. For example, BFTSS Top-K method achieves an average gain of 10.58%, 4.73%, and 1.50% over mixout in 100, 300, and 1K training examples scenario, respectively, on the BERT-base model . Our BFTSS U-V method achieves an average gain of 10.75%, 4.77%, and 1.39% over mixout in 100, 300, and 1K training examples scenario, respectively, on the BERT-base model . The trend is similar for BERT-large models and also when compared to Top-K-layer Tuning.
Because Mixout proposes to replace randomly sampled model parameters with pretrained model parameters while finetuning (Lee et al., 2019), and Top-K-layer Tuning only tunes the Top-K layers while freezing the remaining bottom weights, they both can be considered as putting restrictions on model capacity to avoid overfitting. Different from these methods, the proposed methods utilize information about unseen words in the form of a task-dependent similarity structure to serve as an informative prior for the model in finetuning. The model, especially the embeddings of the unseen
6Task performance values are in the appendix.
Method 100 300 1000
Vanilla 33.11 46.17 65.28 RecAdam 36.65 44.46 68.28 Child-D 38.38 52.65 66.88 Child-F 38.09 50.89 66.52 Top-K-layer 39.91 58.01 68.47 Mixout 43.97 58.28 68.80 EDA 52.95 56.95 62.92
BFTSS Top-K 54.55 63.01 70.30 BFTSS U-V 54.72 63.05 70.19
(a) Test Results (%) on all datasets with a BERT-base model.
Method 100 300 1000
Vanilla 38.70 56.80 69.31 RecAdam 36.53 56.92 70.16 Child-D 48.05 64.14 71.37 Child-F 47.51 63.05 70.18 Top-K-layer 51.86 64.94 72.05 Mixout 52.98 64.22 72.32 EDA 52.75 60.14 65.04 BFTSS Top-K 58.00 66.53 72.86 BFTSS U-V 58.10 66.50 73.11
(b) Test Results (%) on all datasets with a BERT-large model.
Table 1: Experiment results of the baseline methods and proposed methods. The reported numbers are the averaged evaluation metrics across all tasks with different number of sampled training data examples. Bold indicates the highest number, and italic indicates the second highest.
words, receives informative updates from limited training data. Results here show that by providing more information about unseen words in the vocabulary instead of restricting the tunable parameters, models can be trained to have better generalization without overfitting.
We also compare to the popular data augmentation method, EDA (Wei and Zou, 2019). Different from the baselines above, an external lexical resource, WordNet, is used in EDA for synonym replacement. Our method outperforms EDA in all datasplits despite having no access to any additional data. On the BERT-base model , our BFTSS Top-K method outperforms EDA by an average performance gain of 1.6%, 6.06%, and 7.38% in 100, 300, and 1000 training examples scenarios. Similarly, on the BERT-base model, our BFTSS U-V method outperforms EDA by an average performance gain of 1.77%, 6.1%, and 7.27% in 100,
300, and 1000 training examples scenarios. The trend is similar for the BERT-large models.
EDA can be seen as creating symbolic data augmentations by replacing words according to a general similarity structure coupled with other operations such as random swap, insertion and deletion. With increasing training examples, the accuracy improvement of our method over EDA increases. This result indicates that initially, the general similarity structure is helpful due to low amount of training data compared to no augmentation at all. However, as the training data increases, the general similarity structure along with other heuristics brings more noise than information, resulting in smaller gains and even performance loss compared to Vanilla finetuning. The task-specific similarity structure from our method can benefit the models in all cases, because it is close to the general similarity structure when the training data is small, and moves to a similarity structure tailored for the task when training data increases.
Finally, the two dimensionality reduction methods, Top-K and U-V, perform quite similarly under different conditions, which indicates that both dimensionality reduction methods provides similar benefits to model training.
4.4 Ablation experiments 4.4.1 Vanilla S-W The effectiveness of the bi-level optimization training is examined in the Vanilla S-W experiments, where Vanilla S-W method represents training the similarity structure S and the model W on the whole training dataset, DB-train, without the BFTSS framework. We use the U-V procedure for the dimensionality reduction of S. Fig. 2 compares our method with Vanilla S-W on BERT-base and
Method(Top-K) BERT-base BERT-large
Vanilla 33.11 38.70 Random S 40.04 43.78 Initial S 42.33 51.01 BFTSS Random 50.68 51.13 BFTSS 54.55 58.00
(a) Impact of initialization of S and using BFTSS Top-K at 100 data split settings. We report the average scores on all datasets
Method (U-V) BERT-base BERT-large
Vanilla 33.11 38.70 Random S 29.30 28.45 Initial S 42.51 52.46 BFTSS Random 37.01 33.56 BFTSS 54.72 58.10
(b) Impact of initialization of S and using BFTSS U-V at 100 data split settings. We report the average scores on all datasets.
Table 2: Results of experiments showing the impact of initialization of S. The Top-K and U-V indicate if Top-K or U-V, respectively, was used for dimensionality reduction of S.
BERT-large in 100 training examples scenario. Results show our method outperforms Vanilla S-W by a large margin. In Vanilla S-W , both the S and W parameters are learned on the training dataset without the bi-level optimization framework. Compared to Vanilla finetuning where S is not used, performance from Vanilla S-W models are much higher, indicating that the initial similarity structure is already helpful for updating all parts of the model. However, the bi-level learning of S and W is able to provide further benefit for model learning.

heading: 4.4.2 Initialization of S
text: Initial values in the S matrix are important to the success of the BFTSS. Experiments are conducted using the following initialization methods:
• Random S: Use a randomly initialized S for Finetune phase of the optimization algorithm without learning for a task-dependent similarity matrix.
• Initial S: Use the initial S for Finetune phase of the optimization algorithm without learning for a task-dependent similarity matrix.
• BFTSS Random: Perform the optimization algorithm on a randomly initialized S.
Table 2 compares our method with the three initialization methods of S described above. The first two initialization methods do not learn S with task data. Results show that our method outperforms both initialization methods emphasizing the need to learn for a task-dependent S. Interestingly, Random S Top-K outperforms Vanilla, indicating that when S is initialized randomly, static similar word selection strategy (BFTSS Top-K) regularizes the model’s performance where as dynamic similar word selection strategy (BFTSS U-V) is injecting noise into the model that is making the performance worse. Furthermore, Initial S performs better than Vanilla and Random S for both the rank reduction methods, indicating the initialization’s importance.
We further inspect the role of learning of S in our algorithm. By comparing BFTSS , BFTSS Random, and Random S, we can observe that BFTSS Random performs better than Random S. This observation indicates that a learned S is more helpful than a random initialization of S. Furthermore, BFTSS performs better than BFTSS Random, indicating the initialization of S from pretrained language models provides a more informative prior to the similarity structure than a random initialization. The proposed initialization of S is derived from the pretrained model’s embedding layer. Such a model is pretrained on a huge corpus of texts in an unsupervised fashion. The model would have learned a latent representation of the words in the vocabulary that captures semantic proximity between them. Thus an Initialized S is better than a Random S. However, since a pretrained model is pretrained on a huge corpus of texts, there is no guarantee that the proposed initialization of S is task-dependent. Hence, we need to further learn for a task-dependent S.

heading: 5 Conclusion
text: To mitigate the impact of overfitting leading to low generalization ability of large language models, especially in low-resource scenarios, we propose Bilevel Finetuning with Task-dependent Similarity Structure-BFTSS, a bi-level optimization framework to learn a task-specfic similarity structure and further enable the generalization ability of updating ‘unseen’ or ‘similar’ words of language models on a given task. We also introduce two variants, BFTSS Top-K and BFTSS U-V, to reduce the dimensionality and make computation efficient. Extensive experimental results on various datasets show the
effectiveness of our approaches in low-resource scenarios when comparing to different baseline methods.

heading: 6 Limitation
text: The way the method applies to larger datasets needs further exploration. As the number of training examples increases, the accuracy gain over vanilla finetuning reduces, indicating that our method best works in low-resource scenarios. Another limitation is that we performed experiments only in one language. It will be interesting to apply our method to tasks in other languages and understand the impact of task-dependent similarity structure on the model’s performance in those scenarios. BFTSS Top-K and BFTSS U-V methods perform similarly. Scenarios where BFTSS Top-K and BFTSS U-V differ in performance, should be further explored. We plan to address them in our future works.

heading: A Baselines
text: We include the following baselines in our experiments:
• Vanilla: Vanilla finetuning of the language model on the training dataset.
• RecAdam: Chen et al. (2020) proposed RecAdam optimizer to mitigate the effect of catastrophic forgetting while finetuning a large language model. RecAdam is an advanced version of weight decay with timevarying coefficients for the cross-entropy loss term and the regularization loss term.
• Child-D: Xu et al. (2021) proposed to update the weights of a sub-network within a large language model to tackle the fine-tuning instability and catastrophic forgetting issue. ChildD estimates a static mask with probability pD from the fisher information matrix at the beginning of the training. pD = {0.1, 0.2, 0.3}.
• Child-F: Also proposed in (Xu et al., 2021). Unlike Child-D, Child-F utilizes a dynamic mask during gradient computation. At every iteration, a mask is sampled from a Bernoulli distribution parameterized by pF = {0.2, 0.3, 0.4}.
• Top-K-layer Finetuning: Only the top-K layers are finetuned with the remaining bottom layers frozen (Houlsby et al., 2019). K = {0, 3, 6, 12}.
• Mixout: Lee et al. (2019) proposed to replace the language model parameters with their corresponding pretrained weights while finetuning with probability p = {0.1, 0.2, . . . , 0.8}. This way, the authors propose reducing the model’s deviation from the pretrained weights, thereby tackling the catastrophic forgetting issue.

heading: B Hyperparameter Settings
text: For all methods, we finetune the pretrained BERTbase7 and BERT-large8 models provided by Huggingface (Wolf et al., 2020). We follow the same settings for the models as Devlin et al. (2018). All models are trained using a batch size of
7https://huggingface.co/bert-base-cased/tree/main 8https://huggingface.co/bert-large-cased/tree/main
16, a warm-up ratio of 10%, and AdamW optimizer (Loshchilov and Hutter, 2019) with β1 = 0.9, β2 = 0.999, = 1e-6.
For one dataset, the averaged task performance is calculated using model performance evaluated from ten runs with different random seeds. The average scores, reported in the result tables in the main paper, are the averaged value of all task performance numbers. The task performance numbers are reported in the tables below in the appendix.
For our method the training can be decomposed into two phases, 1) Search phase and 2) Finetune phase. We report the hyperparameter settings for each phase and grid searched for the best hyperparameter setting for a task following Xu et al. (2021).
• Search phase: We grid search for Ktop-K parameter in {50, 100} and KU-V decomposition dimension in {100, 300}. We split the training dataset Dtrain into two halves, {DB-train,DB-val} to be used for stage 3.4.1 and then 3.4.2 of search phase respectively. We use the Adam optimizer for S with β1 = 0.9, β2 = 0.999, = 1e-8. We grid search for the learning rate of S in {5e-06, 4e-05}. We follow the same settings for W as Devlin et al. (2018). We grid search for the learning rate of W in {4e-05, 2e-05, 5e-06} and number of epochs in {3, 6, 9}. Since it is an optimization problem, we search for the hyperparameters from the provided choices that lead to a smooth reduction in the loss curves. Grid search is performed in a similar fashion as Xu et al. (2021).
• Finetune phase: We use the default hyperparameter settings for learning W ∗ following Devlin et al. (2018). We train for optimal W on full training datasetDtrain in this phase. We use the S∗ obtained in the search phase.
We follow the settings reported in the paper and use the code provided for all the baselines. For a fair comparison (Mosbach et al., 2020), we run the baselines for the same number of training steps as our entire algorithm (search phase and finetuning phase). We grid-searched over all choices of hyperparameters reported in their respective papers and report results on the best set of hyperparameters found. We used Betty library (Choe et al., 2022) for the MLO implementation. We use the V100 GPU
machine for all our experiments. Our algorithm executed in a couple of minutes which is comparable to the running time of vanilla finetuning.

heading: C Tables
text: Table 3 describes the datasets used for the training and evaluation of the models. It reports the number of examples in the dev set and the evaluation metrices for each task. Tables 4-9, shows the comparison of our method with all the baselines mentioned above for each dataset and split settings. Tables 10- 13 shows the results of our ablation studies with different initializations of S matrix to understand the impact of initialization. Last column (Avg) is the average performance metric of the model for a given approach across all the tasks. Table 14 shows the comparison of our method with Vanilla S-W on BERT-base and BERT-large for each task to understand the impact of BFTSS.
Method CoLA STSB SST-2 QQP QNLI MNLI MNLI-m RTE Avg Vanilla 8.72, 14.33
(37.75)
51.93, 13.06 (68.51) 58.42, 9.64 (78.33)
9.72, 18.35
(46.86)
61.83, 8.11 (70.05) 33.94, 1.96 (36.33)
33.82, 1.63 (35.81)
51.19, 3.63
(56.32) 38.70
Random S U-V 1.14, 3.03
(6.56) -14.66, 35.26 (29.0)
50.5, 1.35
(53.56)
22.42, 28.75 (57.26) 52.59, 3.82 (61.47) 32.99, 1.82 (36.94)
33.01, 1.49 (36.07)
49.57, 3.03
(53.07) 28.45
Initialized S U-V 18.82, 10.59
(31.91)
72.48, 7.79 (81.19) 82.49, 4.32 (87.61) 46.12, 18.53 (66.9)
66.71, 7.48 (73.57) 38.38, 2.79 (43.8)
39.18, 3.64 (43.5)
55.52, 3.92
(61.37) 52.46
BFTSSRandom U-V 0.71, 3.28
(5.42)
7.25, 38.19
(53.02)
50.88, 2.18 (55.62)
38.8, 26.77
(57.17)
51.4, 5.45
(65.07)
34.14, 2.38 (38.32)
34.22, 2.76 (40.5)
51.05, 3.38
(55.23) 33.56
BFTSSU-V 30.48, 8.22
(44.52)
76.62, 2.16 (79.55) 87.31, 1.7 (89.68) 58.35, 4.78 (63.24) 71.98, 1.85 (74.35) 40.72, 2.14 (43.76)
41.88, 4.04 (50.16)
57.44, 3.93
(67.51) 58.10
(a) Comparison of Vanilla S −W with our approaches, with BERT-base model, on 100 data split settings.
(b) Comparison of Vanilla S −W with our approaches, with BERT-large model, on 100 data split settings.
Table 14: Results for the Vanilla S-W experiments. For each task, average, and maximum of the evaluation metric over ten random seeds have been reported in the tables.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We discussed in Section 6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? We discussed in Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We discussed in Section 4.1 and Section Appendix B
3 B1. Did you cite the creators of artifacts you used? We discussed in Section 4.1 and Section Appendix B
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We discussed in Section 4.1 and Table 3.
C 3 Did you run computational experiments? We discussed in Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We discussed in Appendix B.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? We discussed in Section 4.2 and Appendix B.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We discussed in Appendix B.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We plan to address them in our future works