STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models (2024)

Abstract: Though Large Language Models (LLMs) have demonstrated the powerful capabilities of fewshot learning through prompting methods, supervised training is still necessary for complex reasoning tasks. Because of their extensive parameters and memory consumption, both Parameter-Efficient Fine-Tuning (PEFT) methods and Memory-Efficient Fine-Tuning methods have been proposed for LLMs. Nevertheless, the issue of large annotated data consumption, the aim of Data-Efficient FineTuning, remains unexplored. One obvious way is to combine the PEFT method with active learning. However, the experimental results show that such a combination is not trivial and yields inferior results. Through probe experiments, such observation might be explained by two main reasons: uncertainty gap and poor model calibration. Therefore, in this paper, we propose a novel approach to effectively integrate uncertainty-based active learning and Low-Rank Adaptation (LoRA). Specifically, for the uncertainty gap, we introduce a dynamic uncertainty measurement that combines the uncertainty of the base model and the uncertainty of the full model during the iteration of active learning. For poor model calibration, we incorporate the regularization method during LoRA training to keep the model from being overconfident, and the Monte-Carlo dropout mechanism is employed to enhance the uncertainty estimation. Experimental results show that the proposed approach outperforms existing baseline models on three complex reasoning tasks.1

heading: 1 Introduction
text: Large Language Models (LLMs) (Brown et al., 2020; Wei et al., 2021; OpenAI, 2022; Touvron et al., 2023a,b; Zhao et al., 2023) have demonstrated the powerful capabilities of zero/few-shot
∗Equal Contribution. †Corresponding Author.
1Our code and results will be available at https:// github.com/callanwu/STAR.
learning with prompting techniques, including InContext Learning (Dong et al., 2022) and Chainof-Thought (Wei et al., 2022), where no parameter update is required. However, previous studies (Hendrycks et al., 2020; Yuan et al., 2023; Bai et al., 2023; Isik et al., 2024) have shown that further fine-tuning is still crucial for tasks involving complex reasoning such as arithmetic reasoning (Roy and Roth, 2016; Cobbe et al., 2021) and commonsense reasoning (Mihaylov et al., 2018; Clark et al., 2019), etc.
Fine-tuning LLMs requires updating a large number of parameters, which takes a lot of time and consumes considerable memory. Taking LLaMA7B (Touvron et al., 2023a) as an example, finetuning it on a dataset of 52k instances takes over 12 hours on 4 A100 80G GPUs (Bommasani et al., 2021; Taori et al., 2023). Therefore, Parameter Efficient Fine-Tuning (PEFT) methods (Houlsby et al., 2019; Lester et al., 2021; Li and Liang, 2021; Hu et al., 2021; Ding et al., 2023) and Memory Efficient Fine-Tuning (MEFT) methods (Liao et al., 2023) have been proposed. In addition to updating a vast number of parameters and consuming substantial memory, a neglected factor in LLMs fine-tuning is the extensive consumption of annotation data. Moreover, due to the inherent complexity of tasks, the human annotation resources required
3519
for fine-tuning LLMs are also significant (Ouyang et al., 2022).
Therefore, it is important to develop the DataEfficient Fine-Tuning (DEFT) method for LLMs. A common practice to improve data efficiency is active learning (Cohn et al., 1996; Settles, 2009), while it has been shown that the PEFT methods can alleviate the reliance on annotated data to some extent (Ding et al., 2023). A straightforward idea for DEFT is to combine the PEFT method with active learning. However, such a combination is not trivial. As shown in Figure 1 (a), simply finetuning an LLM with LoRA (Hu et al., 2021) under an uncertainty-based active learning framework yields consistently inferior performances compared to passive learning (random selection of data in active learning) on the OpenBookQA dataset, while as shown in Figure 1 (b), fine-tuning LLM with full parameter under active learning yields performances better than passive learning.
To investigate the mechanism behind this uncommon phenomenon, probe experiments are conducted by investigating the prediction confidence and entropy of the LLM with LoRA during uncertainty-based active learning. Based on the experimental results, we deduce two potential reasons for this phenomenon. The first issue is uncertainty gap. To be more specific, the uncertainty calculated for selecting data during active learning comes from the full parameters, while only partial parameters remain tuned during PEFT. It suggests that the conventional way of calculating uncertainty may not reflect the knowledge required by the PEFT parameters and therefore undermines the performance of active learning. The second issue is poor model calibration which becomes particularly significant when using the PEFT method (Wang et al., 2023). It further indicates that the uncertainty calculated in the conventional way is not well-calibrated, and the data selected for active learning becomes suboptimal.
To address the aforementioned issues, we propose conStrainT LoRA with dynamic Active leaRning (STAR), a novel approach to effectively integrate uncertainty-based active learning and LoRA. Specifically, for the uncertainty gap, we introduce a dynamic uncertainty measurement that combines the uncertainty of the base model and the uncertainty of the full model during the iteration of active learning. For poor model calibration, we incorporate the regularization method during
LoRA training to keep the model from being overconfident, and the Monte-Carlo dropout mechanism (Gal and Ghahramani, 2016) is employed to enhance the uncertainty estimation. Experimental results show that the proposed approach outperforms existing baseline models on three complex reasoning tasks. The above issues are partially resolved.
In conclusion, our contributions are three-fold:
• As far as we know, we are the first to investigate and uncover the reasons why directly combining active learning with LoRA fails to achieve comparable performance with passive learning through probe experiments.
• A novel DEFT method, STAR, is proposed to effectively combine PEFT with active learning through criterion revision and model regularization.
• Extensive experimental results show that the proposed method addresses the issues and outperforms other baselines.

heading: 2 Related Work

heading: 2.1 Efficient Fine-tuning Methods
text: As LLMs continue to expand in size, the computational and financial resources required for fine-tuning these models become increasingly prohibitive. To address this challenge, Efficient FineTuning has emerged as an essential area of research (Wan et al., 2023). The methods can be classified into PEFT and MEFT (Wan et al., 2023; Liao et al., 2023). PEFT, in particular, aims to adjust a minimal subset of the model’s parameters, thus conserving computational resources while maintaining or enhancing model performance (Hu et al., 2023). We classify PEFT methods into four categories: Prompt Tuning (Lester et al., 2021; Liu et al., 2023b) only allow an additional k tunable tokens per downstream task to be prepended to the input text. Prefix Tuning (Li and Liang, 2021; Liu et al., 2022) keeps language model parameters frozen but optimizes a small continuous taskspecific vector that pretends to be key-value pairs. Adapter (Houlsby et al., 2019; He et al., 2021) is a new module added between layers of a pretrained network, which is a bottleneck architecture. Low-Rank Adaptation (LoRA) (Aghajanyan et al., 2021; Hu et al., 2021) reduces the parameters and enhances computational efficiency by applying low-rank matrices. Moreover, minimizing
memory usage in fine-tuning for improving efficiency has also emerged as a critical topic (Liao et al., 2023), with several innovative solutions being proposed. Among these, techniques such as QLoRA (Dettmers et al., 2023), QA-LoRA (Xu et al., 2023), and LoftQ (Li et al., 2023) stand out for their ability to significantly reduce memory requirements without compromising model performance. In this paper, we focus on the application of LoRA.

heading: 2.2 Active Learning with LLMs
text: Active Learning (AL) has been extensively investigated across a multitude of NLP tasks, encompassing machine translation (Miura et al., 2016; Zhao et al., 2020), natural language inference (Snijders et al., 2023), named entity recognition (Shen et al., 2017) and text classification (Ein-Dor et al., 2020; Margatina et al., 2022; Schröder et al., 2023). In the era of LLMs, active learning is primarily employed in the selection of prompts and the annotation of data (Zhang et al., 2023b; Liu et al., 2023a; Xiao et al., 2023). For instance, Margatina et al. (2023) explores various active learning strategies for selecting the most relevant examples for in-context learning with LLMs. Diao et al. (2023) introduces an active prompting method that leverages uncertainty metrics to select questions for annotation. In the domain of integrating PEFT with AL, Jukić and Snajder (2023a) explored PEFT methods with different active learning in Pre-trained language models (PLMs) such as BERT (Devlin et al., 2019), demonstrating that the integration of PEFT with active learning can offer substantial performance gains. Different from Jukić and Snajder (2023a), we apply decoder-only generative LLMs as the backbones, to our knowledge, we are the first to integrate LLMs combined PEFT with AL within the realm of reasoning tasks.

heading: 3 Preliminaries

heading: 3.1 Parameter Efficient Fine-tuning
text: Parameter-Efficient Fine-Tuning (PEFT) methods aim to fine-tune only a small set of external parameters while keeping the backbone model frozen and achieving comparable or even superior performance (Hu et al., 2023). The main-steam PEFT methods include the adapter-based methods (Houlsby et al., 2019; He et al., 2021), prefix tuning (Li and Liang, 2021), and LoRA (Hu et al., 2021), among which LoRA is the most ef-
fective and widely used. In this paper, we mainly implement PEFT with LoRA.
LoRA (Low-Rank Adaptation) assumes that the updation of the model weight matrix during training is low-ranked, which can be decomposed as the multiplication of two low-rank matrices.
∆W = αBA (1)
where ∆W is the updation of the model weight matrix, B ∈ Rd×r and A ∈ Rr×k are matrices of rank r, and α is constant scaling factor.
During training, the model weight matrix W is fixed and only ∆W is optimized. It is worth noticing that commonly A is randomly initialized and B is zero-initialized. In this way, we have W = W +∆W at the beginning of training and the fine-tuned model is identical to the base model.

heading: 3.2 Active Learning
text: Active Learning (AL) methods aim to select informative examples from the data pool to maximize the performance with the required data budget or minimize the data budget to achieve the required performance. The family of AL methods mainly includes uncertainty-based methods (Lewis, 1995; Gal and Ghahramani, 2016), diversity-based methods (Sener and Savarese, 2018), and discriminativebased methods (Gissin and Shalev-Shwartz, 2019), where uncertainty-based methods are widely used and easy for implementation.
In our study, we mainly consider three AL strategies, including RANDOM selection as a passive learning baseline and two uncertainty-based criteria. Maximum Entropy (Lewis, 1995) and Predictive Entropy (Duan et al., 2023; Kadavath et al., 2022) are both based on uncertainty, but the former is label independent while the latter is label dependent. The key idea behind uncertainty-based AL methods is that models will learn more efficiently from examples in which they are difficult to predict and have high prediction uncertainty.
LLMs (Touvron et al., 2023b) inherently generate sentences in a free-form and auto-regressive fashion. This process entails the sequential prediction of the probability distribution for the subsequent token in a sentence. Let x represent the input prompt, and s denote the sentence generated by the LLM, comprising N tokens in total. For any given LLM, the probability of producing a specific token zi as the i-th element in the sentence can be mathematically expressed as p(zi|s<i, x), where
1 ≤ i ≤ N . Here, s<i symbolizes the sequence of previously generated tokens {z1, z2, ..., zi−1}. MAXIMUM ENTROPY(ME) is characterized by its independence from golden response. It quantitatively evaluates the uncertainty in a model’s predictions by computing the entropy across all possible outcomes, formulated as:
ME(s, x) =
− N∑
i=1
V∑
j=1
p(vij |s<i, x) log p(vij |s<i, x) (2)
where s is the generated response, p(vij |s<i, x) is the probability of j-th token in vocabulary at i-th element in s, V is the vocabulary size. PREDICTIVE ENTROPY(PE) incorporates golden response dependency, offering a measure of the expected information gain from the true label, given the predictive distribution. It is formulated as:
PE(s, x) = − log p(s|x)
= N∑
i=1
− log p(zi|s<i, x) (3)
where s is the golden response, p(zi|s<i, x) is the probability of i-th token in the golden response.

heading: 4 Probing PEFT on Prediction Uncertainty
text: In this section, we describe how to design probe experiments to investigate the reason behind the failure of LoRA combined with AL methods. We will first introduce the experiment setup, and then we will talk about how to prob LoRA under the AL framework with prediction confidence and prediction entropy. We also discuss how to conclude from the experimental results.

heading: 4.1 Probe Experiment Design
text: As uncertainty-based AL methods mainly depend on the confidence or uncertainty of model predictions to select examples during each iteration, it is straightforward to probe the confidence and uncertainty of model predictions during AL iterations.
We mainly focus on two training variants of LLMs. The first one, denoted as PEFT method, is LLM finetuned with LoRA, which is shown to be problematic under the AL framework. The second one, denoted as Few-shot method, is untuned LLM, which is taken as a control group. To enhance the performance of untuned LLM on downstream tasks,
we employ In-Context Learning (Dong et al., 2022) prompting by adding demonstrations with the input prompt. LLaMA-2 (Touvron et al., 2023b) serves as the backbone LLM. Experiments are conducted on the BoolQ dataset (Clark et al., 2019) because its labels only include “true” and “false”, which makes the prediction uncertainty and prediction confidence easy to calculate.

heading: 4.2 Probing with Prediction Confidence
text: The first probe experiment is designed to explore whether the model prediction confidence of the PEFT method exhibits issues compared to Fewshot methods. The prediction confidence CF is measured by the maximum between the output probability on token “true” and “false”.
CF = max(ptrue, pfalse) (4)
where ptrue and pfalse denotes the probabilities of token “true” and “false”, respectively.
Then the prediction confidence of PEFT and Few-shot are calculated and the density plot is drawn to make a comparison between these two methods as shown in Figure 2. To mitigate differences in model accuracy, we only consider the confidence of the model for the wrong predictions on the test set of BoolQ. The intuition is that for examples that the model is less likely to predict right, they should be less confident.
The PEFT method achieves an accuracy of 73.36%, which is much higher than the Few-shot method with an accuracy of 45.41%. As shown in Figure 2, the PEFT method is overconfident compared to the Few-shot method, where the confidence of the wrong prediction is as high as 70%, which indicates a model calibration issue.

heading: 4.3 Probing with Prediction Entropy
text: The second probe experiment is designed to investigate the change of prediction entropy of PEFT model during active learning iteration. The MAXIMUM ENTROPY(ME) is employed as the uncertainty during active learning. Nine rounds of iteration are performed with 500 examples selected during each iteration. The PEFT model is trained with 500 examples at the beginning as a warm-up.
The correlation between examples with top 1000 entropy at the beginning is calculated and the heatmap of the correlation is shown in Figure 3 (a). As we can observe in Figure 3 (a), the correlation between the base model (model without PEFT tuning) and models after AL iteration is close to 0, which indicates a clear gap between the base model and PEFT model. This phenomenon is even clear with the scatter plot in Figure 3, where the dots in Figure 3 (b) should appear around the red line but appear in the upper triangular region. In Figure 3 (c), the correlation coefficients of entropy between the two iterations become relatively normal, which is consistent with Figure 3 (a), suggesting that the gap between iterations has been alleviated.

heading: 5 Methods
text: In this section, we introduce the proposed method STAR in detail. We will first describe the overall workflow of STAR, then we will discuss methods to address the uncertainty gap issue and the model calibration issue. Finally, we will conclude the proposed method with a pseudocode.

heading: 5.1 LoRA under Active Learning Iteration
text: As shown in Figure 4, the k-th iteration of STAR consists of the following steps.
1. Model Inference that employs the present model Mk to make inference on unlabeled dataset DUk .
2. Data Querying that selects the most informative examples to form a subset SUk with the results of inference based on the dynamic uncertainty estimation method.
3. Data Labeling that labels the unlabeled subset SUk to form the labeled subset S L k .
4. Dataset Updating that updates the labeled dataset DLk by appending appending the labeled subset DLk+1 = D L k ∪ SLk .
5. Model Training that updates the present model with new labeled dataset DLk+1 to get model Mk+1 for next iteration.

heading: 5.2 Dynamic Uncertainty Measurement
text: To address the issue of uncertainty gap, we proposed a dynamic uncertainty measurement to integrate the uncertainty of the frozen LLM (base model) and the uncertainty of LLM fine-tuned with LoRA (full model) dynamically based on the AL iteration.
The key idea is that at the beginning of PEFT training, the extra parameters are under-fitting, where the uncertainty calculated is less reliable than the frozen parameters. As the iteration of active learning increases, the uncertainty of the full model becomes more reliable, which is similar to the zero-initialized attention weight in LLaMAadapater (Zhang et al., 2023a).
µ = λ(t)µb + (1− λ(t))µf (5)
where µb and µf denote the prediction uncertainty of the base model and the full model respectively,
λ(t) ∈ [0, 1] is a monotone decreasing function of AL iteration t. Note that, our measurement approach only requires one additional computation of the base model at the beginning, which remains constant throughout and does not significantly increase the FLOPs.

heading: 5.3 Calibration with Hybrid Regularization
text: To address the issue of poor model calibration, we propose a hybrid regularization method during PEFT training. As discussed in Section 4.2, the PEFT model demonstrates a pronounced tendency toward over-confidence, which indicates that the model is over-fitting.
Common approaches to prevent the model from being over-fitting include early-stoping (Doan and Liong, 2004), regularizations (Santos and Papa, 2022), and ensemble methods (Ganaie et al., 2022). Considering the difference between LoRA parameters A and B, we integrate two regularization methods into a hybrid regularization to keep LoRA from being over-fitting.
For the B matrix, which is zero-initialized, a L2
norm weight decay is employed.
Bt ← Bt−1 − γ(gt−1 − βBt−1) (6)
where gt−1 denotes the normalized gradient acquired from the standard Adam optimizer, and β denotes the strength of regularization.
For the A matrix, which is randomly Gaussian initialized N(0, 1), the Monte-Carlo dropout
(MC dropout) (Gal and Ghahramani, 2016) is adopted for more robust uncertainty estimation. MC dropout works by activating the dropout unit both in the training and inference stages, which can be regarded as an approximation to the Bayesian Neural Network. With the dropout unit activated during the inference stage, neural networks can generate different outputs with the same input, where expectations can be taken for more robust estimation.
µf = 1
K
∑
k
µ (k) f
µ (k) f = ME(LLM(x|Âk, B̂k))
(7)
where K denotes the number of feedforward propagations during the inference stage, µ(k)f denotes the uncertainty estimated at k-th feedforward, Âk and B̂k denote the LoRA matrices sampled from A and B with dropout unit activated.

heading: 5.4 Overall Algorithm
text: The overall algorithm of STAR is shown in Algorithm 1.

heading: 6 Experiments

heading: 6.1 Datasets
text: In our research, we employ three benchmark datasets spanning two categories of reasoning problems for AL evaluation: Arithmetic Reasoning: the GSM8K
dataset (Cobbe et al., 2021) comprises approximately 8.5K high-quality linguistically diverse grade school math word problems created by human problem writers. Commonsense Reasoning: (1) the BoolQ dataset (Clark et al., 2019) is a specialized question-answering dataset designed for yes/no questions; (2) the OpenBookQA dataset (Mihaylov et al., 2018) is a four-way multiple-choice question-answering dataset.
See Appendix A for more details about the dataset.

heading: 6.2 Settings
text: Experimental setup In the experiment conducted on the GSM8K and BoolQ datasets, we incrementally selected 500 new instances in each step of the AL experiment. The initial warm start for the AL setting is established by randomly choosing 500 instances. Furthermore, we adhere to a label-
ing budget constraint of 5,000 instances for each dataset. Considering the size of the training set for OpenBookQA, we design the AL framework to incrementally select 200 new instances during each iteration. The labeling budget for this process is set to 2,000 instances. The details regarding the evaluation can be found in Appendix C. Implementations In the empirical study, we utilize the state-of-the-art openly accessible LLM, LLaMA2-7B (Touvron et al., 2023b)2 as the base model. For comprehensive details on the hyperparameters employed in our experiments, please refer to Appendix B.

heading: 7 Result and analysis

heading: 7.1 Main Result
text: Table 1 presents a detailed comparison of different methods’ performance, evaluated across three
2https://huggingface.co/meta-llama/Llama-2-7b
Algorithm 1 STAR Input:unlabeled dataset DU , labeled dataset DL, the LLM M , number of iteration N , size of subdataset during iteration m.
1: initialize DU0 and D U 0 2: warm-up LLM M0 3: for k = 0 to N : 4: making inference with Mk on DUk 5: querying subset SUk from D U k based on
Equation (2) 6: updating DUk+1 ← DUk \ SUk 7: labeling SUk to get S L k 8: updating DLk+1 ← DLk ∪ SLk 9: fine-tuning LLM Mk to get Mk+1 on DLk+1
based on Equation (6) and Equation (7) 10: return LLM after fine-tuning MN
different datasets: GSM8K, BoolQ, and OpenBookQA. RANDOM serves as a fundamental baseline, with its AUC listed.
Both original PE and ME methods underperform compared to RANDOM on these three datasets in terms of AUC. The RIPL metric also hovers around zero, indicating that the original AL strategy is essentially ineffective.
After applying our proposed STAR method, PE and ME exhibit superior performance across all datasets and metrics. For instance, in the GSM8k dataset, ME w/ STAR achieves an AUC of 28.83 and a RIPL of 2.01, indicating a notable advancement over the baseline RANDOM and ME. The improvements are most pronounced in the OpenBookQA dataset, where ME w/ STAR method achieves a remarkable RIPL of 7.47. Furthermore, in the BoolQ dataset, ME w/ STAR achieves higher performance compared to the PE w/ STAR. This pattern of ME w/ STAR outperforming PE w/ STAR is consistent
across the GSM8K and OpenBookQA datasets as well. These results suggest that the ME w/ STAR is more effective.
Then, we explore how the models’ performance changes as the training set increases. Figure 5 shows the learning curves for corresponding AL methods on GSM8K, BoolQ, and OpenBookQA datasets, respectively. The RANDOM baseline and the two original active learning approaches perform comparably, suggesting that the active learning methods appear to be ineffective. Notably, the BoolQ dataset exhibits particularly high variability in results when using the PE strategy, which may be attributed to BoolQ’s binary output format of “true” and “false”. The gap between the full model and the base model could easily lead to skewed predicting results in a single iteration.
It is evident that w/ STAR methods demonstrate the most significant improvement on the OpenBookQA dataset. After applying our method, the model learned truly more useful samples. For instance, as evidenced by the BoolQ dataset in Figure 5, the performance of the model reaches saturation with just 1000 samples. This indicates that the selected samples are sufficiently diverse and useful for model learning.

heading: 7.2 Ablation Study
text: Since our methods have two main components, which are dynamic uncertain measurement and calibration with hybrid regularization as described in Sec 5. We conduct a detailed ablation study to assess the effect of the two components. As shown in Table 2, upon employing the dynamic uncertain measurement, all AUC are improved, and the RIPL turns positive. This indicates a significant gap between the full model and the base model in the original strategies, which our dynamic indicator effectively mitigated.
Subsequently, building on this foundation and individually incorporating MC dropout and L2 norm weight decay, it is observed that both constraint methods enhance performance, with MC dropout offering a more substantial improvement. The addition of calibration methods indeed effectively mitigates the issue of model over-confidence and improves model calibration.

heading: 8 Conclusion
text: In this paper, to improve the data efficiency of Large Language Models (LLMs) during the finetuning process, we propose a data-efficient parameter tuning method by combining LoRA with active learning. To address the issue that uncertaintybased active learning fails to combine with LoRA, we experimentally identify and summarize two possible reasons: uncertainty gap and poor model calibration. To resolve the uncertainty gap issue, we propose a dynamic uncertainty calculation method, and to address poor model calibration, we introduce a regularization-based constraint method. By integrating these two approaches, we partially solve the aforementioned failure issues. Extensive experiments show that our proposed method outperforms baseline models on multiple reasoning datasets.
Limitations
Though achieving promising results in the experiments, our work still has the following limitations.
• Due to constraints on computational resources, we did not conduct experiments on larger versions of LLaMA2 from 13B to 70B, nor did we experiment with other types of LLMs including BLOOM, Falcon, etc.
• Due to limitations in computational resources and time, we did not explore the combination of other types of PEFT methods (series/parallel adapters, prefix tuning) with different types of active learning methods (diversity-based active learning). Therefore, the validity of the methods and conclusions in this paper for a wider combination of PEFT and active learning remains unexplored. Further work should include exploring a more extensive combination of PEFT and active learning.
• We only speculated on the reasons for the failure of combining LoRA with active learning
through simple probe experiments, without delving deeper into the underlying mechanisms. Future work should involve exploring the deeper mechanisms behind this phenomenon.

heading: Acknowledgement
text: The authors would like to thank the anonymous reviewers for their insightful comments. This work is funded by the National Natural Science Foundation of China (62176053). This work is supported by the Big Data Computing Center of Southeast University.

heading: A Dataset
text: Table 3 shows the statistics of the dataset. In light of the unique tasks associated with each dataset, we implement a structured template approach. This template tailors the content and responses to the specificities of each dataset. We give the data templates for each dataset used to fine-tune LLM in Table 4.

heading: B Model Hyper Parameters
text: Following the prior works (Hu et al., 2021; Li et al., 2023), we maintain the original weights of the backbone architecture unchanged and integrate low-rank adapters into the Multi-Head Attention(MHA) and Feed-Forward Network(FFN) components of all layers. These low-rank adapters are configured with a rank of 64 and a factor of α set to 16, alongside a dropout rate of 0.1 to mitigate overfitting. The model parameters are optimized by AdamW (Loshchilov and Hutter, 2018). We use a batch size of 8 and a learning rate of 1.5e-4 for the GSM8K task and a batch size of 32 and a learning rate of 3e-5 for the BoolQ task and the OBQA task. In the AL setting, the model is trained for a fixed number of epochs: 3 epochs for the GSM8K task, and 15 epochs for both the BoolQ and OBQA tasks. All reported results are averaged over three runs. Our implementation leverages the PyTorch3 framework and HuggingFace Transformers4 library (Wolf et al., 2020). Our experiments are carried out with an NVIDIA A100 80GB GPU.

heading: C Evaluation
text: Following previous work (Schröder et al., 2022; Jukić and Snajder, 2023b,a), our study utilizes the Area Under the Curve (AUC) metric to assess the comprehensive efficacy of the methods we propose. The accuracy metric (Acc.) is employed for evaluating the effectiveness at each individual AL step.
To ascertain the success of AL, we compute the Relative Improvement over Passive Learning
3https://github.com/pytorch/pytorch 4https://github.com/huggingface/transformers
(RIPL), delineated as follows:
RIPL(SAL, SPL) = AUC(SAL)−AUC(SPL)
1−AUC(SPL) (8)
where SAL and SPL denotes AL methods and RANDOM method. RIPL serves as an estimator for the quotient of the maximal attainable enhancement that an AL approach can secure over the conventional passive learning benchmark. A RIPL score of 1 signifies the epitome of theoretical enhancement, equating to achieving an Acc. of 1 during the initial sampling phase and maintaining this optimum performance across all subsequent stages. In contrast, a RIPL score below 0 suggests that the AL strategy is outperformed by passive learning approaches.

heading: Future Work
text: Future work should involve exploring the deeper mechanisms behind this phenomenon