Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values
Year: 2023

Abstract:
Although Shapley values have been shown to be highly effective for identifying harmful training instances, dataset size and model complexity constraints limit the ability to apply Shapleybased data valuation to fine-tuning large pretrained language models. To address this, we propose TS-DSHAPLEY, an algorithm that reduces computational cost of Shapley-based data valuation through: 1) an efficient samplingbased method that aggregates Shapley values computed from subsets for valuation of the entire training set, and 2) a value transfer method that leverages value information extracted from a simple classifier trained using representations from the target language model. Our experiments applying TS-DSHAPLEY to select data for fine-tuning BERT-based language models on benchmark natural language understanding (NLU) datasets show that TS-DSHAPLEY outperforms existing data selection methods. Further, TS-DSHAPLEY can filter fine-tuning data to increase language model performance compared to training with the full fine-tuning dataset.

heading: 1 Introduction
text: Large language models (LMs) have achieved stateof-the-art performance on many natural language processing (NLP) tasks (Radford et al., 2019; Brown et al., 2020; Sanh et al., 2022). To adapt these models to new datasets and tasks, the standard approach is to fine-tune a pre-trained LM on a targeted downstream task. This allows the pre-trained general linguistic knowledge to be leveraged while fine-tuning to learn the task-specific information. However, during fine-tuning, pre-trained LMs are prone to significant performance degradation in the presence of noisy data (Srivastava et al., 2020). This effect may be further amplified when noisy or otherwise harmful instances are highly influential to the model parameters (Koh and Liang, 2017). As a result, it is important to identify harmful in-
stances in the fine-tuning data that may obfuscate the task information and degrade performance.
To automatically identify harmful data, prior works have used training dynamics (Swayamdipta et al., 2020) and estimation of marginal contributions via leave-one-out retraining (Cook, 1977) or influence functions (Koh and Liang, 2017). Shapley values, which satisfy certain desirable fairness guarantees, have also recently been adopted from cooperative game theory to measure datum contributions, where a data point’s Shapley value is the average marginal contribution to every possible data subset (Ghorbani and Zou, 2019).
In practice, Shapley-based data values are approximated using various techniques (Ghorbani and Zou, 2019; Jia et al., 2019b, 2021; Kwon and Zou, 2022; Schoch et al., 2022), as exact Shapley value computation over a dataset would require exhaustively retraining the model for every datum on
266
every possible subset (i.e. exponential complexity with respect to the number of data points). However, many of the existing approximation methods still exhibit a computational bottleneck when considering datasets and models at scale (e.g. datasets larger than 5K instances). This, in turn, directly limits the application of Shapley-based data valuation to state-of-the-art LMs and many NLP datasets.
To address the challenges posed by 1) the model constraint (the model retraining requirement) and 2) the dataset constraint (the time-complexity/dataset size relation), we propose Transferred Sampling Data Shapley (TSDSHAPLEY), an algorithm that utilizes two novel components that directly address each constraint. Specifically, to address the model constraint, we propose to compute Shapley-based data values using a simple, linear model that is trained on the learned representation from the target LM. Additionally, to address the dataset constraint, we propose a sampling-based method that computes Shapley values on data subsets and aggregates them for valuation of the entire training set.
Our contributions are as follows: 1) we propose a sampling-based data Shapley computation method and demonstrate its efficacy empirically using as little as 2% of the original training data; 2) we propose the use of a simple linear classifier with a target model’s pre-trained representation and demonstrate empirically the performance gains achieved over alternate pre-trained embeddings; and 3) we show the efficacy of Shapley-based data valuation and selection methods on benchmark NLU tasks using fine-tuned large LMs.1

heading: 2 Related Work
text: While Shapley values are often applied in a post hoc manner following model training (Ghorbani and Zou, 2019; Kwon and Zou, 2022; Jia et al., 2019a,b, 2021; Schoch et al., 2022), the demonstrated efficacy makes it a natural extension to apply such methods for data selection prior to training. To this end, Shapley values have been used for evaluating data for transfer learning (Parvez and Chang, 2021) and in active learning (Ghorbani et al., 2021).
Further, although Shapley-based data values have primarily been considered model-specific, in practice, a subset of training instances that may
1Code is available at https://github.com/ stephanieschoch/ts-dshapley
harm performance may be mislabeled (Koh and Liang, 2017; Swayamdipta et al., 2020; Ghorbani and Zou, 2019) or exhibit spelling mistakes or grammatical errors (Sun et al., 2020; Srivastava et al., 2020), which should be intrinsic to the dataset. Prior works have demonstrated the transferability of Shapley-based data values across various classifier architectures (Schoch et al., 2022) and have demonstrated the efficacy of surrogate KNN classifiers using pre-trained embeddings (Jia et al., 2021). Notably, our work differs in that we utilize the pre-trained embeddings extracted from the target LM and avoid the k-nearest neighbor assumption that training data far from a test datum do not contribute to its prediction (Jia et al., 2019a).

heading: 3 Method
text: Let D = {(xi, yi)}ni=1 denote a training set containing n training instances. For each training instance i, the Shapley value ϕi is defined as the average marginal contribution of i to every possible subset S ⊆ D that contains this instance (Ghorbani and Zou, 2019): ϕi = ∑ S⊆D;i∈S 1
( n−1|S\{i}|) {vA(S)− vA(S\{i})}
where vA(S) is a value function, typically defined as the development accuracy of model A trained on S. The challenge of calculating ϕi is two-fold: the exponential complexity of all possible subsets S ⊆ D and the computational cost of training A on each S and S\{i}. While Shapley-based data values are approximated in practice, most existing approximation methods are not efficient enough for large scale learning problems.

heading: 3.1 TS-DSHAPLEY
text: Let Atgt be the target classifier (i.e. large LM) that we want to fine-tune on a subset of D. To reduce computational cost, we propose to (1) use a linear classifier Asrc as the proxy of Atgt for data valuation; (2) use multi-chain Monte Carlo sampling to compute Shapley values on different subsets of D. For faithful data valuation, we further propose to train Asrc on the data representations extracted from Atgt.
Representation Extraction. We extract the representations from the penultimate layer of the pretrained LM Atgt as the inputs for training Asrc. Note that training Asrc in this way is equivalent to fixing the LM and only fine-tuning the last classification layer. To further remove the redundancy in
the representations and reduce computational cost, we follow prior work by performing PCA on the collection of representations and selecting the first 32 principal components (Ghorbani and Zou, 2019; Kwon and Zou, 2022; Schoch et al., 2022).
Sampling Data Shapley. Instead of directly estimating Shapley-based data values via Monte Carlo sampling on the whole training set, our approach performs Monte Carlo sampling on subsets of the data, which we refer to as sampling chains. Within a single sampling chain c, we sample a subset of training instances St, estimate their contributions, and repeat T times. The contribution of each instance in St is calculated by removing one instance at a time in a random order. For example, the contribution of the first randomly removed instance i is cSt(i) = vAsrc(St)− vAsrc(St\{i}), the contribution of the second randomly removed instance k is cSt(k) = vAsrc(St\{i}) − vAsrc(St\{i, k}), and so on. On the other hand, if an instance i is not in St, cSt(i) = 0.
After T times, the Shapley value of instance i is approximated as ϕi ≈ 1T ∑ St cSt(i). To balance the computational efficiency and approximation, we empirically define a range of the size |St| ∈ [ s2 , s], with subset size s as the sampling upper bound.
Computation can be further sped up with multiple Monte Carlo sampling chains S(c)t , c ∈ {1, . . . , J}. The corresponding value approximation is defined as ϕi = 1J ∑ c 1 T ∑ S (c) t c S (c) t
(i). As each chain can be computed independently, the efficiency can be boosted with parallel computing. This novel idea of multi-chain sampling serves as the core of TS-DSHAPLEY and significantly speeds up computation, in practice working with a simple model Asrc.

heading: Data Selection with TS-DSHAPLEY Values. To
text: identify harmful data points, we use the data removal strategy of Ghorbani and Zou (2019) on Asrc and transfer the selection outcome to the target model Atgt. Specifically, we gradually remove training instances from the lowest estimated contribution value to the highest estimated contribution value. Following each removal, we retrain Asrc and evaluate predictive performance on the heldout development data. As a result, this removal procedure will identify a optimal subset Sopt that gives the best predictive performance on Asrc. With the assumption of data value transferability (Schoch
et al., 2022), we expect that Atgt trained on Sopt will give no worse, and likely better performance, than Atgt trained on D. While this data removal strategy is proposed in prior work (Ghorbani and Zou, 2019), the data selection use case is novel in NLP.

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: Pre-trained Large Language Models. We utilize two transformer-based large LMs for which traditional Shapley-based data value computation would be intractable: RoBERTa-base (Liu et al., 2019, 125M parameters) and DistilBERT (Sanh et al., 2019, 66M parameters).
Datasets. We select one GLUE benchmark (Wang et al., 2019) dataset from each task category: SST-2 (Socher et al., 2013), QQP (Iyer et al., 2017), and RTE (Dagan et al., 2006), representing Single-Sentence Tasks, Similarity and Paraphrase Tasks, and Inference Tasks, respectively. Additional dataset details are reported in Appendix A. Notably, we select datasets of varied sizes to reflect diverse sampling subset to training set size ratios.
Data Selection Baselines. We compare against performance when training on the full data subset as well as three selection baselines: leave-one-out (LOO) (Cook, 1977), KNN-shapley (KNN) (Jia et al., 2019a, 2021), and random sampling. For LOO, we use the same classifier architecture as with TS-DSHAPLEY to compute value estimates. For both LOO and KNN, we reduce the dataset using the data removal procedure defined in section 3. Finally, for random sampling, we remove a random sample of data points equal to the number of points removed via TS-DSHAPLEY.

heading: 4.2 Data Selection Experiment
text: To test the efficacy of using TS-DSHAPLEY to select data for fine-tuning large LMs, we compute data values using each method and perform the data removal procedure described in section 3. Specifically, we remove the lowest value data points preceding the data removal step that achieved the highest development accuracy using Asrc. For TSDSHAPLEY, we vary the subset size and number of chains based on dataset size, using subset size = 6.7k(10%), 7.28k(2%), 374(15%) and number of chains = 25, 10, 25 for SST-2, QQP, and RTE,
respectively. Additional training and hyperparameter details, including details of a limited hyperparameter sweep, can be found in Appendix A.
Results Results are shown in Table 1. TSDSHAPLEY consistently outperforms baseline selection methods as well as performance using the full fine-tuning dataset. Notably, data selection using TS-DSHAPLEY resulted in performance improvements of up to 1.3% and 3.4% for RoBERTa and DistilBERT, respectively, over the predictive performance when training using the full fine-tuning dataset. These results indicate TSDSHAPLEY successfully identifies data points that harm model performance. As an additional analysis, for the RTE dataset we show the location of harmful points identified by TS-DSHAPLEY on a data map (Swayamdipta et al., 2020) in Appendix B.

heading: 4.3 Sampling Hyperparameter Analysis
text: TS-DSHAPLEY exhibited good performance for data selection across various subset sizes and numbers of chains. For example, on QQP TSDSHAPLEY outperformed the full dataset and baseline methods when using a subset of just 2% of the training set. To better understand the impact of different parameter values, we utilize a parameter value grid on the RTE dataset and re-compute TS-DSHAPLEY. Specifically, using the best hyperparameters from subsection 4.2 (see Appendix A), we evaluate performance of RoBERTa and DistilBERT using a parameter sweep of subset size as a percentage of the total training set size, subset size ∈ {1, 2, 5, 10, 15}%, and number of chains ∈ {2, 5, 10, 15} and report the Pearson’s correlation between each parameter and performance.
Results. All correlations are reported in Appendix B and summarized here. When subset
size > 2%, both models demonstrate a high positive correlation between number of chains and performance. For example, when using 15% of the training data, RoBERTa on RTE had a correlation of 0.94. Across the different number of chains, however, there was no consistent pattern of correlation between subset size and performance. This indicates that increasing number of chains (which can be computed in-parallel) may be of more benefit compared to increasing sampling subset size.

heading: 4.4 Effect of Different Embeddings
text: To test the efficacy of computing TS-DSHAPLEY using the extracted representations from the target LM, we perform an experiment where we use the removal indices computed with 1) the representation from a different language model (e.g. removing indices for fine-tuning RoBERTa using the optimal removal index identified using DistilBERT data representations), and 2) GloVe pre-trained word embeddings (Pennington et al., 2014), as a thirdparty representation repository.
Results. As shown in Table 2, while alternate embeddings can still lead to improvements over the full data, using the representation from the target LM is beneficial and consistently outperforms other embeddings. The results suggest that low value data is likely a combination of (i) inherently noisy
data (e.g. mislabeled instances) and (ii) instances that are harmful to specific models due to different model architectures and pre-training strategies.

heading: 5 Conclusion
text: In this work, we propose TS-DSHAPLEY to address the model and dataset constraints that currently contribute to a computational bottleneck when computing Shapley-based data value estimates.

heading: Limitations
text: While we demonstrate the efficacy of TSDSHAPLEY empirically, the current work is limited in terms of theoretical analysis. For example, while we have good empirical performance with a linear SVM, additional analysis could determine if there are optimal ways to select an alternative simple model architecture for the source classifier depending on the target classifier or dataset. Additionally, while we found a strong correlation between number of sampling chains and performance when the subset size was > 2% of the training data size, the lower subset size threshold to observe this correlation may be dataset dependent, which additional analysis could address.

heading: A Additional Experiment Details
text: In this section, we include additional experiment setup details.

heading: A.1 Datasets
text: Dataset statistics are provided in Table 3, with further description provided below.
SST-2: Stanford Sentiment Treebank (Socher et al., 2013) is a collection of English movie reviews with human annotations of their sentiment. The model is tasked with predicting a review’s sentiment as positive or negative.
QQP: Quora Question Pairs (Iyer et al., 2017) is a collection of English question pairs from the website Quora where the task is to determine if a pair of questions are similar in meaning.
RTE: Recognizing Textual Entailment (Dagan et al., 2006) combines several English datasets from annual textual entailment challenges, where the task is to predict if the text entails the hypothesis or not.

heading: A.2 Hyperparameters
text: For each experiment, we consider a limited hyperparameter sweep for each model, selection method, and task, with batch size ∈ {16, 32} and learning rate ∈ {10−5, 3 × 10−5}. The rest of the hyperparameters are kept consistent across experiment conditions. We report the mean development set accuracy from five random initializations for which we fine-tune for 10 epochs and select the model checkpoint with the highest development set accuracy. Results from each hyperparameter sweep are reported in Table 4 and Table 5.

heading: B Additional Results

heading: B.1 Additional Data Selection Analysis
text: While we compare directly with baseline selection methods that directly measure estimated data contribution, we perform an additional analysis by comparing the indices removed with TSDSHAPLEY with the mapped training dynamics using data maps (Swayamdipta et al., 2020). Specifically, we first plot the data map for RoBERTa trained on RTE using the same hyperparameters as in subsection 4.2. Then, we plot the same data map showing only the data points that were identified by TS-DSHAPLEY to be harmful, i.e. removed from
the fine-tuning training data. These are shown in Figure 2 and Figure 3, respectively.
We observe that a handful of instances in the hard-to-learn region (identified by Swayamdipta et al. (2020) to contain some mislabeled examples) were removed, as well as a small number of instances in the ambiguous region. Interestingly though, we observe that 1) most of the data points in RTE belonged to the easy-to-learn region, and 2) a cluster of easy-to-learn points were removed. Swayamdipta et al. (2020) found that too many easy-to-learn instances could decrease both in-distribution and out-of-distribution performance and noted that determining how to select an optimal balance of easy-to-learn and ambiguous examples, particularly in low data settings, was an open problem. As TS-DSHAPLEY achieved a performance gain over the full dataset performance, these results suggest that TS-DSHAPLEY may be effective to potentially determine an optimal balance and address this problem. We leave further analysis of this to future work.

heading: B.2 Sampling Hyperparameter Analysis.
text: Pearson’s correlation coefficients for the sampling parameter analysis in section 4 are reported in Table 6 and Table 7, where each result represents the mean of five sampling and chain computation trials.

heading: Future Work
text: We leave further analysis of this to future work