CLUES: Collaborative Private-domain High-quality Data Selection for LLMs via Training Dynamics
Authors: Wanru Zhao
Hongxiang Fan
Shell Xu Hu
Wangchunshu Zhou
Nicholas Donald Lane
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
data quality
model merging
federated learning
Primary area: safety_in_machine_learning

Abstract: Recent research has highlighted the importance of data quality in scaling large language models (LLMs). However, automated data quality control faces unique challenges in collaborative settings where sharing is not allowed directly between data silos. To tackle this issue, this paper proposes a novel data quality control technique based on the notion of data influence on the training dynamics of LLMs, that high quality data are more likely to have similar training dynamics to the anchor dataset. We then leverage the influence of the training dynamics to select high-quality data from different private domains, with centralized model updates on the server side in a collaborative training fashion by either model merging or federated learning. As for the data quality indicator, we compute the per-sample gradients with respect to the private data and the anchor dataset, and use the trace of the accumulated inner products as a measurement of data quality. In addition, we develop a quality control evaluation tailored for collaborative settings with heterogeneous medical domain data. Experiments show that training on the high-quality data selected by our method can often outperform other data selection methods for collaborative fine-tuning of LLMs, across diverse private domain datasets, in medical, multilingual and financial settings.

CLUES: Collaborative Private-domain High-quality Data Selection for LLMs via Training Dynamics
Wanru Zhao Hongxiang Fan Shell Xu Hu 3, Bofan Chen Nicholas Lane
University of Cambridge, UK Immeerial College London, UK
3 Samsung AI Center Cambridge Flowwer Labs
Abstract
Recent research has highlighted the importance of data quality in scaling large language models (LLMs). However, automated data quality control faces unique challenges in collaborative settings where sharing is not allowed directly between data silos. To tackle this issue, this paper proposes a novel data quality control technique based on the notion of data influence the training dynamics of LLMs, that high quality data are more likely to have similar training dynamics to the anchor dataset. We then leverage the influence of the training dynamics select high-quality data from different private domains, with centralized model updates the server side in collaborative training fashion by either model merging federated learning. As for the data quality indicator, we compute the per- sample gradients with respect the private data and the anchor dataset, and use the trace of the accumulated inner products as a measurement of data quality. In addition, we develop a quality control evaluation tailored for collaborative settings with heterogeneous medical domain data. Experiments show that training on the high- quality data selected by our method can often outperform other data selection methods for collaborative fine-tuning of LLMs, across diverse private domain datasets, in medical, multilingual and financial settings. Our code is released CLUES
Introduction
Large language models (LLMs) training has predominantly relied on the accumulation of vast datasets. Recent observations suggest that even a modest quantity of high-quality diverse data can significantly enhance the instruction following capacity of LLMs. Previously, data quality control relied heavily on manual selection processes [37 36 This approach, while being commonly used, rendered scalability challenges due to the substantial labor costs. Recent advancements have seen automated low-quality data filters 33, such as perplexity filters (29 and de-duplication filters [22].. However, their effectiveness in data quality control in more complex environments remains to be explored, where data are spread across silos and locations in different formats and difficult to find.
Collaborative training techniques, such as model merging [1] and federated learning 20 are common paradigms for addressing data-sharing constraints and GDPR (28) compliance. However, data quality control for private data is even more challenging if users are in charge of manually filtering data. We summarize here the two unique challenges: (1) Quality Heterogeneity Some clients may possess higher proportion of low-quality data compared to others, thus we should not select data from all clients with fixed selection ratio. (2) Domain Heterogeneity Different data silos may come from different vertical domains, for example, in the multilingual setting, different languages have different quality standards that are never unified.
Coorresponding Author: Wanru Zhao (wz3410cam.. ac uk)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we propose CLUES (collaborative learning under selection), an automated nugh-quality data selection method for collaborative fine-tuning of Large Language Models (LLMs), showcasing notable performance improvements in mixed-quality data environments from different private domain data. In these domains, private L.M vendors are supposed to build their specialized applications based on open-source L.LMs using their own private data, which represent specialized domains with significant private (e.g., patient records) and public data (c.g., scientific papers). By tracing the training dynamics of each training sample, we leverage public dataset to define an anchor dataset and compute the influence of each training sample on the anchor dataset, and set a global threshold to provide effective collaborative quality controls compared with traditional local data quality selection methods in the following aspects: (I) General: Our method is a general pipeline to improve the generalization performance for LLM fine-tuning It has interpretation in terms of bi-level optimization with inner optimization in the client side and outer optimization in the server side to minimize the loss on the anchor dataset. (2) Collaborative: Our method is collaborative fine-tuning paradigm that can be seamlessly integrated into existing model merging and federated learning frameworks, where the modification occurs the server side only to incorporate data selection. (3) Scalable: We only employ an approximation to solve the bi-level optimization, which makes it scalable to LLMs.
We evaluate our proposed method medical, multilingual and financial Question Answering (QA) datasets, demonstrating significant improvements of up to 67.3% on challenging medical and financial QA datasets, highlighting the effectiveness of our proposed method. Through extensive analyses, demonstrate the significant impact of leveraging training dynamics on the collaborative data quality control of LLMs.
2 Problem formulation: Collaborative Data Quality
2.1 Related Work
Collaborative LLM Fine-Tuning Paradigms: Model Merging and Federated Learning Col- laborative fine-tuning exhibits certain advantageous properties as a distributed machine learning paradigm by shifting the traditional model training process towards sharing model parameters instead of raw data. Participating clients train models using their own private datasets locally, and the updated model parameters are aggregated on the server. This preserves the privacy of the underlying data while collectively benefiting from the knowledge gained during the training process (20). We focus on merging fine-tuned models that are optimized from the same pre-trained backbone. Different fine-tuned models initialized from the same pre-trained model effectively share a part of the optimiza- tion trajectory and can often be merged without accounting for permutation symmetry [40 11 17..
Therefore, merging fine-tuned models can improve performance on a single target task 13 6
improve out-of-domain generalization
create multitask models from different tasks 23 and
other settings [23 4
One of the most significant challenges plaguing model merging and federated learning methods in previous research is the concern that the model parameters might interfere with each other during weighted averaging other merging operations. This undesirable interaction could potentially lead to merged model that performs worse than the individual models before merging. We argue that can be tackled from the perspective of data attribution.
Model merging, let fo F denote the language model and D3 E 'D denote the training dataset on client k. Given the training datasets Dr., we can define a model merging operator Mxx(; D;, k e K (1,... n.) : F - F. The model merging process can expressed as
fmerging = ~M (f)
Federated Averaging. Based the notation of model merging, the federated averaging process can be expressed as
T
fred = II (s,
1=1
where T' is the round number. S,K)) is the index set of clients participated in the training in round t.
Data Attribution and Selection for LLMSs The quality of the training data of a machine learning model can have significant impact on its performance. One measure of data quality is the notion
2

---

of valuation, i.e., the degree to which given training example affects the model and its predictive performance. Although data attribution is a well-known concept for researchers, the complexity behind large language models, coupled with their growing size, features, and datasets, has made quantification difficult. Recent methods include Perplexity Score, IFD [24] and DataInf [21], etc. More details are provided in Appendix [ However, those data attribution above have not been used in collaborative settings where each client has statistical heterogeneous and quality heterogencous private data. And previous data selection methods have not provide way to determine the golden threshold to decide whether a training data sample should be kept or filter out.
Training Dynamies Previous works [38 25 35 that analyze training dynamics focus primarily on supervised learning and are largely model- and data-agnostic. Swayamdipta et al. (34] empirically demonstrated the influence of data by visually mapping individual training samples according their impact on the correctness, confidence, and variability of a model.
2.2 Assumption and Objective: Collaborative High-quality Data Selection for LLMs
Definition 1.1 (Data Quality on Specific Domain k). Given a model architecture 0, training configuration (optimizer, etc.), and a validation set D in a specific domain the quality of training data 2 is defined as follows: for Z1, E Deroin. if Lear (0(2))())))))))) D < (o Deat), then the quality of 21 is considered higher than that of 22. Here, Leat denotes the validation loss. In other words, the lower the validation loss, the higher the data quality.
Training Loss
Loss
Definition 1.2 (Data Quality in Collaborative Private Domains), Given model architecture 0, a training configuration (optimizer, etc.), and a validation set val:
0
50
100
150
200
250
steps
Validation Loss
Deal = (()) ((2) .ua....., ((K) for all K tasks, the qual-
2.3
ity of training data is defined based on the validation loss of the global model 6 merrged on D Specifically,
2.2
for 21,22 22 D(A) Dirain if Leat (0mereee)) (21, Dvat) Leal (0 (Omerggd ged (Z2), paat), then the quality of z1 is con- sidered higher than that of 22. As in the single-domain case, lower validation loss indicates higher data quality.
2.1
2.0
1.9
1.8
Remarks 1 (Impact of Low Quality Data in Collabo- rative Private Domains). We manually construct low- quality data samples on each client. We change the pro-
50
100
150
200
250
ssstpps
100% low-quality data
80% low-quality data 20% high-quality data
portion of low-quality data from 0% to 100%. Higher
50% low quality data 50% high quality data
low-quality data 80% high-quality
scores indicate better performance, From Fig. [ larger portion of low-quality data results in higher validation loss, and more unstable and less effective training loss curve. Fig. 2 shows the performance drop when we change the proportion of low-quality data from 0% to 60%.
100% high-quality data
Figure 1: Validation loss and training loss.
Mistral-7b
Llama2-7b
0.45
0.424
0.36
0.343
0.4
0.11
0.25
0.3
0.14
0.35
0.3
0:205
0,311
lowar
0.2
0.165
0.154
0.25
0.17
0.15
0.085
0.08
0.16
0 0825
0.05
0.05
GPT-4 Scoring
Knowledge Avg
GPT-4 Scoring
Knowledge Avg
Raw Data
Low--uul Data
DRaw Data
Low--uual Data
Figure 2: Performance drop on the performance of collaborative fine-tuning of LLMs when we change the proportion of low-quality data from 0% to 60%. Higher scores indicate better performance.
3

---

Gloa Thredoad
Step One: Local Training for Data Quality Scoring Step Two:
Collaborative Training with High-Quality
Data
Figure 3: Overall workflow diagram consists of two phases: 1) Step One: client-side computes each sample's 's quality score with scoring functions using the public validation set and global model, then server-side calculates the score of global threshold by anchor data 2) Step Two: clients filter data according to the global threshold and starts collaborative learning on selected high-quality data with adaptive weights on the model side.
Remarks 2 (Enhancing Data Quality Collaborative Private Domains). In the collaborative learning framework, the ratio and distribution of low-quality data are unknown a priori. Only the server has access to the global distribution of both high-quality and low-quality data, while individual clients cannot infer the global distribution from their local distributions due to statistical heterogeneity. The server can infer the distribution of high-quality data from public anchor data. Our objective is to select data points that most significantly reduce the validation loss of the global model, rather than optimizing for each local model independently. It is important note that the scope of this study does not consider new models joining during training or continual learning paradigms.
3 Methodology: CLUES
3.1 Overview
In our workflow, each client performs local training using his own high-quality private data. We have a public validation set located on both the clients and the server, which consists of commonly recognized, high-quality public data. As illustrated in Figure 3 the overall workflow consists of two phases designed to achieve data quality control in the collaborative development of LLMs.
Step One. Local Training for Data Quality Scoring Local clients compute each sample's 's quality score via training dynamics-basee methods using the public validation and their own fine-tuned model. The server determines global threshold score, serving as a unified standard of data quality with only a very small amount of anchor data, and sends it the clients.
Step Two. Collaborative Learning with High-Quality Data Each client then discards data samples that fall below the global threshold received, ensuring that only high-quality data verified by the unified standard are retained. The clients then utilize the high-quality filtered data sets D'2 (where [P]] S (D&) ) and the initial global model 6 for collaborative learning. After local fine-tuning with the selected high-quality curation data, clients send their local LoRA adapter to the server. The server then aggregates the LoRA parameters of the individual models.
3.2 Step One: Training Dynamics- Data Scoring
The idea behind our method is straightforward - trace the training process to capture changes in prediction as individual training examples are visited.
For each client, we have designed a data scoring step to calculate the score for each training data sample to measure its contribution to model prediction. Specifically, considering the training set of examples D2 = { 2x} } and a model f, we represent the validation set as D = (=]...., 2. 2 We measure the performance of model using a loss function E: Rp x Z - IR. The loss of the model noted by 0 on an example z is given by E(O, 2). We fine-tune the model by finding parameters 0 that minimize the training loss ΣK e (0, 21), through iterative optimization procedure, such as

---

Stochastic Gradient Descent (SGD) or its variant, which utilizes one training example 2, in iteration t, updating the parameter from 0, 0--1:
0 - (e; (2;
(I)
We trace the training process capture changes in prediction individual training examples visited. The contribution of a particular training example 2 on a given test example 2' defined as the total reduction in loss on the test example 2' that is induced by the training process whenever the training example is utilized. We define the data quality of a particular training example z as the sum of the contribution of the whole validation dataset.
The simplified expression for data quality is as follows:
S(z) = ΣΣ T [+)=EE)
The per-sample gradients calculated for each training sample from the checkpoint t saved during the model training. LLMs are generally tuned using Adam W, which has a more complicated update formula involving the moving averages of the gradient moments.
For Adam,
m
0 --'
=-n - (2)
(3)
Vo++I +
For AdamW,
m
-' (z, (2,,0'))
+20*
(4)
Vet+1 +
Therefore, the training data quality score for LLMs is calculated using the following formula:
8(z) ΣΣ TL 0.) (z,
(5)
The dot product of the loss gradients of the training example (2) and the test example (2') is weighted by the learning rate (17) at different checkpoints and summed up. where we implemented applying point-wise loss gradients to disentangle the relative contributions of each training example, We use the output of the checkpoints from the learning algorithm to capture the training process. The higher score S(=), the higher the quality of the training sample z. We demonstrates an optimized training approach for collaborative learning of multiple models. By selecting high-que -uaaliuulyty training data for each local model, we select gradients that positively impact loss trajectories. These trimmed gradients accumulate, Icading to improved position in the weight space. Considering interference during our data selection (gradient selection) of A&, and A6, we reduce the interference of weight updates from different models. After parameter aggregation, the merged model menered be improved to an enhanced position in the weight space represented by Aftargeted.
It is particularly well-suited for parameter-efficient fine-tuning techniques such as Low-Rank Adap- tation (LoRA) [O], which involves freezing the pre-trained model weights and injecting trainable rank decomposition matrices into linear projects of the Transformer architecture. A neural network contains many dense layers that perform matrix multiplication. In the self-attention module, we denote the query projection matrices We the key projection matrices WR. the value projection matrices as W.. the output project matrices as W.. In principle, we can apply LoRA to any subset of weight matrices in neural network to reduce the number of trainable parameters. In the Transformer architecture, there are four weight matrices in the self-attention module (W g. Wa. W., Wo). In our implementation, we apply LoRA only to W3 and W. in most experiments for simplicity.
One straightforward solution to calculate the quality scores on weight parameters LoRA, but may be computationally infeasible when larger models with several millions of parameters are used. To address the memory bottleneck of calculating and saving gradients, we take gradients with respect to given layer. We propose operating the first layer of the model, which contains the
5

---

least cancelation effect, since the early layers encode unique logit. Therefore, we develop the idea of LoRA-based training-data influence in the context of gradient descent. Our proposed influence score is scalable due to the sparse nature of low-rank gradients and contains both low-level and high-level information since the gradient to the low-rank layer can capture both high-level and low-level information about the input sentence.
Note that the above gradient computation process is based on one single checkpoint and there is no parameter update throughout the process. Hence, for each training data point, we can perform this process in parallel, which can facilitate the computation.
3.3 Step Two: Global Standard with Anchor Data Scoring
On the server, we use small set of public data (10 samples in our paper) as our anchor data and calculate the average score of these 10 data points as the global threshold. This establishes a unified standard for division between low- and high-quality data for heterogeneous clients, allowing for the further filtering of local data.
Then merge the parameters of individual models with adaptive weights different models. For model merging techniques, we implemented Task Arithmetic [12] on task weights, the LoRA matrices involved in weighted sum. In task arithmetic, one first computes the task weights which is the difference between fine-tuned and base model weights, then calculates a weighted sum of these task weights. Here, the delta weights considered are the individual matrices A and B instead of their product BA. Consider two LoRA adapters (Ai, Bi) and (A2, B2) along with weights w and w2 for the weighted merging of these adapters, then the merging happens follows:
Amerged A1A1 W2242
(6)
B = /w B + V2
We also implement a more efficient method for merging LoRA adapters by eliminating redundant parameters: Trim, Elect, and Merge (TIES) (42). First, redundant parameters are trimmed, then conflicting signs are resolved into an aggregated vector, and finally, the parameters whose signs are the same as the aggregate sign are averaged. This method takes into account that some values (redundant and sign disagreement) can degrade performance in the merged model.
4 Experiments
Unlike traditional data quality selection methods for pre-trained models or traditional fine-tuning, in our collaborative setting, the training data from vertical domains is very sensitive and subject to strict restrictions regarding sharing and privacy. Therefore, we propose a new experimental setting using medical domain data for downstream tasks and evaluation for open-ended medical QA tasks, considering both quality heterogeneity and domain heterogeneity.
4.1 Experimental Setup
Tasks and Datasets
We conduct our evaluation on the open-ended
(QA) tasks.
(1) Medical QA: PMC-LLama 41 and Medssaacaaflashccarss 7 cover medical question-answeringg rationale for reasoning, and conversational dialogues, comprising a total of 202M tokens. We use 16k samples in total, with 8k samples randomly sampled from PMC-I_Laaa and Medalpaca- flashcards each. We uniformly partition the total samples into 20 clients in this task to demonstrate the effectiveness of CLUES in terms of the scalability of the clients, where the clients are IID subsets of the original distribution. For low-quality data, 3.2k samples (40% total data) are polluted with cutting, deletion, or substitution. These 40% low-quality data, together with the rest of the high-quality data, composites the mix-quality data set.
(2) Multilingual QA: MMedBench B3 is medical muti-choice dataset of 6 different languages. It contains 45k samples for the trainset and 8,518 samples for the testset. Each question is accompanied by right answer and high-quality rationale. We use 6312 samples randomly sampled from MMed- Bench and 1052 samples per language for each of the clients. For the low-quality data, a certain ratio is either polluted with random noisc.

---

(3) Financial QA: To demonstrate the generalizability of our proposed method across various domains, we also include FiQA [5], part of the training corpus of FinGPT [43], which consists 17.1k financial open Question-Answerinn the 4 clients from FiQA dataset, and pollute each of them with low-quality data ratio of 80%, 20%, 10%, and 50% respectively.
instructions. We randomly sample 2000 data samples for each of
Note that for all tasks, the anchor data and validation dataset used in our proposed method are selected as held-out high-quality dataset from the same data source.
Models We LLama2-7b 37 and Mistral-7b [1] as our pre-trained models, and fine-tune them with Low-Rank Adaptation (LoRA) [10] on each of the client side. As for the model merging technique, in our main experiments, we use TIES merging. We also compare it with Task Arithmetic in our ablation studies.
The Oracle shows the results that train only on the remaining high quality data in the mixed-quality dataset, serving as the theoretical upper bound. We implement the three baselines: existing methods mentioned in 2.1 Perplexity score, IFD 24 and DataInf 21 independently on each client.
Baselines
Evaluation metrics The evaluations focus on two main aspects: (1) Question-Answering capa- bilities, assessed by GPT-4 (30) scoring within the test set splited from the same sources of the training dataset. In the medical QA and multilingual QA tasks, 200 samples are randomly selected from the medical dataset to serve as the test set. We evaluate the models that need to be compared on the test set to generate responses respectively. Then we use the OpenAl GPT-4 model API to assign scores to their responses. Each response of is rated by the judge on scale from 1, reflecting how well the answer aligns with the ground truth. In our financial QA task, GP rate the responses of the fine-tuned model on our data set on scale of 10, reflecting criteria including relevance, precision and fluency. To address potential positional bias, we send our response along with the benchmark output to GPT-4 twice, with different orders. We then calculate the average of these scores as the final performance score. (2) Knowledge acquisition, measured by average accuracy of responses to multiple-choie questions in the MMLU clinical topics [8, 91. MedMCQA [31), PubMedQA [16), and USMLE [15] datasets. Although the goal of private domain fine-tuning is not to increase knowledge, there shouldn't be too much knowledge forgetting during this process. (3) Data selection correctness Precision, Recall, F-1 Score, and Accuracy are widely-used evaluation metrics that provide complementary insights into the model's effectiveness from the data selection perspective. In case, positive instances represent high-quality data, while negative instances rep- resent low-quality data. Precision quantifies the proportion of correctly identified positive instances among all instances predicted as positive, while Recall measures the proportion of correctly identified positive instances among all actual positive instances in the dataset. The Fl Score offers a balanced measure of Precision and Recall, while Accuracy reflects the overall correctness of our data selection (based on our data scoring and threshold determining method) across all classes.
4.2 Main Results
Based on the low-quality dataset setup, we evaluate our data-quality control pipeline in collaborative LLM fine-tuning in both federated (communication round er 300) and model merging (com- munication round er = 1) settings. Note that in federated learning, the server and clients need to intensively communicate the model updates during model training. We implement the three baseline methods described in the Section to calculate scores for each training data, and set the unified scoring standard using corresponding scoring functions with anchor data.
We demonstrate the performance of data quality control methods in collaborative settings in the medical QA task (Tab. I and Multilingual QA task (Tab. 2}
Federated Learning v.s. Model Merging Firstly, for both pre- pre-trained models and tasks, with other settings remaining the same, model merging performs better than federated learning. This indicates that loose communication between the local model and the server, compared to frequent communication, might lead to better generalization. Additionally, the performance boost with selected data in the federated setting is larger than the model merging setting. This might be because during federated learning, we calculate the data score based on the global model (instead of the local model
7

---

Table 1: Data selection performance in federated setting on MedicalQA. We bold the highest performance and underline the second highest performance for each row.
Mistral-7b
Llama2-7b
Evaluation Metric
GPT-4 Scoring
Knowledge Avg
GPT-4 Scoring
Knowledge Avg
Mix-qual Data
0.085
0.194
0.0952
0.311
Oracle
0.160
0.233
0.099
0.440
PPL
0.079
0.346
0.045
0,362
IFD 19
0.087
0.287
0.050
0.346
Datalnf
0.093
0.106
0.103
0.335
CLUES (ours)
0.161 (100.6%)
0.309 (132%)
0.210 (212. %)
0.356 (80.9%)
Table 2: Data selection performance on MMedBench. We bold the highest performance and underline the second highest performance for each row.
Mistral-7b
Llama2-7b
Setting
Federated
Model Merging
Federated
Model Merging
Mix-qual Data
0.420
0.515
0.440
0.485
Oracle
0.451
0.530
0.449
0.490
CLUES (ours)
0.435 (96.5%)
0.525 (99. %)
0.477(100..2%)
0.487 (99.4%)
in the model merging setting) at each timestamp, which can better trace and regularize the training trajectory to the optimal location.
Data Selection Performance In both federated and model merging settings, our data selection achieve over 96% and over 91 % the theoretical upper bound performance, respectively. Our method outperforms the other local data selection basclines under the GPT4 Scoring metries. Compared to the other methods which cause severe forgetting during instruction tuning, the performance of our method the Knowledge-baat -aaee benchmark remains within acceptable range. This shows that our methods are able to improve domain-specinf tasks without forgetting knowledge injected during pretraining.
Analysis
5.1 Qualitative Analysis
We performed qualitative analysis manually comparing the outputs generated by models fine- tuned on our selected high-quality data versus the original low-quality data. This comparison (Tab. 7 and Tab. g provides insights into the tangible improvements in model performance and output quality.
5.2 Varying Levels of Low-Quality Data
To evaluate the robustness of our data selection method under different data quality conditions, conducted series of experiments with varying proportions of low-quality data. We maintained a consistent proportion of low-quality data across all clients for each experiment, ranging from 0% to 100%, including pollution levels 20%, 50%, and 80%.
Fig. presents the performance of models trained with and without our data selection method across these different proportions. The results demonstrate that our method effectively enhances data quality across all scenarios with GPT-4 scoring. And in terms of accuracy of the data selection, our method consistently selected over 99% of the high-quality data across different proportions low-quality data. Additionally, to understand the adaptability of our global threshold, we analyzed how the global threshold changes with different proportions of low-quality data. Fig. Fillustrates that our global threshold adjusts across varying levels of data quality.

---

100.00
0.0040
99.8375%
175
99.73
0.0033
99.7250%
0.0035
14735
0.0030
29.3125%
3.25
(8)
0.0025
hold
19955
99.00
0.0020
0:0017
2.75
96.75
0.0015
0.0012
0.0010
225
0.0005
98.00
0.0000
Thend wio selection
mixed-oa qualty data
... 0% bad sata (raw data) (3.74001
CLUES
.... 100% tao data (2.2860)
Accuracy
Global
(a) GPT-4 Scoring Performance in different propor- tions of low-quality data.
(b) Selection accuracy and global threshold (scorc) in different proportion of low-quality data.
Figure 4: Experimental results for different levels of low-quality data
5.3 Quality Heterogeneity
To provide more comprehensive analysis, in addition to the experiments in the domain heterogeneity setting shown above, we conducted additional experiments in a quality heterogeneity setting using the FiQA dataset, which focuses on the answer of financial questions. Specifically, we randomly polluted 80%, 20%, 10%, and 50% of the training set for each of the four clients, respectively. The findings demonstrate that our method significantly enhances the quality of the data even when clients have different proportions of low-quality data.
Varying Merging Techniques Fig. demon- strates that different weighted merging or aggre- gation techniques lead to varying performance. Notably, performance of our data selection method with the Linear Merging technique does not even reach the performance of low-quality data with TIES Merging technique, highlighting the significant impact of weighted merging tech- niques overall performance. Furthermore, experimented with different merging techniques on the FiQA dataset, demonstrating the impor- tance of weighted merging, shown in Fig. 5
90. 60.6
3.4%
Dika selection accurscy
3,45
3375
01.0%%
2.m
3709
Branc QA
Rubingual Q4
Rist Rast Last Lay
Figure 5: Left: Comparison of different merging techniques. Right: First layer v.s. last layer for low-rank tracing gradient.
Layer Selection for Low-Rank Tracing Gra- dient In terms of layer selection, we evaluated both the last layer and the token embeddings. We show that layer selection distorts the score (the inner product of two gradients). In our ablation study, we observe that since the activation connected the last layer of weights contains shared logic. the data influenced calculated through the last layer weights are prone to a cancellation effect, where the data influence of different examples has large magnitude that contradicts each other. The cancelation effect lowers the power of the influence score, and deleting influential examples according this measure often does not change the model's behavior by much. From Fig. [] show that the first layer has less severe cancelation effect than the last layer.
Unified Scoring with Anchor Data We conducted an ablation study on our global threshold to further validate our approach. Tab. iillustrates the advantage of using a global threshold determined by our anchor data for data selection in this heterogeneous setting, compared selection based on average ratio or pre-determined scores. These results demonstrate that our approach successfully balances the identification of positive cases with the minimization of false positives, offering a robust and superior solution.

---

Table 3: Data selection performance on FiQA. We bold the highest performance for each row.
Precision
Recall
F1 Score
Accuracy
Select by ratio
79.17%
79.17%
79.17%
75.00%
Select by pre-determined score
92.77%
99.13%
95.84%
95.00%
Select by global threshold (Ours)
97.44%
99.38%
98.39%
97.91%
Discussion and Conclusion
Collaborative model development, including model merging and federated averaging, would benefit from different kinds of high-quality data, and for each of them, the definition of quality is slightly different. In this paper, we establish a data quality control pipeline for collaborative fine-tuning of LLMs, avoiding directly sharing any private data. Our experiments show that the selected high-quality data ensures an effective and reliable learning process, leading to improved model performance.
To the best of our knowledge, we are the first to propose data selection method for large language models collaborative setting, while previous work has mainly focused on traditional centralized settings. We bring up the insights to view federated learning and model merging within the same framework, incorporate different experimental setups and unify federated learning and model merging methods, making it universally applicable. Additionally, our method performs well on generation datasets and takes into account scenarios with bad data, while previous work has not considered downstream domain-specife generation tasks for large language models. Our method does not require repeated training.
Societal impact Our work builds large language models that make it possible to create collab- orative instead of a monolithic ecosystem from open-source models while preserving the privacy of users' own data. The constant progress being made in machine learning needs to extend across borders we are to democratize ML in developing countries. Adapting state-of-the-an (SOTA) methods to resource-constraine environments such developing countries can be challenging in practice, pushing open source and inclusion.
Limitations and future work Our data quality control methods are based on the assumption that all the local models share the same model architectures. is easy to achieve when our fine-tuning is based on the LoRA adapter. However, it may be worth extending it to adapt different local model architectures, for example, different low ranks. Future work may explore the intrinsic relation between data selection and the model parameters and how our data selection methods can help reduce the interference of parameter vectors from different models.
Acknowledgement
The authors would like thank Colin Raffel, Haokun Liu, Meghdad Kurmanji and Stefanos Laskaridis for useful discussions and feedback. This research was supported by the European Research Council via the REDIAL project and the Royal Academy of Engineering via the DANTE.
10

---

References
[1] Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble averages: Im- proving model selection and boosting performance in domain generalization. arXiv preprint arXiv:2/10. 10832, 2021.
[2] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[3] Together Computer. Redpajama: an open dataset for training large language models, 2023.
[4] Shachar Don- Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen, Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.01378, 2022.
[5] FinGPT. fingpt-fiqa_qe http:///wwgaaaaccccc//c/eeeaee/a 2023. Accessed: [Insert Access Date Here].
[6] Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste.
Stochastic weight averaging in parallel: Large-batch training that generalizes well. arXiv preprint arXiv:2001.023/2 2020.
[7] Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. Medalpaca- open-source collection of medical conversational ai models and training data. arXiv preprint vrXiv:2304.08244 2023.
[8] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[9] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021.
[10] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, 00442100.09668 2021.
[11] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212. 12.04889 2022.
[12] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL id=6t0kwf8-
[13] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407. 2018.
[14] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[15] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
[16] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint 1rXiv:1909.0614 2019.
[17] Xisen Jin, Xiang Ren, Daniel Preotiuc- Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint 2rXiv:2212.098-99 2022.
11

---

[18] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[19] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894. PMLR, 2017.
[20] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency, arXiv preprint arXiv: 2016.
[21] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Ilms and diffusion models. arXiv preprint arXiv:2310.00900 2023.
[22] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint 1e.Xiv22107.000 2021.
[23] Margaret Li, Suchin Gururangan, Tim Dettmners, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge:: Embarrassingly parallel training of expert language models. arXiv preprint arXiv:2208.03336 2022.
[24] Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self- guided data selection for instruction tuning. ArXiv, abs/2308. 12032, 2023.
[25] Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. Advances in neural information processing systems, 30, 2017.
[26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[27] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Ben- jamin Bossan. Peft: State-of-the-arr parameter-efficentt fine-tuning methods, 2022.
[28] Christopher F. Mondschein and Cosimo Monda. The EU's General Data Protection Regulation (GDPR) in a Research Context, pages 55-71. Springer International Publishing, Cham, 2019. ISBN 978-3-319-99713--1 doi: 10. (1007/978-3-319-997733
[29] Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint arXiv:2305. 16264, 2023.
[30] OpenAL, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Sernadett-Shapine Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux. Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao. Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan
12

---

Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak. Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pan- tuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard. Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky. Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
[31] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248-260. PMLR, 2022.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detooo, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[33] Pengeheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards building multilingual language model for medicine, 2024.
[34] Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 9275-9293, Online, November 2020. Association for Computational Linguistics. doi: 18.1...3/ wat5533////2220.
[35] Yuandong Tian, Tina Jiang, Qucheng Gong, and Ari Morcos. Luck matters: Understanding training dynamics of deep relu networks. arXiv preprint arXiv: J 3405, 2019.
[36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas
13

---

Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[38] Charles L. Wilson, James L. Blue, and Omid M. Omidvar. Training dynamics and neural network performance. Neural Networks, 10(5):007-923, 1997. ISSN 0893-6080. doi: https: //doi.org//0.. 101
1tbbi.org/10.1016/08889999 00000//////000000000000000000000
[39] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v 1e120000.mmlppdemmmms...
[40] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965-23999 PMLR, 2022.
[41] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pme-llama: Further finetuning Ilama on medical papers. arXiv preprint 1xXx:22344.1444, 2023.
[42] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Teeg-nerging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Informa- tion Processing Systems, 2023.
[43] Hongyang Yang, Xiao Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.00 0600310000060000000 2023.
14

---

A Data Attribution
Perplexity (PPL) serves fundamental metric in language modeling measure the model's ability to predict text sequences accurately. Mathematically, it is defined as the exponential of the average negative log- oogliikelhood: PPL = exp(- Ei=1 log P(w:|w1, .... w;-1)), where N is the number of tokens the sequence and (( W1: w;-1) represents the probability the model assigns to token w; given its preceding context. A lower perplexity score indicates better model performance, as suggests that the model assigns higher probabilities to the correct tokens the sequence, effectively measuring how "surprised" the model is by new text.
Instruction Following Difficulty (IFD) 244 provides a quantitative metric for evaluating the difficulty of instruction-following tasks in language models. This score is calculated as the ratio between two key measurements: the Conditioned Answer Score se( (AlQ) and the Direct Answer Score sp(A): sa(AlQ) where so (AlQ) measures the model's ability to generate responses with Aa(
IQD A)
instructional context, and (A) evaluates the model's capability to generate answers in isolation. N Σ log P(w;+ / 1;=1;;)
The Direct Answer Score computed as so(A)
0). This metric quantifies the extent to which instructions aid in response generation, where a higher IFD score suggests that the given instruction provides limited useful context for the model's response generation, indicating greater difficulty in following the instruction.
Influence Functions 179]: DataInf represents an efficient algorithm for computing in- fluence functions, distinguished by its closed-form expression that reduces computa- tional and memory complexity compared existing methods. The algorithm approx- imates the inverse Hessian calculation (Gi(6*) + \pdd)) through the key transforma-
V,,VV,, where the Sherman- Morrison formula enables a closed-form solution. The influence function is computed as Damtia M1a) = Et= di En Las Arthas
tion: + E (Vo of + ArTa;) e Nh E (Ia, A,+ Vo. Vo.t.
Lt,i -1,)),,))).
B Priliminary Results on Low-quality Data
Table 4: Preliminary results on MedicalQA.
Mistral-7b
Llama2-7b
Evaluation Metric
GPT-4 Scoring
KnowledgeAvg
GPT-4 Scoring
Knowledge Avg
Raw Data
0.165 0.085 (148.5%)
0.343
0.265
0.424
Mix-qual Data
0.194 (143.4%)
0.0925 (165.1%)
0.311 (126.7%)
Table 5: Preliminary results on MMedBench.
Mistral-7b
Llama2-7b
Setting
Federated
Model Merging
Federated
Model Merging
Raw Data
0.455
0.540 0.515 (16.48%)
0.450
0.505 0.485 (43.96%)
Mix-qual Data
0.420 (17.69%)
0.440 (42.22%)
C Detailed Method Description
Stage (On each client) Local fine-tuning with low-quality data; save model checkpoints. Stage (On each client) Calculate gradients, compute scores for each training sample. send scores to the server.
Stage 3 (On the server) Calculate gradients, compute scores of anchor data, determine the global threshold using anchor data scores and client scores.
Stage (On each client) Select data with scores not lower than the global threshold. Stage 5 (On each client) Local fine-tuning with high-quality data, then send model parameters to the server.
Stage 6 (On the server) Merge client model parameters to obtain the final global model.
15

---

Algorithm 1 Our data selection method for collaborative fine-tuning
Initialization
Initial global model: 09; Training datasets (private): Drvain (D(L) D(2) D(A) D(R) train j.
train train
train
= =(2 (R)
2n ):
Validation datasets (public): Deal {D 222)
val ................................ D( 1
f. val { (()) ....!
1()
km J
Anchor data (public): Danc (Danc, Danc. Dune f. D (=) (21(1) 3..., -(k)
On the Client k
Train model 0g on train
(k)
Training with Mixed Quality data
for each training sample 2, E DD
train do
Calculate V (=',0,) for each checkpoint in T do
for each validation sample z/ E D do Calculate (= 0,) end for end for S(z) = E.' Et F1: V vl(z,
Data Scoring
end for (r(k) troin
-fz0 2 ((0 (ramn 2 2 r}
b Select training data with scores above the threshold b Training with High--Qualtyy Data
Train model 8% on (r(k)
train
Send updated B/ to server
On the Server
for each anchor data 2; E Di!A) do
Calculate V (z. 0,)
for each checkpoint in 7 do
for each validation sample 2f (*)) Calculate end for end for S(z) -T= TA. (2 0.) V (z, 0:) end for
vat do
D Anchor data score
Determine the global threshold r with anchor data Danc Send each clients for training with High-quality data Aggregate client updates: 6'
D2
0%
Model Merging or Aggregation
P..
D Complexity of Data Scoring
The overall compute complexity, where N is number of checkpoints, and d is gradient dimension.
O (N- | ld)
The overall storage complexity:
O(D d Doatl N.
E Implementation Details
We use Hugging Face's transformers library and PEFT library 227 for loading pre-trained models and LoRA configurations. For our training and evaluation, we use PyTorch as the underlying
auto-differentiation framework [32]. We use the Adam optimizer [26 18] for experiments.
All the experiments conducted on the machines with the same hardware configuration using
16

---

Bad data
Good data
English
Mismatched Answer
English
0,7748348255407
Score: 4.7146087:1199226
linstruction] Given your background doctor, please provide your insight addreessing medical quesiions based the patien's
[inetrutioon] Given your background as doctor, please provide your insight in addressing medical questions based the pasieut's
linput) A 52 year old man presents to his peimary case physiciam for generalized pain. Which of the following associated with this patlent's condition?. A proteuts Hypercalcemia D. Destractive joundice [outpui] Hearing loss. Choesing option A. beazing loss, may seem counserintu nive given the patient's primary complaint of generalized pain. However, the notable tspect of the case the slevated alkaline phosghatase (ALP) level while other liver function (AST. ALT) and lipase nurmal. suggesing source the likeyy not liver or pancreas.
past medical history ansicty approprianely treated schizophrenia present to the emergency degartment for listt time seizure.... Which of the following is the most likely explanation of this patient's Increased water consunattion P450 induction Sub-teenpetic dose
[input] year old patient
Bence
presentation? A. Elevated blood levels medication B.
[outpnt] Throm/scemoolis.. The patient's history of recest sungery for finctured
femur. coupled pasied immobility during rebab, increases the risk of deep venous thrombossi (DVT).
Chinese
Incomplete Answer
Chinese
Score: 0.2890661177115702
Score: 4422751445703947
[ustmuction] 36 1
12--0 WiB0 R2
[instruction] de: 1192 DE A
[espur] BB 12 Ht $1,
T A.
800-180mmHHD R.
(E: TF 22 11228
[inpus] 41 [L 2:.8:
50- 70mmH20
[outpu]] 80 180-18mm0020
100l (CSP) IEW 2 3
230 200002500mm20000
&
260 3002200mH200
2200 180mmH20
WERE
1-JARt 10:98 15 R/R
12:01 (+0
MUB. 1211 #F B. RRHAA
(ERDE T2500 250mm02200000
14550-70mm1120
16. R/
280mmH300)
a. Ci
[ounput]
NOA RJ (I
Spanish
Mixed Languages
Spunish
Scure: 0673086533114344
Score: -0.67 1996666441144
instruction] Dado su experiencia como médicu, pur favor, peopercione su opinión abordar las preguntas médicas basándose relato del pacieare. [input] (Cuil
[itatruction] Dado experiencia camo médico, por favor, proparciose su opinián
siguientes matrentes requirre atenciko recomendación
para abordar las preguntas médicas busindos en relato del pacienic.
mis especial en la edad avancads.. A. Selemio. B. Calio. C. Aminuócidos esenciales, D. Lipados. E. Fibca.
[inyus) Los corpisculos de Pacini. A. Eatán incrvados per filma amislinicas.
Son mecamorssaptees adaptación
Presemia campos receptores
[ouspuc] ósco. que se deterionn con fracturas aumentan debido una disminacinn de la densadad deca. que sucle ser particllamente notable en mujeres posmenoppusicas debido la dismin- ución de estrégeass. Ademis, absorción calcio decreee is los requerimients son fäcilmente satisfechos critico asegurarse de que los adultos mayores consuman suliciente calcio, ya sen tavés de la alimentación mediante suplamentos, para mantener hncsos furstes y prevenir fracturas.
elecciin del calcio (B) debe avanaaa. La ostesporosis y riesgo de
fundamental para la salad
poqueñoss Se localizan on zons profundas de demiss Son temorrocep- Inres.
[outpui] Knerıs,, coxpanиsmиe oxcrmecoooorooers nocne supvrrennmmi wxcrAa is pazsurиe memaaoreseeos uponecoon, Taa Bak nomposs,ee- miee IHH, mayurpponamce restyeMmee,, uриnоgиr i ma- anorrosy, remposy nossomnnny onsorenesy.
sysenoro nopa socnee, Moryr
dieta. cllo,
Table 6: Low-quality and high-quality data samples with scores.
NVIDIA A40 and A6000. In the experiments of Medical QA, we consider 300 communication rounds, n 20 clients, 10 local steps of model training. In the experiments of Multilingual Medical QA, we distrubute 6 languages on 6 clients, with each clients 1052 dataset. In all experiments, we use 8 bit quantization, set batch_size have same rank), and we do 3 epochs for local training.
16, and LoRA rank 16 (all the participating LoRA adapters
F Additional Experimental Results
17

---

Questions
Ground Truth Response
Model
Re- data
Model Response (w/ our data selection)
sponse
(w/
selection)
23
D #
t M KA T P J B H # #L BÜST
FR
10
A (
AS B
#
X
X
A -
# By
77cm. 7
A E ta fit AM
H
E66mm/
1
*1 [i]
E A. * バ.. UE it II th AI
27cm
#
#
RI
* +1,
N1:
A T *े
B1 (IEM) T
C.
1400K13
16
$1408
:B 1T X4 7
# #1 2 R 07
7g
X MM A TH N A6 A - H
1T 16 XII
8K
A. 3E
TH (N) 9N 6R
# B. N9 Bi
3 E.. #F BE 4:
- * C.
#1 AÜKE A DE
A LIBA D. TR
- 1 ith
101.
11 #*
1K AP -
0T.
M NM H
م. H0 17 75
z JJJ 1TT # XL
T A
What type of injury to the arm/elbow most often leads to supracondylar fractures?
Supracondylar fractures most ten occur after hyperextension in- juries of the arm/elbow.
1000 Hg is the pres- sure required to rup- ture an average aortic aortic
Supracondylar frac- tures are type of fracture that occurs above the elbow joint, and they are most com- monly caused by fall onto an outstretched arm.
Table 7: Samples of the output of merged models on multilingual MMedBench dataset.
Questions
Model Response (w/o data lection)
Model Response (w/ our data selection)
How does a year treasury note work?
It is a 2 year treasury note.
Treasury notes are short-term government security that matures in two years. They are issued by the U.S. Treasury Department to help inance the nation's 's deicit.
What causes a stock to drop in price?
Investors are afraid to buy stocks because they are afraid the stock will drop in price.
stock price is falling, it's because there's 's a lot of supply, but not enough de- mand. If there's 's a lot of demand, but not enough supply, the price will risc.
Table 8: Samples of the output of merged models on FiQA dataset.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA). The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it
18

---

(after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes]" " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] " "provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found. IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section 3 and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
19

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: See Section Hnd Appendix 6
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they (c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification:
See the submitted artifacts.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
cablic/guidess Btaaai//uuids//ooouuubnnsssssss "ooeeSumisssoonoo
oniioiiyolyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ prapss.cc/ublli/guiddee Seepuidddde/oodSdbiii SooodeSummissionlee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See the bar plots in Section
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: See Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ccttta///eearrrpp..cppppl//////e
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
22

---

Justification:
We have provided the statement of societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper proposes approach for data selection, and does not pose high risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification; This paper is licensed under CC-BY-NC-SA 4.0. All other codes, datasets, and references are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We did not have any released new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not involve crowdsoureing research with human subjects.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "y9SpV5xXeD", "reviewer": "Reviewer_Xfnf", "summary": "This paper proposes a data quality control technique for the collaborative training of large language models from filtered private heterogeneous data domains via a quality score function that tracks the gradient of each training sample. The proposed framework is tested in medical and multilingual settings to demonstrate its effectiveness.", "questions": "- Why use a global threshold as the unified standard of data quality? How were the anchor data selected? Would the usage of anchor data (validation set) introduce an additional bias toward the overall data quality (some datasets could potentially be all excluded) and harm the generalization ability? It would be better to see the performance comparison between global and local thresholds.\n- How is the per-sample gradient computed from Eq. (1) line 172? Could the authors provide a complexity analysis on this part?\n- Line 196, “sparse nature of low-rank gradients” refers to the trainable weight matrices being low rank or the obtained per-sample gradient being sparse and low-rank.\n- It would be great to show the ablation study of directly applying the proposed methods to raw data (supposed high-quality data), given the federated results reported in Table 2.\n- Could the authors provide several examples of low-quality data samples as presented in Table 4?", "limitations": "See W1-2, Q1-2, 4", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper studies an important issue in private data quality control for collaborative model development, especially for the applications of LLMs.\n- The proposed gradient-based score function is straightforward and effective.", "weaknesses": "- The assumption of homogeneity for model architecture, usage of low-rank adaptation, and anchor data with global threshold limit the overall applicability of the proposed framework.\n- The experimental settings/details and the ablation study are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1**\n\n> The assumption of homogeneity for model architecture\n\nWe clarify that we follow the commonly agreed assumption of previous work on collaborative learning (federated learning, model merging) [1, 2, 3]: in order to aggregate the model in the parameter space, we should have the same model architecture or the same adapter. It's impossible to perform model aggregation if the models have different architectures. Exploring different model architectures may be beneficial for other applications but is not the focus of our paper; for example, multimodality.\n\n> Usage of low-rank adaptation\n\nWe wanted to clarify that the setup and motivation for using collaborative fine-tuning is to gather knowledge in an efficient manner. We focus on situations where individual clients don't have enough data or compute resources to pre-train a large language model or even perform full parameter fine-tuning. Low-rank adaptation is a commonly used parameter-efficient fine-tuning method for such scenarios. Therefore, the usage of low-rank adaptation is not a limitation but a way to achieve broader applicability. It is also regarded as a strength from other reviewers' perspectives: ngAe and igLM.\n\n**W2**\n\n> The experimental settings/details and the ablation study are limited.\n\nWe have provided more ablation studies and experimental details in the uploaded PDF. We would really appreciate hearing your feedback regarding the limitations in more detail!\n\n-------\n**Q1**\n\n> Global threshold as the unified standard, versus local thresholds:\n\nIt is necessary to have a global threshold to deal with the heterogeneity of data quality. Since the ratio of local low-quality data is unknown, and the ratios for each client are different from each other, local thresholds are not able to adapt to different and diverse low-quality data ratio scenarios. We show the comparison between global and local thresholds in Figure 1 of our PDF. There, the local optimal threshold represents the optimal thresholds for each client and is unknown in practice. The global threshold can be close to the local optimal, while local thresholds are far from the optimal thresholds.\n\n> Anchor data selection and biases\n\nThe anchor data is selected as a held-out high-quality dataset from the same data source as the test data. We show the ablation study of the anchor data selection and validation set selection in Figure 2 in our uploaded PDF, which demonstrates the robustness and unbiased selection in our experiments. In Figure 2 (Left), the global threshold remains stable and robust with different numbers of anchor data selected. In Figure 2 (Right), the box plot shows the data scores for 20 training samples randomly sampled from our training dataset over 5 different validation sets of equal size. Although the scores vary depending on the validation set used, the variance is within an acceptable range, ensuring that the order of their scores remains stable and robust. Thus, bias is not a concerning factor in our paper. We acknowledge that, in other cases, validation set selection will significantly affect the data selection process, and this is a common issue for all data selection work using validation sets.\n\n\n**Q2**\n\nPer-sample gradients are calculated for each training sample from the checkpoint saved during the model training. \n\nFor SGD,\n$\\boldsymbol{\\theta}^{t+1}-\\boldsymbol{\\theta}^t=-\\eta_t \\nabla \\ell\\left(\\boldsymbol{z} ; \\boldsymbol{\\theta}^t\\right)$\n\nFor Adam, $\\boldsymbol{\\theta}^{t+1}-\\boldsymbol{\\theta}^t=-\\eta_t \\mathcal{L}\\left(\\boldsymbol{z}, \\boldsymbol{\\theta}^t\\right) $,  $\\mathcal{L}\\left(\\boldsymbol{z}, \\boldsymbol{\\theta}^t\\right) \\triangleq \\frac{\\boldsymbol{m}^{t+1}}{\\sqrt{\\boldsymbol{v}^{t+1}}+\\epsilon} $\n\nFor AdamW, $\\boldsymbol{\\theta}^{t+1}-\\boldsymbol{\\theta}^t=-\\eta_t \\mathcal{L} \\left(\\boldsymbol{z}, \\boldsymbol{\\theta}^t\\right) $, $\\mathcal{L}\\left(\\boldsymbol{z}, \\boldsymbol{\\theta}^t\\right) \\triangleq \\frac{\\boldsymbol{m}^{t+1}}{\\sqrt{\\boldsymbol{v}^{t+1}}+\\epsilon}{+\\lambda \\boldsymbol{\\theta}^t}$\n\n- overall compute complexity, where $N$ is number of checkpoints, and $d$ is gradient dimension.\n$$\n\\mathcal{O}\\left(N \\cdot |\\mathcal{D}| \\cdot\\left|\\mathcal{D}_{\\text {val }}\\right| \\cdot d\\right)\n$$\n- overall storage complexity:\n$$\n\\mathcal{O}(|\\mathcal{D}| \\cdot N \\cdot d + |D_{val}| \\cdot N \\cdot d)\n$$\n\n**Q3**\n\nWe use LoRA to reduce the number of trainable parameters, which freezes the pre-trained weights and adds a low-rank adaptor to linear layers throughout the network. This means that, by nature, the trainable matrices are low rank. While our method is based on LoRA, it would also be very easy to adapt our proposed method to training where per-sample gradients are sparse and low-rank, for example, GaLore [4]. Our method is orthogonal to different training methods, and we believe this is a very promising direction for another future work.\n\n**Q4**\n\n> ablation study of directly applying the proposed methods to raw data (supposed high-quality data).\n\nAs we clarified in our global response, raw data is not one of the baselines we are comparing our data selection method with, since the scope of our paper is to conduct data selection to filter out the low-quality data mixed in with the high-quality data. However, we agree it would be interesting to see how our methods perform on the dataset without low-quality data. We are currently running these experiments and will update the results once we have them.\n\n**Q5**\n> Examples of low-quality data samples \n\nSure, we have provided the low-quality and high-quality data samples with their scores in our PDF for your kind review.\n\n[1] TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.\n\n[2] Editing Models with Task Arithmetic, ICLR 2023. \n\n[3] Communication-Efficient Learning of Deep Networks From Decentralized Data, Artificial Intelligence and Statistics, 2017.\n\n[4] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L5byn8ymYv", "reviewer": "Reviewer_igLM", "summary": "This paper proposes a novel approach for data quality control in collaborative fine-tuning of large language models (LLMs), particularly in settings where data cannot be directly shared between different silos due to privacy concerns. The authors introduce a method that scores training samples based on tracing gradients of low-rank adapters, filters out low-quality data locally, and then adaptively merges model parameters. They evaluate their approach on medical and multilingual question-answering datasets, demonstrating significant performance improvements compared to baselines.", "questions": "Please refer to weaknesses part", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors have presented their ideas in a well-organized and logical structure, making the paper easy to follow.\n2. The paper addresses an important problem in collaborative LLM training, where data quality control is challenging due to privacy constraints. The proposed method shows promising results and could have significant implications for improving the performance of collaboratively trained language models across various clients.\n3. The method leverages training dynamics, providing a more fine-grained and comprehensive approach to data quality measurement compared to traditional methods which only looked at the last model checkpoint.\n4. The use of LoRA and layer selection in the method of tracing gradients cleverly makes it particularly well-suited for large language models, addressing the computational challenges often associated with training such models.", "weaknesses": "1. Although the methodology is thoroughly described, the authors could improve the clarity of Figure 3 by providing a more intuitive explanation of how training dynamics enhance collaborative training processes.\n2. The paper contains some grammatical errors and typos. For example, in Table 3, \"Performance of Data quality\" should be \"Performance of data quality\", and on line 290, \"the data influenced calculated via the last layer weights prone to a cancellation effect\" should be corrected to \"the data influence calculated via the last layer weights is prone to a cancellation effect\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and positive feedback!\n\n**W1 ans Q1**:\n\n> Although the methodology is thoroughly described, the authors could improve the clarity of Figure 3 by providing a more intuitive explanation of how training dynamics enhance collaborative training processes.\n\nFigure 3 demonstrates an optimized training approach for collaborative learning of multiple models. By selecting high-quality training data for each local model, we are selecting gradients that positively impact loss trajectories. These trimmed gradients are accumulated, leading to an improved position in the weight space. By considering the interference during our data selection (gradient selection) of $\\Delta\\theta'{1}$ and $\\Delta\\theta'{2}$, we reduce the interference of weight updates from different models. After parameter aggregation, the merged model $\\Delta\\theta_{merged}$ can be improved to an enhanced position in the weight space represented by $\\Delta\\theta_{targeted}$. \n\n**W2 and Q2**:\n\n> The paper contains some grammatical errors and typos. For example, in Table 3, \"Performance of Data quality\" should be \"Performance of data quality\", and on line 290, \"the data influenced calculated via the last layer weights prone to a cancellation effect\" should be corrected to \"the data influence calculated via the last layer weights is prone to a cancellation effect\".\n\nWe appreciate your detailed suggestions on the typos! We have fixed them all and will update them in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5dRHjfAlvX", "reviewer": "Reviewer_ngAe", "summary": "This work proposes a method for finetuning LLMs in federated or collaborative training scenarios that selects the most informative examples for each client to train on such that their local parameter updates are likely to improve the global, merged model's performance on a public (shared) test set. The proposed two phase solution first scores local samples by computing the inner products between a gradient computed on a training sample and a gradient computed on a limited number of shared test samples, and only performs the next episode of local training on points that are above a specific quality threshold determined and distributed by the server in the previous round. Then, the server combines the parameter updates sent from each client, and using the new global parameters, iterates over the same shared anchor set to compute a new quality threshold to send back to the clients. By iterating this approach, in some settings they achieve performance improvements on medical domain tasks as compared to existing data valuation methods and baselines of training on curated subsets or all available data, without federation.", "questions": "- It seems like the table and figure reference numbering might be off in a few places, please check the \\ref commands.\n\n- I would like to better understand the dataset construction process. There are 3 raw sources listed, but what is the exact sampling used to build the training dataset? How are the n=20 client local datasets selected (are they uniformly sampled from the full training or grouped in some way?) Also, critically, how are the test evaluation samples (the GPT4 evaluation 200) sampled, and what is the relation to the _anchor_ data? Are the anchor data held out Medalpaca, just like the test set?\n\n- If the above description is somewhat correct, followup question is what are the statistics for which samples were selected in each round by each client as \"above the quality threshold\"... were these locally filtered subsets more often than not the medalpaca training questions? (and when running the other baseline scores, was this not true?) \n\nMy current hypothesis is that this grad x grad product is a similarity score between the local training data and the global test/anchor samples. Thus, it biases each client to train on the test-like data, and not the other training data.", "limitations": "- The primary limitation is in the fact that the method does not improve over the baselines or other scores in many settings evaluated. In Table 2 and 3, while \"Ours\" beats the low quality baseline, it doesn't beat the oracle or raw baselines often. \n- Similarly in Table 1, \"Ours\" only beats the other quality criteria in the GPT4 based evaluation. Depending on the clarification about the construction of the anchor set, I worry that the method only works well when the test set and the anchor set are closely related which limits the real world applicability of the method.\n\nI would be happy to engage with the authors to help improve the explication of the method and the presentation of the results, but my concerns with the strength of the empiricals are not insignificant. Clarifications would need to demonstrate a misunderstanding on my part of the evaluation setup or the results themselves in order for my score to improve.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The diagrams are well done and help the reader understand the method in an intuitive manner.\n\n- The method is principled, and relatively simple to implement requiring only gradients (rather than higher order computations), it is grounded well in prior works on data valuation and active learning, and is amenable to efficient implementations when using adapter training methods as demonstrated by their LoRA based experiments.\n\n- The ablations chosen are interesting, particularly the effect of layer selection for the gradient comparisons.", "weaknesses": "- The precise ordering of the algorithmic steps is not clear. Section 3.1-3.3 would greatly benefit from an Algorithm definition that clearly defines the order of operations regarding what gradients are computed on what data using what version of the parameters throughout the training process. It is hard to interpret the experimental results due to the lack of clarity in the section that introduces them.\n\n- Unless the reviewer is misunderstanding something about the evaluation, Table 1, 2, and 3 are all misleadingly bolded. The row label \"Ours\" is sufficient to tell that the row indicates the proposed method, however, \"Ours\" is bolded in all three tables, and all columns, regardless of whether or not the method outperforms other valuation criteria, or baseline training settings. On a quick glance this suggests the method offers uniform improvement in all settings, but this is not the case. Please only bold the best score in each column and note the convention in the captions.\n\n- The ablations (while appropriately chosen) are not convincing. The differences reported in 4.3 are likely not statistically significant, and in 4.4, the \"hit rate\" is not defined, which hurts because this part is of technical interest. Section 4.5 is confusing and the final sentence claims that the anchor score leads to the lowest proportion of low quality data, but this is exactly the opposite of Fig 4(c)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your valuable comments and feedback.\n\n**W1**:\n> Detailed algorithm definition\n\nThank you for your valuable suggestions! Following the reviewer's constructive feedback, we have included our pseudo-code algorithm in the PDF. \n\n**W2 & L1**:\n\n> Evaluation\nWe appreciate your questions on the bolded numbers! We have fixed them all in our uploaded PDF.\n\nTo further clarify our evaluation setup and clear up misunderstandings:\nOur preliminary results (Table 1 and Table 2 in the uploaded PDF) show severe performance degradation when mixing our constructed bad data into the raw data. This demonstrates the large impact of low-quality data in collaborative settings, which motivates the importance and pressing challenges for data selection methods in such settings. Our objective is to achieve better performance compared to the low-quality baseline, not the raw data performance. Oracle serves as the theoretical upper bound.\n\nTo further highlight our empirical results:\n1. For both pre-trained models and tasks, with other settings remaining the same, model merging performs better than federated learning. This indicates that loose communication between the local model and the server, compared to frequent communication, might lead to better generalization.\n2. The performance boost with selected data in the federated setting is larger than in the model merging setting. This might be because during federated learning, we calculate the data score based on the global model (instead of the local model in the model merging setting) at each timestamp, which can better trace and regularize the training trajectory to the optimal location.\n3. In both federated and model merging settings, our data selection can achieve over 96% and over 91% of the theoretical upper bound performance, respectively.\n4. Our method outperforms the other centralized data selection baselines under the GPT4 Scoring metrics. Compared to the other methods which cause severe forgetting during instruction tuning, the performance of our method on the Knowledge-based benchmark remains within an acceptable range. This shows that our methods are able to improve domain-specific tasks without forgetting knowledge injected during pretraining.\n\n**Q1**: We appreciate your detailed suggestions on the table and figure reference numbering! We have fixed them all and will update in our final version. \n\n**Q2**:\nWe split the whole dataset into the training set, the validation set (and anchor data), and the test set. They are all from the same data sources. The anchor data/validation set is in the same distribution as the training data, which is essential to ensure that the measurement of high-quality data aligns both during the selection (validation set and anchor set) and in the evaluation (test set).\n\n> Dataset construction process:\n\nFor the Medical QA task: We use 16k samples in total, with 8k samples randomly sampled from PMC-LLama and Medalpaca-flashcards each. We uniformly partition the total samples into 20 clients in this task.\nFor the Multilingual QA task: We use 6312 samples randomly sampled from MMedBench and 1052 samples per language for each of 6 clients. (Domain Heterogeneity setting)\nFor our newly added Financial QA task: We randomly sampled 2000 data samples for each of 4 clients from the FiQA dataset, and polluted each of them with low-quality data ratios of 80%, 20%, 10%, and 50% respectively. (Quality Heterogeneity setting)\n\nWe provided examples of raw data and low-quality data with their scores calculated by our proposed method in our uploaded PDF.\n> Test set:\n\nTest data are sampled from the same data sources. For open-ended GPT4 evaluation, the test samples are randomly sampled from Medalpaca-flashcards and MMedBench for the medical QA task and multilingual QA task, respectively.\nFor knowledge-based evaluation, the test samples are multiple-choice tasks, which use accuracy as the main metric. This additional evaluation is to see if fine-tuning will degrade or forget the knowledge obtained from pretraining. Following convention, we adopt three prominent medical benchmarks for evaluation.\n> Anchor data and validation data:\n\nWe followed the convention of existing data selection/valuation methods, which requires a clean validation set drawn from the target distribution. This setup has been adopted and proven to work well in many previous data selection methods, including Data Shapley and Influence Function, etc.[1] It doesn't limit the real-world applicability of the method, because there are always a few public high-quality data available; otherwise, we couldn't clearly define what high-quality data looks like.\n\n[1] LAVA: Data Valuation without Pre-Specified Learning Algorithms, ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sj0Pml1Abt", "reviewer": "Reviewer_V7za", "summary": "The paper proposes a novel data quality control technique to enhance the quality of data from different private domains in collaborative training settings, such as model merging. The technique scores each training sample by tracing gradients of low-rank adapters and filters out low-quality data locally. The model parameters are then adaptively merged based on individual training. The method is evaluated on heterogeneous medical domain data, demonstrating that training on high-quality data selected by this method outperforms other data selection methods across diverse private domain datasets. The paper claims performance improvements and offers a general, efficient way to identify high-quality data for large language models.", "questions": "Q1. Please explain how the proposed method performs on datasets from different domains beyond the medical field. Have you considered conducting experiments with datasets from various other domains to validate the generalizability of your approach? (w1)\n\n2. Please clarify if 40% is representative of real-world scenarios. Would you consider testing with different proportions of low-quality data to assess the robustness and effectiveness of your method under varying conditions? (w2)\n\n3. Have you tested the impact of using different numbers of anchor samples on the performance and reliability of the method?(w3)\n\n4. Please explain why our method performs better than Oracle and why the other three data evaluation methods perform so poorly in Table 1. (w4)", "limitations": "1. The author's work is similar to federated learning. Should it be compared with the recent federated learning?", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "s1. The use of training dynamics to score and filter data for collaborative fine-tuning is a good idea.\n\ns2. The method provides a general pipeline applicable to various models without the need for task-specific adjustments.\n\ns3. The structure of the article is complete and easy to understand", "weaknesses": "w1. The paper's evaluation is confined to medical datasets. To demonstrate the generalizability and robustness of the proposed method, experiments should be conducted on a wider range of datasets from different domains.\n\nw2. The paper generates 40% of the data as low-quality, which may not accurately reflect real-world scenarios. It would be beneficial to experiment with varying proportions of low-quality data to understand the method's effectiveness across different levels of data quality.\n\nw3. The method uses only a small amount of public data (10 samples in the paper) as anchor data to calculate the global threshold. The reliability of selecting just 10 samples is questionable, and the basis for this choice should be clarified. A more rigorous justification or experimentation with different numbers of anchor samples could strengthen the method's credibility.\n\nw4. Table 1 in the paper shows that the proposed method performs better than the Oracle method, which uses only high-quality data. The paper does not provide an analysis or explanation for this superior performance. Understanding and explaining why this method outperforms the Oracle method could provide valuable insights and strengthen the overall argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 & Q1**:\n> Please explain how the proposed method performs on datasets from different domains beyond the medical field. \n\nAs mentioned in s1, our method by nature can be adapted to various models ‘without the need for task-specific adjustments’. We conduct comprehensive evaluations on different medical and healthcare dataset which covers different domains in medicine. Our assumptions are not limited to any specific domain, and our selection of anchor data is not domain-specific. We chose medical data because of the availability of high-quality public datasets in this field.\n\nWe are running the experiments on the financial dataset: [FiQA](https://huggingface.co/datasets/LLukas22/fiqa). We randomly sampled 2000 data samples for each of the 4 clients from FiQA dataset, and polluted each of them with a low-quality data ratio of 80%, 20%, 10%, and 50% respectively. Once we get the result, we’ll update our response. \n\n**W2 & Q2**\n> Experiment with varying proportions of low-quality data to understand the method's effectiveness across different levels of data quality. \nWe have been running the experiments on different proportions of low-quality data: 80% and 100%. Once we get the result, we’ll update our response. \n\nWe reported the estimated running time of our collaborative fine-tuning pipeline (for each proportion of low-quality data on a certain dataset on A40). Due to the time limits, we are still running the experiments and expected to have additional experimental results within the discussion period. \n\n|Step\t|Estimated Time|\n|----|-----|\n|Collaborative finetuning with low-quality data|\t9 hours|\n|Calculate gradients|\t3 hours|\n|Compute scores and select|5 hours|\n|Collaborative finetuning with selected data|\t9 hours|\n|Prediction on test set|\t8 hours|\n|GPT-4 evaluation|\t0.5 hour|\n\n**W3 & Q3**\n> Impact of using different numbers of anchor samples on the performance and reliability of the method\n\nWe show the ablation study of the anchor data selection and validation set selection in Figure 2 in our PDF, which demonstrates the robustness and unbiased selection in our experiments. Fig 2 shows that the threshold keeps stable with the increase of the number of anchor data, thus demonstrating the robustness of the anchor data selection.\n\nI would like to also highlight that the number of anchor data will not affect the order of the selected high-quality data, the only impact is the number of data selected. \n\n**W4 & Q4**\n> Why our method performs better than Oracle?\n\nIn our experimental setup, the global threshold is $0.48$, and the total number of selected data is slightly more than the number of oracle data, and considering more data diversity potentially lead to better performance. We also discussed it in our future work part in our paper. \n\n> Why the other three data evaluation methods perform so poorly in Table 1?\n\nIFD assumes that all data in the dataset is of good quality where it selects informative data samples. It cannot effectively handle scenarios with poor-quality data, where IFD tends to select complex, difficult-to-learn noise patterns. PPL tends to select longer data samples. Both IFD and PPL are not well-equipped to deal with situations where malicious or intentionally corrupted data is present in the dataset. The dataInf method is only related to the optimal theta point and does not consider whether the validation loss actually decreases. This may result in selecting bad data that could increase the loss. In contrast, our approach can better handle settings with bad data. \n\nAdditionally, DataInf makes several strong assumptions, which lead to more limitations such as:\n1. the total approximation error is bounded by $O\\left(\\sum_{l=1}^L d_l^2\\right)$, and the approximation error is tolerable only when $d$ is small.\n2. The effectiveness of adapting DataInf to LoRA is limited to the number of learnable parameters is small, such as 1, 2, 4. The correlation coefficient of DataInf generally decreases as the rank increases.\n\nThese limitations highlight the challenges in developing influence measurement functions and data selection, especially when dealing with diverse, heterogeneous data, whereas our proposed methods are good at broader scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback and appreciate the great efforts made by all reviewers, ACs, SACs, and PCs. We are grateful that the reviewers have multiple positive impressions of our work, including: 
* *[Motivation]* **studies an important issue** (Xfnf, igLM) 
* *[Method]* **straightforward and effective** (Xfnf) and **principled, and relatively simple to implement requiring only gradients (rather than higher order computations)** (ngAe) and **provides a general pipeline applicable to various models without the need for task-specific adjustments**(V7za)
* *[Results]* **promising results and could have significant implications** (igLM)
* *[Presentation]* **diagrams are well done and help the reader understand the method in an intuitive manner** (ngAe), **presented their ideas in a well-organized and logical structure, making the paper easy to follow** (igLM), and **the structure of the article is complete and easy to understand**(V7za). 


Firstly, we would like to restate and clarify our assumptions for data selection in collaborative fine-tuning to help audiences better understand the scope of our work:
* We maintain the same setup as other previous works. To enable collaborative fine-tuning, each model shares the same architecture or the same LoRA adapter to adapt to different target tasks. To improve the efficiency of assigning utilities to each data point, we only calculate gradients for the parameter-efficient QLoRA modules. 
* Evaluation Setup: For all methods, we use the same setup following previous literature. Our raw training set, hold-out validation set, and test set are IID. For the experimental setup, they are from the same data source. For the Medical QA task, we split the MedAlpaca dataset into raw training set, hold-out validation set, and test set. For the multilingual task, we split the MMedBench dataset into raw training set, validation set, and test set. For the low-quality training set, we pollute the raw training set by mixing data from other domains into it; more details are in the PDF. In our paper, we focus on in-domain evaluation and leave out-of-domain or cross-domain (where training set and test set are from different data sources) for future work.

Secondly, we wanted to highlight our contributions drawn from our method design and experimental results:
* To the best of our knowledge, we are the first to propose a data selection method for large language models in a collaborative setting, while previous work has mainly focused on traditional centralized settings. We bring up the insights to view federated learning and model merging within the same framework, incorporate different experimental setups and unify federated learning and model merging methods, making it universally applicable. 
* Our method performs well on generation datasets and takes into account scenarios with bad data, while previous work has not considered downstream domain-specific generation tasks for large language models. Our method does not require repeated training.

We once again express our gratitude to all reviewers for their time and effort devoted to evaluating our work. We eagerly anticipate your further responses and hope for a favorable consideration of our revised manuscript.

Meta-review: The paper proposes a data selection method for large language models in a collaborative fine-tuning setting. The authors present a framework that unifies federated learning and model merging approaches, applicable without task-specific adjustments. 

The proposed method scores training samples by tracing gradients, filters out low-quality data locally, and adaptively merges model parameters offering potential benefits over naive averaging as per FedAvg, model soups and BTM. The authors evaluate their approach on medical and multilingual question-answering datasets, demonstrating performance improvements compared to established baselines.

Reviews are mixed but leaning positive. Most negative aspects have been addressed in the response. Some reviewers noted that the precise ordering of the algorithmic steps was not clear in the original submission. The authors have responded by including pseudo-code in the rebuttal supplemental materialks. There were also concerns regarding the presentation of results, particularly the bolding of values in tables, which may mislead readers into thinking that the method uniformly outperforms all baselines. The authors have provided justification in their rebuttal which should also be added in the final manuscript.

The method’s effectiveness seems to hinge on the relationship between the anchor set and the test set, raising concerns about its generalisability to other settings. The authors have clarified their dataset construction and evaluation processes, but the method’s applicability in more diverse, real-world scenarios remains a potential limitation.
Decision: Accept (poster)