CLUES: Collaborative Private-domain High-quality Data Selection for LLMs via Training Dynamics

Authors: wanru  zhao
hongxiang  fan
shell xu hu
bofan  chen
nicholas d lane
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:38:25

Abstract: Recent research has highlighted the importance of data quality in scaling large language models (LLMs). However, automated data quality control faces unique challenges in collaborative settings where sharing is not allowed directly between data silos. To tackle this issue, this paper proposes a novel data quality control technique based on the notion of data influence on the training dynamics of LLMs, that high quality data are more likely to have similar training dynamics to the anchor dataset. We then leverage the influence of the training dynamics to select high-quality data from different private domains, with centralized model updates on the server side in a collaborative training fashion by either model merging or federated learning. As for the data quality indicator, we compute the per-sample gradients with respect to the private data and the anchor dataset, and use the trace of the accumulated inner products as a measurement of data quality. In addition, we develop a quality control evaluation tailored for collaborative settings with heterogeneous medical domain data. Experiments show that training on the high-quality data selected by our method can often outperform other data selection methods for collaborative fine-tuning of LLMs, across diverse private domain datasets, in medical, multilingual and financial settings.

Main paper: 
 introduction: 
 Large language models (LLMs) training has predominantly relied on the accumulation of vast datasets. Recent observations suggest that even a modest quantity of high-quality diverse data can significantly enhance the instruction following capacity of LLMs. Previously, data quality control relied heavily on manual selection processes [37, 36] . This approach, while being commonly used, rendered scalability challenges due to the substantial labor costs. Recent advancements have seen automated low-quality data filters [3] , such as perplexity filters [29] and de-duplication filters [22] . However, their effectiveness in data quality control in more complex environments remains to be explored, where data are spread across silos and locations in different formats and difficult to find.Collaborative training techniques, such as model merging [11] and federated learning [20] , are common paradigms for addressing data-sharing constraints and GDPR [28] compliance. However, data quality control for private data is even more challenging if users are in charge of manually filtering data. We summarize here the two unique challenges: (1) Quality Heterogeneity Some clients may possess a higher proportion of low-quality data compared to others, thus we should not select data from all clients with a fixed selection ratio. (2) Domain Heterogeneity Different data silos may come from different vertical domains, for example, in the multilingual setting, different languages have different quality standards that are never unified.In this paper, we propose CLUES (collaborative learning under selection), an automated high-quality data selection method for collaborative fine-tuning of Large Language Models (LLMs), showcasing notable performance improvements in mixed-quality data environments from different private domain data. In these domains, private LLM vendors are supposed to build their specialized applications based on open-source LLMs using their own private data, which represent specialized domains with significant private (e.g., patient records) and public data (e.g., scientific papers). By tracing the training dynamics of each training sample, we leverage public dataset to define an anchor dataset and compute the influence of each training sample on the anchor dataset, and set a global threshold to provide effective collaborative quality controls compared with traditional local data quality selection methods in the following aspects: (1) General: Our method is a general pipeline to improve the generalization performance for LLM fine-tuning. It has an interpretation in terms of bi-level optimization with inner optimization in the client side and outer optimization in the server side to minimize the loss on the anchor dataset. (2) Collaborative: Our method is a collaborative fine-tuning paradigm that can be seamlessly integrated into existing model merging and federated learning frameworks, where the modification occurs on the server side only to incorporate data selection. (3) Scalable: We only employ an approximation to solve the bi-level optimization, which makes it scalable to LLMs.We evaluate our proposed method on medical, multilingual and financial Question Answering (QA) datasets, demonstrating significant improvements of up to 67.3% on challenging medical and financial QA datasets, highlighting the effectiveness of our proposed method. Through extensive analyses, we demonstrate the significant impact of leveraging training dynamics on the collaborative data quality control of LLMs.2 Problem formulation: Collaborative Data QualityQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: See Section 3 and Section 4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Collaborative LLM Fine-Tuning Paradigms: Model Merging and Federated Learning Collaborative fine-tuning exhibits certain advantageous properties as a distributed machine learning paradigm by shifting the traditional model training process towards sharing model parameters instead of raw data. Participating clients train models using their own private datasets locally, and the updated model parameters are aggregated on the server. This preserves the privacy of the underlying data while collectively benefiting from the knowledge gained during the training process [20] . We focus on merging fine-tuned models that are optimized from the same pre-trained backbone. Different fine-tuned models initialized from the same pre-trained model effectively share a part of the optimization trajectory and can often be merged without accounting for permutation symmetry [40, 11, 17] . Therefore, merging fine-tuned models can improve performance on a single target task [13, 6] , improve out-of-domain generalization [2, 1] , create multitask models from different tasks [23] , and other settings [23, 4] .One of the most significant challenges plaguing model merging and federated learning methods in previous research is the concern that the model parameters might interfere with each other during weighted averaging or other merging operations. This undesirable interaction could potentially lead to a merged model that performs worse than the individual models before merging. We argue that it can be tackled from the perspective of data attribution. K (•; D k , k ∈ K = {1, • • • , n}) : F → F.The model merging process can be expressed asf merging = M K (f )Federated Averaging. Based on the notation of model merging, the federated averaging process can be expressed asf f ed = ( T t=1 M St(K) )(f )where T is the round number. S t (K) is the index set of clients participated in the training in round t.The quality of the training data of a machine learning model can have a significant impact on its performance. One measure of data quality is the notion of valuation, i.e., the degree to which a given training example affects the model and its predictive performance. Although data attribution is a well-known concept for researchers, the complexity behind large language models, coupled with their growing size, features, and datasets, has made quantification difficult. Recent methods include Perplexity Score, IFD [24] , and DataInf [21] , etc. More details are provided in Appendix E. However, those data attribution above have not been used in collaborative settings where each client has statistical heterogeneous and quality heterogeneous private-domain data. And previous data selection methods have not provide a way to determine the golden threshold to decide whether a training data sample should be kept or filter out. Training Dynamics Previous works [38, 25, 35] that analyze training dynamics focus primarily on supervised learning and are largely model-and data-agnostic. Swayamdipta et al. [34] empirically demonstrated the influence of data by visually mapping individual training samples according to their impact on the correctness, confidence, and variability of a model. : for z 1 , z 2 ∈ D train , if L val (θ(z 1 ), D val ) < L val (θ(z 2 ), D val ), then the quality of z 1 is considered higher than that of z 2 . Here, L val denotes the validation loss. In other words, the lower the validation loss, the higher the data quality. train , if L val (θ merged (z 1 ), D val ) < L val (θ merged (z 2 ), D val ), then the quality of z 1 is considered higher than that of z 2 . As in the single-domain case, lower validation loss indicates higher data quality.Remarks 1 (Impact of Low Quality Data in Collaborative Private Domains). We manually construct lowquality data samples on each client. We change the proportion of low-quality data from 0% to 100%. Higher scores indicate better performance. From Fig. 1 , a larger portion of low-quality data results in higher validation loss, and more unstable and less effective training loss curve. Fig. 2 shows the performance drop when we change the proportion of low-quality data from 0% to 60%. Raw Data Low-Qual Data 0.17 3 Methodology: CLUESQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Appendix 6. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 In our workflow, each client performs local training using his own high-quality private data. We have a public validation set located on both the clients and the server, which consists of commonly recognized, high-quality public data. As illustrated in Figure 3 , the overall workflow consists of two phases designed to achieve data quality control in the collaborative development of LLMs.Step One. Local Training for Data Quality Scoring Local clients compute each sample's quality score via our training dynamics-based methods using the public validation set and their own fine-tuned model. The server determines a global threshold score, serving as a unified standard of data quality with only a very small amount of anchor data, and sends it to the clients.Step Two. Collaborative Learning with High-Quality Data Each client then discards data samples that fall below the global threshold received, ensuring that only high-quality data verified by the unified standard are retained. The clients then utilize the high-quality filtered data sets D ′ k (where|D ′ k | ≤ |D k |)and the initial global model θ 0 for collaborative learning. After local fine-tuning with the selected high-quality curation data, clients send their local LoRA adapter to the server. The server then aggregates the LoRA parameters of the individual models.The idea behind our method is straightforward -trace the training process to capture changes in prediction as individual training examples are visited.For each client, we have designed a data scoring step to calculate the score for each training data sample to measure its contribution to model prediction. Specifically, considering the training set of examples D k = {z 1 , . . . , z K } and a model θ, we represent the validation set as D ′ k = {z ′ 1 , . . . , z ′ K }. We measure the performance of a model using a loss function ℓ : R p × Z → R. The loss of the model noted by θ on an example z is given by ℓ(θ, z). We fine-tune the model by finding parameters θ that minimize the training loss K i=1 ℓ (θ, z i ), through an iterative optimization procedure, such as Stochastic Gradient Descent (SGD) or its variant, which utilizes one training example z t in iteration t, updating the parameter from θ t to θ t+1 :θ t+1 -θ t = -η t ∇ℓ z; θ t (1)We trace the training process to capture changes in prediction as individual training examples are visited. The contribution of a particular training example z on a given test example z ′ is defined as the total reduction in loss on the test example z ′ that is induced by the training process whenever the training example z is utilized. We define the data quality of a particular training example z as the sum of the contribution of the whole validation dataset.The simplified expression for data quality is as follows:EQUATIONThe per-sample gradients are calculated for each training sample from the checkpoint t saved during the model training. LLMs are generally tuned using AdamW, which has a more complicated update formula involving the moving averages of the gradient moments.For Adam,EQUATIONFor AdamW,EQUATIONTherefore, the training data quality score for LLMs is calculated using the following formula:EQUATIONThe dot product of the loss gradients of the training example (z) and the test example (z ′ ) is weighted by the learning rate (η i ) at different checkpoints and summed up, where we implemented applying point-wise loss gradients to disentangle the relative contributions of each training example. We use the output of the checkpoints from the learning algorithm to capture the training process. The higher the score S(z), the higher the quality of the training sample z. We demonstrates an optimized training approach for collaborative learning of multiple models. By selecting high-quality training data for each local model, we select gradients that positively impact loss trajectories. These trimmed gradients accumulate, leading to an improved position in the weight space. Considering interference during our data selection (gradient selection) of ∆θ ′ 1 and ∆θ ′ 2 , we reduce the interference of weight updates from different models. After parameter aggregation, the merged model ∆θ merged can be improved to an enhanced position in the weight space represented by ∆θ targeted .It is particularly well-suited for parameter-efficient fine-tuning techniques such as Low-Rank Adaptation (LoRA) [10] , which involves freezing the pre-trained model weights and injecting trainable rank decomposition matrices into linear projects of the Transformer architecture. A neural network contains many dense layers that perform matrix multiplication. In the self-attention module, we denote the query projection matrices as W q , the key projection matrices as W k , the value projection matrices as W v , the output project matrices as W o . In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters. In the Transformer architecture, there are four weight matrices in the self-attention module (W q , W k , W v , W o ). In our implementation, we apply LoRA only to W q and W v in most experiments for simplicity.One straightforward solution is to calculate the quality scores on all weight parameters of LoRA, but may be computationally infeasible when larger models with several millions of parameters are used. To address the memory bottleneck of calculating and saving gradients, we take gradients with respect to a given layer. We propose operating on the first layer of the model, which contains the least cancelation effect, since the early layers encode unique logit. Therefore, we develop the idea of LoRA-based training-data influence in the context of gradient descent. Our proposed influence score is scalable due to the sparse nature of low-rank gradients and contains both low-level and high-level information since the gradient to the low-rank layer can capture both high-level and low-level information about the input sentence. Note that the above gradient computation process is based on one single checkpoint and there is no parameter update throughout the process. Hence, for each training data point, we can perform this process in parallel, which can facilitate the computation.On the server, we use a small set of public data (10 samples in our paper) as our anchor data and calculate the average score of these 10 data points as the global threshold. This establishes a unified standard for division between low-and high-quality data for heterogeneous clients, allowing for the further filtering of local data.Then we merge the parameters of individual models with adaptive weights on different models. For model merging techniques, we implemented Task Arithmetic [12] on task weights, the LoRA matrices are involved in weighted sum. In task arithmetic, one first computes the task weights which is the difference between fine-tuned and base model weights, then calculates a weighted sum of these task weights. Here, the delta weights considered are the individual matrices A and B instead of their product BA. Consider two LoRA adapters (A 1 , B 1 ) and (A 2 , B 2 ) along with weights w 1 and w 2 for the weighted merging of these two adapters, then the merging happens as follows:EQUATIONEQUATIONWe also implement a more efficient method for merging LoRA adapters by eliminating redundant parameters: TrIm, Elect, and Merge (TIES) [42] . First, redundant parameters are trimmed, then conflicting signs are resolved into an aggregated vector, and finally, the parameters whose signs are the same as the aggregate sign are averaged. This method takes into account that some values (redundant and sign disagreement) can degrade performance in the merged model.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not have any theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 Unlike traditional data quality selection methods for pre-trained models or traditional fine-tuning, in our collaborative setting, the training data from vertical domains is very sensitive and subject to strict restrictions regarding sharing and privacy. Therefore, we propose a new experimental setting using medical domain data for downstream tasks and evaluation for open-ended medical QA tasks, considering both quality heterogeneity and domain heterogeneity.Tasks and Datasets We conduct our evaluation on the open-ended question-answering (QA) tasks.(1) Medical QA: PMC-LLama [41] and Medalpaca-flashcards [7] cover medical question-answering, rationale for reasoning, and conversational dialogues, comprising a total of 202M tokens. We use 16k samples in total, with 8k samples randomly sampled from PMC-LLama and Medalpacaflashcards each. We uniformly partition the total samples into 20 clients in this task to demonstrate the effectiveness of CLUES in terms of the scalability of the clients, where the clients are IID subsets of the original distribution. For low-quality data, 3.2k samples (40% total data) are polluted with cutting, deletion, or substitution. These 40% low-quality data, together with the rest of the high-quality data, composites the mix-quality data set.(2) Multilingual QA: MMedBench [33] is a medical muti-choice dataset of 6 different languages. It contains 45k samples for the trainset and 8,518 samples for the testset. Each question is accompanied by a right answer and high-quality rationale. We use 6312 samples randomly sampled from MMed-Bench and 1052 samples per language for each of the 6 clients. For the low-quality data, a certain ratio is either polluted with random noise.(3) Financial QA: To demonstrate the generalizability of our proposed method across various domains, we also include FiQA [5] , part of the training corpus of FinGPT [43] , which consists 17.1k financial open Question-Answering instructions. We randomly sample 2000 data samples for each of the 4 clients from FiQA dataset, and pollute each of them with a low-quality data ratio of 80%, 20%, 10%, and 50% respectively.Note that for all tasks, the anchor data and validation dataset used in our proposed method are selected as a held-out high-quality dataset from the same data source.Models We use LLama2-7b [37] and Mistral-7b [14] as our pre-trained models, and fine-tune them with Low-Rank Adaptation (LoRA) [10] on each of the client side. As for the model merging technique, in our main experiments, we use TIES merging. We also compare it with Task Arithmetic in our ablation studies.Baselines The Oracle shows the results that train only on the remaining high quality data in the mixed-quality dataset, serving as the theoretical upper bound. We implement the three baselines: existing methods mentioned in 2.1: Perplexity score, IFD [24] , and DataInf [21] independently on each client.Evaluation metrics The evaluations focus on two main aspects: (1) Question-Answering capabilities, assessed by GPT-4 [30] scoring within the test set splited from the same sources of the training dataset. In the medical QA and multilingual QA tasks, 200 samples are randomly selected from the medical dataset to serve as the test set. We evaluate the models that need to be compared on the test set to generate responses respectively. Then we use the OpenAI GPT-4 model API to assign scores to their responses. Each response of is rated by the judge on a scale from 0 to 1, reflecting how well the answer aligns with the ground truth. In our financial QA task, GPT-4 rate the responses of the fine-tuned model on our data set on a scale of 1 to 10, reflecting criteria including relevance, precision and fluency. To address potential positional bias, we send our response along with the benchmark output to GPT-4 twice, with different orders. We then calculate the average of these scores as the final performance score.(2) Knowledge acquisition, measured by average accuracy of responses to multiple-choice questions in the MMLU clinical topics [8, 9] , MedMCQA [31] , PubMedQA [16] , and USMLE [15] datasets. Although the goal of private domain fine-tuning is not to increase knowledge, there shouldn't be too much knowledge forgetting during this process. (3) Data selection correctness Precision, Recall, F-1 Score, and Accuracy are widely-used evaluation metrics that provide complementary insights into the model's effectiveness from the data selection perspective. In our case, positive instances represent high-quality data, while negative instances represent low-quality data. Precision quantifies the proportion of correctly identified positive instances among all instances predicted as positive, while Recall measures the proportion of correctly identified positive instances among all actual positive instances in the dataset. The F1 Score offers a balanced measure of Precision and Recall, while Accuracy reflects the overall correctness of our data selection (based on our data scoring and threshold determining method) across all classes.Based on the low-quality dataset setup, we evaluate our data-quality control pipeline in collaborative LLM fine-tuning in both federated (communication round cr = 300) and model merging (communication round cr = 1) settings. Note that in federated learning, the server and clients need to intensively communicate the model updates during model training. We implement the three baseline methods described in the Section 2.1 to calculate scores for each training data, and set the unified scoring standard using corresponding scoring functions with anchor data.We demonstrate the performance of data quality control methods in collaborative settings in the medical QA task (Tab. 1) and Multilingual QA task (Tab. 2).Federated Learning v.s. Model Merging Firstly, for both pre-trained models and tasks, with other settings remaining the same, model merging performs better than federated learning. This indicates that loose communication between the local model and the server, compared to frequent communication, might lead to better generalization. Additionally, the performance boost with selected data in the federated setting is larger than in the model merging setting. This might be because during federated learning, we calculate the data score based on the global model (instead of the local model In both federated and model merging settings, our data selection can achieve over 96% and over 91% of the theoretical upper bound performance, respectively. Our method outperforms the other local data selection baselines under the GPT4 Scoring metrics. Compared to the other methods which cause severe forgetting during instruction tuning, the performance of our method on the Knowledge-based benchmark remains within an acceptable range. This shows that our methods are able to improve domain-specific tasks without forgetting knowledge injected during pretraining.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: See Section 4 and Appendix 6. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. Justification: See Section 4 and Appendix 6.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
qualitative analysis: 
 We performed a qualitative analysis by manually comparing the outputs generated by models finetuned on our selected high-quality data versus the original low-quality data. This comparison (Tab. 7 and Tab. 8) provides insights into the tangible improvements in model performance and output quality.To evaluate the robustness of our data selection method under different data quality conditions, we conducted a series of experiments with varying proportions of low-quality data. We maintained a consistent proportion of low-quality data across all clients for each experiment, ranging from 0% to 100%, including pollution levels 20%, 50%, and 80%.Fig. 4 presents the performance of models trained with and without our data selection method across these different proportions. The results demonstrate that our method effectively enhances data quality across all scenarios with GPT-4 scoring. And in terms of accuracy of the data selection, our method consistently selected over 99% of the high-quality data across different proportions of low-quality data. Additionally, to understand the adaptability of our global threshold, we analyzed how the global threshold changes with different proportions of low-quality data. Fig. 4 illustrates that our global threshold adjusts across varying levels of data quality. To provide a more comprehensive analysis, in addition to the experiments in the domain heterogeneity setting shown above, we conducted additional experiments in a quality heterogeneity setting using the FiQA dataset, which focuses on the answer of financial questions. Specifically, we randomly polluted 80%, 20%, 10%, and 50% of the training set for each of the four clients, respectively. The findings demonstrate that our method significantly enhances the quality of the data even when clients have different proportions of low-quality data. Varying Merging Techniques Fig. 5 demonstrates that different weighted merging or aggregation techniques lead to varying performance. Notably, the performance of our data selection method with the Linear Merging technique does not even reach the performance of low-quality data with TIES Merging technique, highlighting the significant impact of weighted merging techniques on overall performance. Furthermore, we experimented with different merging techniques on the FiQA dataset, demonstrating the importance of weighted merging, shown in Fig. 5 .Layer Selection for Low-Rank Tracing Gradient In terms of layer selection, we evaluated both the last layer and the token embeddings. We show that layer selection distorts the score (the inner product of two gradients). In our ablation study, we observe that since the activation connected to the last layer of weights contains shared logic, the data influenced calculated through the last layer weights are prone to a cancellation effect, where the data influence of different examples has a large magnitude that contradicts each other. The cancelation effect lowers the power of the influence score, and deleting influential examples according to this measure often does not change the model's behavior by much. From Fig. 5 , we show that the first layer has a less severe cancelation effect than the last layer.We conducted an ablation study on our global threshold to further validate our approach. Tab. 3 illustrates the advantage of using a global threshold determined by our anchor data for data selection in this heterogeneous setting, compared to selection based on average ratio or pre-determined scores. These results demonstrate that our approach successfully balances the identification of positive cases with the minimization of false positives, offering a robust and superior solution. Collaborative model development, including model merging and federated averaging, would benefit from different kinds of high-quality data, and for each of them, the definition of quality is slightly different. In this paper, we establish a data quality control pipeline for collaborative fine-tuning of LLMs, avoiding directly sharing any private data. Our experiments show that the selected high-quality data ensures an effective and reliable learning process, leading to improved model performance.To the best of our knowledge, we are the first to propose a data selection method for large language models in a collaborative setting, while previous work has mainly focused on traditional centralized settings. We bring up the insights to view federated learning and model merging within the same framework, incorporate different experimental setups and unify federated learning and model merging methods, making it universally applicable. Additionally, our method performs well on generation datasets and takes into account scenarios with bad data, while previous work has not considered downstream domain-specific generation tasks for large language models. Our method does not require repeated training.Societal impact Our work builds large language models that make it possible to create a collaborative instead of a monolithic ecosystem from open-source models while preserving the privacy of users' own data. The constant progress being made in machine learning needs to extend across borders if we are to democratize ML in developing countries. Adapting state-of-the-art (SOTA) methods to resource-constrained environments such as developing countries can be challenging in practice, pushing open source and inclusion.Limitations and future work Our data quality control methods are based on the assumption that all the local models share the same model architectures. It is easy to achieve when our fine-tuning is based on the LoRA adapter. However, it may be worth extending it to adapt to different local model architectures, for example, different low ranks. Future work may explore the intrinsic relation between data selection and the model parameters and how our data selection methods can help reduce the interference of parameter vectors from different models.Perplexity (PPL) serves as a fundamental metric in language modeling to measure the model's ability to predict text sequences accurately. Mathematically, it is defined as the exponential of the average negative log-likelihood: PPL = exp(-1 N N i=1 log P (w i |w 1 , . . . , w i-1 )), where N is the number of tokens in the sequence and P (w i |w 1 , . . . , w i-1 ) represents the probability the model assigns to token w i given its preceding context. A lower perplexity score indicates better model performance, as it suggests that the model assigns higher probabilities to the correct tokens in the sequence, effectively measuring how "surprised" the model is by new text.Instruction Following Difficulty (IFD) [24] provides a quantitative metric for evaluating the difficulty of instruction-following tasks in language models. This score is calculated as the ratio between two key measurements: the Conditioned Answer Score s θ (A|Q) and the Direct Answer Score s θ (A): ) , where s θ (A|Q) measures the model's ability to generate responses with instructional context, and s θ (A) evaluates the model's capability to generate answers in isolation. The Direct Answer Score is computed asIFD θ (Q, A) = s θ (A|Q) s θ (As θ (A) = -1 N N i=1 log P (w A i |w A 1 , . . . , w A i-1 ; θ). This metric quantifies the extent to which instructions aid in response generation, where a higher IFD score suggests that the given instruction provides limited useful context for the model's response generation, indicating greater difficulty in following the instruction.Influence Functions [19] : DataInf represents an efficient algorithm for computing influence functions, distinguished by its closed-form expression that reduces computational and memory complexity compared to existing methods.The algorithm approximates the inverse Hessian calculation (G l (θ * ) + λ l I d l ) -1 through the key transformation:1 n n i=1 ∇ θ l ℓ i ∇ θ l ℓ T i + λ l I d l -1 ≈ 1 n n i=1 I d l - ∇ θ l ℓi∇ θ l ℓ T i λ l +∇ θ l ℓ T i ∇ θ l ℓi, where the Sherman-Morrison formula enables a closed-form solution. The influence function is computed as I DataInf (x k , y k ) = L l=1 1 λ l 1 n n i=1 L l,i λ l +L l,ii L l,ik -L l,k .Llama2-7b anc do Calculate ∇ℓ (z, θ t ) for each checkpoint t in T do for each validation sampletrain do Calculate ∇ℓ (z ′ , θ t ) for each checkpoint t in T do for each validation sample z ′ i ∈ D (k) val do Calculate ∇ℓ (z ′ i , θ t ) end for end for S(z) = z ′ T t=1 ηi ∇ℓ (z ′ , θ t ) • ∇ℓ (z, θ t ) ▷ Data Scoring end for D ′ (k) train = z i ∈ D (k) train , z i ≥ τ ▷ Selectz ′ i ∈ D (k) val do Calculate ∇ℓ (z ′ i , θ t ) end for end for S(z) = T t=1 ηi ∇ℓ (z ′ , θ t ) • ∇ℓ (z, θ t )▷ Anchor data score end for Determine the global threshold τ with anchor data D anc Send τ to each clients for training with High-quality data Aggregate client updates:θ ′ = K k=1 |D k | K k=1 |D k | θ ′ k ▷ Model Merging or AggregationThe overall compute complexity, where N is number of checkpoints, and d is gradient dimension.O (N • |D| • |D val | • d)The overall storage complexity:O(|D| • N • d + |D val | • N • d)We use Hugging Face's transformers library [39] and PEFT library [27] for loading pre-trained models and LoRA configurations. For our training and evaluation, we use PyTorch as the underlying auto-differentiation framework [32] . We use the AdamW optimizer [26, 18] for all experiments. All the experiments are conducted on the machines with the same hardware configuration using Model Response (w/o data selection)Model Response (w/ our data selection)初产妇，23岁， 规 律 宫 缩10小 时 ， 持 续 观 察2小 时 ， 宫 口 由6cm开 大 至7cm， 胎 头 ＋1， 胎 心140次/分 。 恰 当 的 处 置 应 为. A. 严密观察产程 进 展 B. 静 脉 滴 注 缩 宫 素 C. 立 即行人工破膜 D. 立即行剖宫手术 严 密 观 察 产 程 进 展. 选 择A(严密观察产程进展)的 原 因 是 因 为 初 产 妇 宫 口 开 大 至7cm， 这 表 明 产 程 正 在 缓 慢 进 展 ， 但 并 无 明 显 异 常 。 胎 头 已 下 降 至+1位 置 ， 这 是 正 常 的 胎 位 下 降 过 程 。 胎 心 率140次/分属于正常范围，说 明胎儿目前处于安全状态。 选择D. 立即行剖宫 手 术. 初 产 妇 的 宫 缩 是 一 种 常 见 的 产 前 症 状 ， 宫 缩 的 规 律 性 和 持 续 时 间 可 以 反 映 出 妊 娠 的 进 展 ， 但 它 并 不 代 表 产 妇 的 产 前 状 况 。 选 项A. 严 密 观 察 产 程 进 展 ， 是 一 种 产 前 监 测 的 方 法 ， 但 不 能 代 表 产 妇 的 产 前 状 况 。 选 项B. 静 脉 滴 注 缩 宫 素 ， 是 一 种 产 前 治 疗 方 法 ， 但 不 能 代 表 产 妇的产前状况。 恰 当 的 处 置 应 为A. 严 密 观 察 产 程 进 展. 初 产 妇 的 宫 缩 是 一 种常见的产前症状， 它 可 能 表 示 妊 娠 进 展正常，也可能是产 前妊娠综合征的一种 表现。严密观察产程 进展是对初产妇的产 前监测的重要组成部 分，它可以帮助医生 确定产前妊娠综合征 的发生情况，并采取 适当的治疗措施。因 此 ， 选 项A. 严 密 观 察产程进展是最合适 的处置选项。What type of injury to the arm/elbow most often leads to supracondylar fractures? Supracondylar fractures most often occur after hyperextension injuries of the arm/elbow. 1000 Hg is the pressure required to rupture an average aortic aortic Supracondylar fractures are a type of fracture that occurs above the elbow joint, and they are most commonly caused by a fall onto an outstretched arm. Investors are afraid to buy stocks because they are afraid the stock will drop in price.If the stock price is falling, it's because there's a lot of supply, but not enough demand. If there's a lot of demand, but not enough supply, the price will rise.Table 8 : Samples of the output of merged models on FiQA dataset.(after eventual revisions) with the final version of your paper, and its final version will be published with the paper. The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found. IMPORTANT, please:• Delete this instruction block, but keep the section heading "NeurIPS paper checklist",• Keep the checklist subsection headings, questions/answers and guidelines below.• Do not modify the questions and only use the provided macros for your answers. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We followed the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We have provided the statement of societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper proposes an approach for data selection, and does not pose high risks for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: This paper is licensed under CC-BY-NC-SA 4.0. All other codes, datasets, and references are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.fracturas aumentan debido a una disminución de la densidad ósea, que suele ser particularmente notable en mujeres posmenopáusicas debido a la disminución de estrógenos. Además, la absorción de calcio decrece con la edad, y los requerimientos no son fácilmente satisfechos solo con la dieta. Por ello, es crítico asegurarse de que los adultos mayores consuman suficiente calcio, ya sea a través de la alimentación o mediante suplementos, para mantener huesos fuertes y prevenir fracturas. 
 
 Appendix:  


Decision: The paper proposes a data selection method for large language models in a collaborative fine-tuning setting. The authors present a framework that unifies federated learning and model merging approaches, applicable without task-specific adjustments. 

The proposed method scores training samples by tracing gradients, filters out low-quality data locally, and adaptively merges model parameters offering potential benefits over naive averaging as per FedAvg, model soups and BTM. The authors evaluate their approach on medical and multilingual question-answering datasets, demonstrating performance improvements compared to established baselines.

Reviews are mixed but leaning positive. Most negative aspects have been addressed in the response. Some reviewers noted that the precise ordering of the algorithmic steps was not clear in the original submission. The authors have responded by including pseudo-code in the rebuttal supplemental materialks. There were also concerns regarding the presentation of results, particularly the bolding of values in tables, which may mislead readers into thinking that the method uniformly outperforms all baselines. The authors have provided justification in their rebuttal which should also be added in the final manuscript.

The method’s effectiveness seems to hinge on the relationship between the anchor set and the test set, raising concerns about its generalisability to other settings. The authors have clarified their dataset construction and evaluation processes, but the method’s applicability in more diverse, real-world scenarios remains a potential limitation.

Reviews: review: paper_summary: paper_summary: This paper proposes a data quality control technique for the collaborative training of large language models from filtered private heterogeneous data domains via a quality score function that tracks the gradient of each training sample. The proposed framework is tested in medical and multilingual settings to demonstrate its effectiveness.
strength_weakness: strength_weakness: - The assumption of homogeneity for model architecture, usage of low-rank adaptation, and anchor data with global threshold limit the overall applicability of the proposed framework.
- The experimental settings/details and the ablation study are limited.
questions: questions: - Why use a global threshold as the unified standard of data quality? How were the anchor data selected? Would the usage of anchor data (validation set) introduce an additional bias toward the overall data quality (some datasets could potentially be all excluded) and harm the generalization ability? It would be better to see the performance comparison between global and local thresholds.
- How is the per-sample gradient computed from Eq. (1) line 172? Could the authors provide a complexity analysis on this part?
- Line 196, “sparse nature of low-rank gradients” refers to the trainable weight matrices being low rank or the obtained per-sample gradient being sparse and low-rank.
- It would be great to show the ablation study of directly applying the proposed methods to raw data (supposed high-quality data), given the federated results reported in Table 2.
- Could the authors provide several examples of low-quality data samples as presented in Table 4?
limitations: limitations: See W1-2, Q1-2, 4
paper_summary: paper_summary: This paper proposes a novel approach for data quality control in collaborative fine-tuning of large language models (LLMs), particularly in settings where data cannot be directly shared between different silos due to privacy concerns. The authors introduce a method that scores training samples based on tracing gradients of low-rank adapters, filters out low-quality data locally, and then adaptively merges model parameters. They evaluate their approach on medical and multilingual question-answering datasets, demonstrating significant performance improvements compared to baselines.
strength_weakness: strength_weakness: 1. Although the methodology is thoroughly described, the authors could improve the clarity of Figure 3 by providing a more intuitive explanation of how training dynamics enhance collaborative training processes.
2. The paper contains some grammatical errors and typos. For example, in Table 3, "Performance of Data quality" should be "Performance of data quality", and on line 290, "the data influenced calculated via the last layer weights prone to a cancellation effect" should be corrected to "the data influence calculated via the last layer weights is prone to a cancellation effect".
questions: questions: Please refer to weaknesses part
limitations: limitations: yes
paper_summary: paper_summary: This work proposes a method for finetuning LLMs in federated or collaborative training scenarios that selects the most informative examples for each client to train on such that their local parameter updates are likely to improve the global, merged model's performance on a public (shared) test set. The proposed two phase solution first scores local samples by computing the inner products between a gradient computed on a training sample and a gradient computed on a limited number of shared test samples, and only performs the next episode of local training on points that are above a specific quality threshold determined and distributed by the server in the previous round. Then, the server combines the parameter updates sent from each client, and using the new global parameters, iterates over the same shared anchor set to compute a new quality threshold to send back to the clients. By iterating this approach, in some settings they achieve performance improvements on medical domain tasks as compared to existing data valuation methods and baselines of training on curated subsets or all available data, without federation.
strength_weakness: strength_weakness: - The precise ordering of the algorithmic steps is not clear. Section 3.1-3.3 would greatly benefit from an Algorithm definition that clearly defines the order of operations regarding what gradients are computed on what data using what version of the parameters throughout the training process. It is hard to interpret the experimental results due to the lack of clarity in the section that introduces them.

- Unless the reviewer is misunderstanding something about the evaluation, Table 1, 2, and 3 are all misleadingly bolded. The row label "Ours" is sufficient to tell that the row indicates the proposed method, however, "Ours" is bolded in all three tables, and all columns, regardless of whether or not the method outperforms other valuation criteria, or baseline training settings. On a quick glance this suggests the method offers uniform improvement in all settings, but this is not the case. Please only bold the best score in each column and note the convention in the captions.

- The ablations (while appropriately chosen) are not convincing. The differences reported in 4.3 are likely not statistically significant, and in 4.4, the "hit rate" is not defined, which hurts because this part is of technical interest. Section 4.5 is confusing and the final sentence claims that the anchor score leads to the lowest proportion of low quality data, but this is exactly the opposite of Fig 4(c)?
questions: questions: - It seems like the table and figure reference numbering might be off in a few places, please check the \ref commands.

- I would like to better understand the dataset construction process. There are 3 raw sources listed, but what is the exact sampling used to build the training dataset? How are the n=20 client local datasets selected (are they uniformly sampled from the full training or grouped in some way?) Also, critically, how are the test evaluation samples (the GPT4 evaluation 200) sampled, and what is the relation to the _anchor_ data? Are the anchor data held out Medalpaca, just like the test set?

- If the above description is somewhat correct, followup question is what are the statistics for which samples were selected in each round by each client as "above the quality threshold"... were these locally filtered subsets more often than not the medalpaca training questions? (and when running the other baseline scores, was this not true?) 

My current hypothesis is that this grad x grad product is a similarity score between the local training data and the global test/anchor samples. Thus, it biases each client to train on the test-like data, and not the other training data.
limitations: limitations: - The primary limitation is in the fact that the method does not improve over the baselines or other scores in many settings evaluated. In Table 2 and 3, while "Ours" beats the low quality baseline, it doesn't beat the oracle or raw baselines often. 
- Similarly in Table 1, "Ours" only beats the other quality criteria in the GPT4 based evaluation. Depending on the clarification about the construction of the anchor set, I worry that the method only works well when the test set and the anchor set are closely related which limits the real world applicability of the method.

I would be happy to engage with the authors to help improve the explication of the method and the presentation of the results, but my concerns with the strength of the empiricals are not insignificant. Clarifications would need to demonstrate a misunderstanding on my part of the evaluation setup or the results themselves in order for my score to improve.
paper_summary: paper_summary: The paper proposes a novel data quality control technique to enhance the quality of data from different private domains in collaborative training settings, such as model merging. The technique scores each training sample by tracing gradients of low-rank adapters and filters out low-quality data locally. The model parameters are then adaptively merged based on individual training. The method is evaluated on heterogeneous medical domain data, demonstrating that training on high-quality data selected by this method outperforms other data selection methods across diverse private domain datasets. The paper claims performance improvements and offers a general, efficient way to identify high-quality data for large language models.
strength_weakness: strength_weakness: w1. The paper's evaluation is confined to medical datasets. To demonstrate the generalizability and robustness of the proposed method, experiments should be conducted on a wider range of datasets from different domains.

w2. The paper generates 40% of the data as low-quality, which may not accurately reflect real-world scenarios. It would be beneficial to experiment with varying proportions of low-quality data to understand the method's effectiveness across different levels of data quality.

w3. The method uses only a small amount of public data (10 samples in the paper) as anchor data to calculate the global threshold. The reliability of selecting just 10 samples is questionable, and the basis for this choice should be clarified. A more rigorous justification or experimentation with different numbers of anchor samples could strengthen the method's credibility.

w4. Table 1 in the paper shows that the proposed method performs better than the Oracle method, which uses only high-quality data. The paper does not provide an analysis or explanation for this superior performance. Understanding and explaining why this method outperforms the Oracle method could provide valuable insights and strengthen the overall argument.
questions: questions: Q1. Please explain how the proposed method performs on datasets from different domains beyond the medical field. Have you considered conducting experiments with datasets from various other domains to validate the generalizability of your approach? (w1)

2. Please clarify if 40% is representative of real-world scenarios. Would you consider testing with different proportions of low-quality data to assess the robustness and effectiveness of your method under varying conditions? (w2)

3. Have you tested the impact of using different numbers of anchor samples on the performance and reliability of the method?(w3)

4. Please explain why our method performs better than Oracle and why the other three data evaluation methods perform so poorly in Table 1. (w4)
limitations: limitations: 1. The author's work is similar to federated learning. Should it be compared with the recent federated learning?

Comments: title: W4 & Q4 Further experimental results
Experimental results on raw data
Summary of follow-up questions and discussions
Looking forward to your feedback and further discussion!
Looking forward to your responses and further discussion!
Looking forward to your responses and further discussion!
(contd)
Response to Rebuttal (contd)
Response to Rebuttal
comment: We are grateful for the reviewer's endorsement! We would also like to highlight that we have additional results on FiQA with different settings that all support our claims and conclusions, further strengthening our contribution. We plan to add all of these results to the final version of our paper and the appendix. For more details, please refer to the global comment and our response to Reviewer V7za.
> Why the other three data evaluation methods perform so poorly in Table 1?

We have compared different data scoring methods for their accuracy in selecting high-quality data out of the mixed-quality datasets in the FiQA Experiment I setting mentioned above. Our method demonstrates significant advantages as follows.

|          | Accuracy |
| -------- | -------- |
| PPL      | 58.44% |
| IFD      | 58.90% |
| **Ours** | **97.91%**   |
**Q4**
> ablation study of directly applying the proposed methods to raw data (supposed high-quality data).

We have completed all the requested experiments and are pleased to share the results.

Regarding generalizability beyond the medical domain, we conducted experiments using the FiQA dataset, which focuses on financial question answering. For other results, please refer to our discussion with Reviewer V7za.

*Dataset*: We have four clients, each with 2000 training samples. We split the data into a training set (8000 samples in total), a validation set (also called the anchor set, 100 samples), and a test set (500 samples).

*Evaluation metric*: Responses from the model fine-tuned on our dataset are rated by GPT-4 on a scale from 1 to 10, reflecting criteria including relevance, accuracy, and fluency. To address potential positional bias, we send our response along with the benchmark output to GPT-4 twice, with different orders. We then calculate the average of these scores as the final performance score.

The table presents the performance of models trained with and without our data selection method across these different proportions. The results demonstrate that our method effectively enhances data quality across all scenarios with **GPT-4 scoring**. And in terms of **accuracy** of the data selection, our method consistently selected over 99% of the high-quality data across different proportions of low-quality data. 

|                        | GPT-4 Scoring Performancew/o data selection | GPT-4 Scoring Performancew/ our data selection | Accuracy | Global Threshold (score) |
| ---------------------- | ------------------------------------------- | ---------------------------------------------- | -------- | ------------------------ |
| 0% bad data (raw data) | 3.7400                                      | ———                                            | ———      | ———                      |
| 20% bad data           | 3.4735                                      | 3.6270                                         | 99.3125% | 0.0033                   |
| 50% bad data           | 2.9625                                      | 3.7175                                         | 99.7250% | 0.0017                   |
| 80% bad data           | 2.9155                                      | 3.8335                                         | 99.9375% | 0.0012                   |
| 100% bad data          | 2.2860                                      | ———                                            | ———      | ———                      |

To understand the adaptability of our global threshold, we analyzed how the global threshold changes with different proportions of low-quality data. It illustrates that our **global threshold adjusts across varying levels of data quality**.
We would like to thank all the reviewers for their comments and discussions, and provide a summary of our responses. We will add all corresponding discussions and experimental results to our updated paper.

**Background and Motivation:**

- We have clarified our assumptions on collaborative settings, definition of data quality, and goals of data selection in the collaborative setting. (for Reviewer Xfnf)
- We have validated the reasonableness of our low-quality data construction methods: a larger portion of low-quality data has resulted in lower validation loss. (Please refer to the $\textcolor{blue}{Table}$ showing validation loss at the bottom of this response.)
- We have clarified the data sources of validation data and anchor data. (for Reviewer Xfnf and ngAe)
- We have discussed the same nature of federated learning and model merging. We have expressed the relationship and differences between federated learning and model merging using mathematical formulas. (for Reviewer ngAe)

**Methods:**

- We have refined the notations for improved clarity and have provided more detailed pseudo-code for our method in the PDF file. (for Reviewers igLM, ngAe)
- We have elaborated on the order of the whole procedure. (for Reviewer ngAe)
- We have elaborated on how we determine the global threshold. (for Reviewer Xfnf)
- We have discussed how we calculate the per-sample gradient (for SGD, Adam, and AdamW) and its complexity and scalability. (for Reviewer Xfnf)
- We have offered additional explanation on our method's training dynamics. (for Reviewer igLM)
- We have provided low-quality and high-quality data samples with their scores in our PDF. (for Reviewer Xfnf)

**Experiments:** *We have spent considerable time and effort preparing these new results. We have more results that all support our claims and conclusions and we plan to add all of them to the paper or to the appendix.*

- We have added experiments on a new dataset from a data-sensitive private domain - financial question answering. (for Reviewer V7za)
- We have clarified the baselines we are comparing to in our PDF. (for Reviewer ngAe)
- We have conducted evaluations on two challenging scenarios mentioned in the introduction of our paper: domain heterogeneity and quality heterogeneity.
- We have elaborated on our ablation study on weighted merging techniques. (for Reviewer ngAe)
- We have presented a comparison study on the global threshold determined by anchor data, comparing with other thresholds. (for Reviewers Xfnf, ngAe, and V7za)
- We have provided further discussions of the experimental results, including a comparison of data selection between the federated setting (frequent communication) and model merging (loose communication) setting and the potential reasons for the results. (for Reviewer ngAe) 
- We have experimented with varying proportions of low-quality data to understand the method's effectiveness across different levels of data quality. (for Reviewer V7za)
- We have compared our gradient-based method with other centralized data selection methods including PPL, IFD, and DataInf, and have shown our advantages. (for Reviewer V7za)
- We have conducted an ablation study on the impact of using different numbers of anchor samples on the performance and reliability of the method. (for Reviewer V7za)

We once again express our gratitude to all reviewers for their time and effort devoted to evaluating our work. We hope our discussion and summary have addressed your concerns, misunderstandings, or questions.

--------

$\textcolor{blue}{Table}$: A larger portion of low-quality data results in lower validation loss. After data selection, the validation loss becomes the same as that of the raw data, demonstrating the effectiveness of our low-quality data construction and data selection methods. 

|                        | Validation Loss | Validation Loss After Data Selection |
| ---------------------- | --------------- | ------------------------------------ |
| 0% bad data (raw data) | 1.77342         | —————                                |
| 20% bad data           | 1.77893         | 1.77397                              |
| 50% bad data           | 1.80144         | 1.77344                              |
| 80% bad data           | 1.85195         | 1.77322                              |
| 100% bad data          | 1.93056         | —————                                |
We would like to thank you once again for your time and effort in providing insightful feedback and comments. As there are only a few hours remaining for further discussion, could you kindly let us know if you have any remaining concerns or points of confusion? We would be more than happy to discuss, address, and resolve them with you! If our responses have addressed your follow-up concerns, we would greatly appreciate your considering an increase in your rating to reflect these new improvements and clarifications, which we'll incorporate in our updated paper.

Thank you for your continued engagement and support in strengthening our research!
We would like to thank you once again for your time and effort in providing feedback and comments. As there are less than 12 hours remaining for further discussion, we would greatly appreciate it if you could let us know of any remaining concerns or points of confusion. We would be more than glad to discuss, address, and resolve them with you!
We would like to thank you once again for your time and effort in providing feedback and comments. As there are less than 12 hours remaining for further discussion, we would greatly appreciate if you could let us know of any remaining concerns or points of confusion. We would be more than glad to discuss, address, and resolve them with you!
We appreciate your insightful question regarding the scalability of our method! 

From an implementation perspective, per-sample gradient calculation is a widely encountered quantity in differential privacy and meta-learning. There are many tools we can use to compute per-sample gradients efficiently if needed, such as: (1) the Opacus library, which focuses on extending PyTorch for differential privacy, or (2) functorch, where composing *vmap* and *grad* provide significant speedup. In general, vectorization with *vmap* should be faster than running a function in a for-loop and competitive with manual batching.

Moreover, it's important to clarify the context of our work: our focus is on collaborative fine-tuning scenarios where clients have limited resources and data, naturally constraining the scale of per-client data selection. This setting often doesn't involve extremely large datasets on individual clients. In multi-GPU setups, we can leverage parallel computation across GPUs to enhance efficiency.

For cases involving clients with extremely large datasets or single GPU, in addition to using tools orthogonal to our proposed method like the Opacus library or functorch, we can also adapt our algorithm to address scalability requirements: We can compute a coarse-grained per-batch score, select all samples in batches exceeding our per-batch threshold, and apply per-sample gradient computation only to lower-scoring batches. This extended approach allows us to balance efficiency and precision in data selection, addressing potential scalability issues in large-scale scenarios from the algorithm perspective.
(contd)

We express the relationship and differences between federated learning and model merging using mathematical formulas as follows:


**Model merging:**

let $f_\theta \in \mathcal{F}$ denote the language model and $D\_k \in \mathcal{D}$ denote the training dataset on client $k$. Given the training datasets $D_k$, we can define a model merging operator $\mathcal{M}\_K( \cdot ; D_k , k \in K=\{1, \cdots, n\}): \mathcal{F} \rightarrow \mathcal{F}$.

The model merging process can be expressed as 
$$
f\_{merging}=\mathcal{M}\_K(f)
$$


**Federated Learning:**

Based on the notation of model merging, the federated averaging process can be expressed as

$$
f\_{fed}= (\prod\_{t=1}^T \mathcal{M}\_{S\_t(K)})(f)
$$

where $T$ is the round number. $S_t(K)$ is the index set of selected datasets to train on each client at round $t$.

------

**Ablation Studies:**

Thank you for giving us the opportunity to clarify our approach! 

1. Merging Techniques (Section 4.3): Figure 4(a) in our paper demonstrates that different weighted merging or aggregation techniques lead to varying performance. Notably, the performance of our data selection method with the Linear Merging technique doesn't even reach the performance of low-quality data with TIES merging technique, highlighting the **significant impact of weighted merging techniques** on overall performance. To make the comparison more clear, we presented it in the following $\textcolor{blue}{Table 1}$:

$\textcolor{blue}{Table 1:}$

|                              | GPT-4 Scoring Performance of Model Merging |
| ---------------------------- | --------------------------------------- |
| Raw Data                     | 0.505                                   |
| Low-qual Data                | 0.485                                   |
| Oracle                       | 0.490                                   |
| **Ours (w/ linear merging)** | 0.470                                   |
| **Ours (w/ TIES merging)**   | **0.487**                               |

2. Hit Rate and Low-Quality Data Selection (Sections 4.4 and 4.5):
We apologize for the confusion. The "hit rate" in 4.4 and the "number of selected low-quality data" in 4.5 refer to the same metric: the proportion of bad data being filtered out. We presented this in two forms (percentage and data size) to provide different perspectives. A higher value indicates more effective filtering of bad data by our algorithm.

3. Additional Experiments: (Since the start of the rebuttal period, we have been conducting additional experiments to perform ablation studies. We now have comprehensive results that we believe are both convincing and robust. We are pleased to report these findings as follows)

To provide a more comprehensive ablation study, in addition to the experiments in the "domain heterogeneity" setting shown above, we conducted additional experiments in a "quality heterogeneity" setting using the FiQA dataset, which focuses on financial question answering. We randomly polluted the training sets of four clients with varying degrees of low-quality data: 80%, 20%, 10%, and 50%, respectively. The following $\textcolor{blue}{Table 2}$ illustrates the advantage of using a global threshold determined by our anchor data for data selection:

$\textcolor{blue}{Table 2:}$
|                            | Precision  | Recall     | F1 Score   | Accuracy   |
| -------------------------- | ---------- | ---------- | ---------- | ---------- |
| select by ratio            | 79.17%     | 79.17%     | 79.17%     | 75.00%        |
| select by hardcode score     | 92.77%     | 99.13%    | 95.84%     | 95.00%        |
| select by global threshold (**Ours**) | **97.44%** | **99.38%** | **98.39%** | **97.91%** |

These results demonstrate that our approach successfully balances the identification of positive cases with the minimization of false positives, offering a robust and superior solution. We have also conducted additional ablation studies on the global threshold, which can be found in Figure 2 of the updated manuscript.

Furthermore, we conducted the ablation study of different merging techniques on the FiQA dataset, demonstrating the importance of weighted merging, shown in $\textcolor{blue}{Table 3}$. We use pairwise evaluation here, where responses from the model fine-tuned on our dataset are rated by GPT-4 on a scale from 1 to 10, reflecting criteria including relevance, accuracy, and fluency.

$\textcolor{blue}{Table 3:}$
|                              | GPT-4 Scoring |
| ---------------------------- | --------------------------------------- |
| Low-qual Data                | 3.650                                   |
|Oracle| 3.748|
| **Ours (w/ linear merging)**   | 3.776                               |
| **Ours (w/ weighted merging)** | **3.855**                                  |
**Evaluation:**

Thank you for your suggestions on presenting the experimental results. Since we cannot directly modify the original PDF at this stage, we will implement the following changes in the updated version of the paper:

* We will bold the best-performing selection method in each column of Table 1 (Table 4 in the supplement), as you suggested. This includes bolding "Ours" in the Mistral GPT-4 and Llama GPT-4 columns, and "PPL" in the Mistral Knowledge and Llama Knowledge columns.

* We will update the table caption to read: "The best performing selection method (row) in the lower subtable is bolded for each benchmark evaluation (column)."

If you have any further questions or concerns about the experimental setup or results, we would be more than glad to continue the discussion on OpenReview.

-----


**Threshold:**

We do not use dynamic thresholds, nor is there any description related to dynamic thresholds in our submitted paper. We appreciate the opportunity to clarify this misunderstanding and address the confusion: The threshold is calculated only once throughout the entire pipeline. For details and proof on determining the global threshold, please refer to [our response to R1](https://openreview.net/forum?id=OU1uqd1vyw&noteId=xKx1wdiFS3).

To clarify our method, here's a detailed breakdown of the data selection pipeline in collaborative fine-tuning, summerizing from Section 3.1 in our original paper and the pseudocode in our manuscript:

* Stage 1 (On each client): Local fine-tuning with low-quality data; save model checkpoints.
* Stage 2 (On each client): Calculate gradients, compute scores for each training sample, send scores to the server.
* Stage 3 (On server): Calculate gradients, compute scores of anchor data, determine the global threshold using anchor data scores and client scores.
* Stage 4 (On each client): Select data with scores not lower than the global threshold.
* Stage 5 (On each client): Local fine-tuning with high-quality data, then send model parameters to the server.
* Stage 6 (On server): Merge client model parameters to obtain the final global model.

Your understanding that *"in expectation, grad x grad products between local training samples and local validation samples will be large for clean samples and small for corrupted samples"* is correct! However, we want to clarify that we do not use "a contextual threshold based on the current loss." As mentioned earlier, the threshold is calculated only once. 

-----

**Federated versus Model Merging**

We integrate both federated learning and model merging into a unified algorithmic framework because they fundamentally involve local training followed by model aggregation in the parameter space. Since weights in neural networks are updated through optimization algorithms based on accumulated gradients, our gradient-based method influences the model's weight space through selective gradient use. This underlying intuition allows our method to be applicable to both collaborative learning scenarios.

The only difference between federated and model merging algorithms is: In the federated setting, there's periodic communication between the server (global model, or $\theta\_{merge}$ in our paper) and clients (local models). Clients perform local training between communication rounds. At each round, clients send their current model parameters to the server, which aggregates them to form the current global model, then sends it back to all clients. After each communication round, server and client models are aligned. In contrast, in the model merging setting, there's no periodic communication during local training; local models train independently throughout the entire process. Parameters are sent to the server only once, at the end of training, to form the final global model. 

To clarify **Point 2** further: In the federated setting, the checkpoints we save for each local client are based on the global model updated from the last model aggregation on the server, which typically occurs after a fixed number of local training epochs or iterations. This process incorporates information from other clients' models. Crucially, at the end of each communication round, the model parameters on the server and all participated clients are synchronized. This ensures that all participating clients start the next round from the same point, incorporating collective knowledge. This synchronization can help our data selection method better trace and regularize the training trajectory toward the optimal location.

We don't calculate gradients or scores during the FL process itself; instead, we compute the scores only after the entire training is complete.
**Experiment II: Varying Levels of Low-Quality Data**

To evaluate the robustness of our data selection method under different data quality conditions, we conducted a series of experiments with varying proportions of low-quality data. We maintained a consistent proportion of low-quality data across all clients for each experiment, ranging from 0% to 100%, including 20%, 50%, and 80% pollution levels.

$\textcolor{blue}{Table 4}$ presents the performance of models trained with and without our data selection method across these different proportions. The results demonstrate that our method effectively enhances data quality across all scenarios with **GPT-4 scoring**. And in terms of **accuracy** of the data selection, our method consistently selected over 99% of the high-quality data across different proportions of low-quality data. 

$\textcolor{blue}{Table 4:}$
|                        | GPT-4 Scoring Performancew/o data selection | GPT-4 Scoring Performancew/ our data selection | Accuracy | Global Threshold (score) |
| ---------------------- | ------------------------------------------- | ---------------------------------------------- | -------- | ------------------------ |
| 0% bad data (raw data) | 3.7400                                      | ———                                            | ———      | ———                      |
| 20% bad data           | 3.4735                                      | 3.6270                                         | 99.3125% | 0.0033                   |
| 50% bad data           | 2.9625                                      | 3.7175                                         | 99.7250% | 0.0017                   |
| 80% bad data           | 2.9155                                      | 3.8335                                         | 99.9375% | 0.0012                   |
| 100% bad data          | 2.2860                                      | ———                                            | ———      | ———                      |

To understand the adaptability of our global threshold, we analyzed how the global threshold changes with different proportions of low-quality data. $\textcolor{blue}{Table 4}$ illustrates that our **global threshold adjusts across varying levels of data quality**.

-------

> Limitation: The author's work is similar to federated learning. Should it be compared with the recent federated learning?

Our work has covered federated learning as one of the collaborative training paradigms, and our evaluation was conducted in a federated learning setting. For a detailed discussion of empirical results, please refer to [our response to Reviewer ngAe regarding W2 & L1](https://openreview.net/forum?id=OU1uqd1vyw&noteId=rALL4TsjnZ). If you're referring to specific recent developments in federated learning that warrant additional comparison, we'd appreciate more details and elaboration.

----

To summarize, these results collectively demonstrate the robustness and effectiveness of our method across a wide spectrum of data quality scenarios, reinforcing its potential for real-world applications.
We have completed all the requested experiments and are pleased to share the results. 

Regarding generalizability beyond the medical domain, we conducted experiments using the FiQA dataset, which focuses on financial question answering. 

*Dataset:* We have four clients, each with 2000 training samples. We split the data into a training set (8000 samples in total), a validation set (also called the anchor set, 100 samples), and a test set (500 samples).

*Evaluation metric:* Responses from the model fine-tuned on our dataset are rated by GPT-4 on a scale from 1 to 10, reflecting criteria including relevance, accuracy, and fluency. To address potential positional bias, we send our response along with the benchmark output to GPT-4 twice, with different orders. We then calculate the average of these scores as the final performance score.

**Experiment I: Quality Heterogeneity**

To evaluate our data selection method's effectiveness in scenarios with **heterogeneous data quality**, we simulated varying levels of data pollution across clients. Specifically, we randomly polluted 80%, 20%, 10%, and 50% of the training set for each of the four clients, respectively.

$\textcolor{blue}{Table 1}$ presents the results of our model merging experiments. These findings demonstrate that our method significantly enhances data quality even when clients have differing proportions of low-quality data:

$\textcolor{blue}{Table 1:}$
|               | GPT-4 Scoring |
| ------------- | ------------- |
| Low-qual Data | 3.6495        |
| Oracle        | 3.7480        |
| **Ours (w/ linear merging)**      | 3.7755    |
|**Ours (w/ weighted merging)**|	**3.8550**|

We conducted an **ablation study** on our global threshold to further validate our approach. $\textcolor{blue}{Table 2}$ illustrates the advantage of using a global threshold determined by our anchor data for data selection in this heterogeneous setting, compared to selection based on average ratio or pre-determined scores. These results demonstrate that our approach successfully balances the identification of positive cases with the minimization of false positives, offering a robust and superior solution.

$\textcolor{blue}{Table 2:}$
|                            | Precision  | Recall     | F1 Score   | Accuracy   |
| -------------------------- | ---------- | ---------- | ---------- | ---------- |
| select by ratio            | 79.17%     | 79.17%     | 79.17%     | 75%        |
| select by static score     | 92.77%	| 99.13%	| 95.84%	| 95.00%| 
| select by global threshold (**Ours**) | **97.44%** | **99.38%** | **98.39%** | **97.91%** |

Additionally, we performed a **qualitative analysis** by manually comparing the outputs generated by models fine-tuned on our selected high-quality data versus the original low-quality data. This comparison ($\textcolor{blue}{Table 3}$) provided insights into the tangible improvements in model performance and output quality.

$\textcolor{blue}{Table 3:}$
| Questions | Model Response  (w/ data selection) | Model Response (w/ our data selection) | 
| ----------------- | ------------- | --------- |
| How does a 2 year treasury note work? | It is a 2 year treasury note.  | Treasury notes are a short-term government security that matures in two years. They are issued by the U.S. Treasury Department to help finance the nation's deficit. |0.505|
|What causes a stock to drop in price?|Investors are afraid to buy stocks because they are afraid the stock will drop in price. |If the stock price is falling, it's because there's a lot of supply, but not enough demand. If there's a lot of demand, but not enough supply, the price will rise. |
Thanks for the response. As for point 3, it should be 'micro-batching.' And I understand that the computation of the per-sample gradient is based on a single checkpoint. But I'm not sure what the authors mean here by "allow us to process each training data point in parallel". Popular auto differentiation libraries such as PyTroch provide the aggregated gradient for a batch but not per-sample gradients. So, to get the per-sample gradient, micro-batching is generally used to run backpropagation one time per data sample, where the benefit of parallelism is lost. So, could the authors further elaborate on why this part's scalability is **not** an issue?
**2. Global Threshold:**

> I suggest that the authors incorporate the discussion of the necessity and justification for global thresholds and their implications

According to **Definition 1.2** and **Remarks**, global thresholds are essential in our framework to effectively handle **domain heterogeneity** mentioned in our paper. As we highlight in our paper, local models can interfere with each other when merged in the weight space. Since our objective is to reduce the validation loss of the global model rather than local models, data selected by local thresholds don't necessarily translate to an optimal global model when merged. Global thresholds are necessary to coordinate different local distributions, thereby avoiding potential conflicts or interference during model merging or aggregation. 

>  The current conclusion is based on limited observation from the selected datasets.

We have also conducted experiments on the FiQA dataset with different settings (see more details in our response to Reviewer V7za). They all demonstrate the same pattern, showing that the global threshold minimizes the sum of the distances to each local optimal threshold, while local thresholds cannot lead to global optimal solution. According to the instructions for the discussion period, we are not allowed to include new figures. However, we will incorporate figures showing the score distribution with different thresholds in the final version of our paper.


In addition, we provide details and proof on how we determine the global threshold as follows:

Suppose the score distribution of all data (including both good and bad data) is denoted by $F(x)$. The score distributions for good data and bad data are represented by $f(x)$ and $g(x)$ respectively. The relationship between these distributions can be expressed as:
$$
F(x)=\alpha*f(x)+(1-\alpha)*g(x)
$$
where $\alpha$ is the proportion of good data in the dataset.

We can estimate $F(x)$ using the scores from all clients:
$$
\hat F(x) = \Sigma_{z_i \in D_{train}} kernel(x-S(z_i))
$$
Similarly, we estimate $f(x)$ using the scores from anchor data:
$$
\hat f(x) = \Sigma\_{z\_i^* \in D\_{anchor}} kernel(x-S(z^*\_i))
$$
Assuming $\alpha$ is known, which can be estimated from the quality of the global data, we can estimate the bad data distribution as follows:
$$
\hat g(x)=\frac{\hat F(x)-\alpha*\hat f(x)}{1-\alpha}
$$
The optimal global threshold $\tau^*$ can be obtained by :
% $$
% (1-\alpha)\hat g(\tau^*)=\alpha*\hat f(\tau^*)
% $$
% This implies that the proportion of good data above the threshold $\tau^*$ is maximized.
$$
\tau^*= argmax\{\frac{\int_{\tau}^{+\infty}\alpha\hat f(x) \, dx}{\int_{\tau}^{+\infty}(1-\alpha)\hat g(x) \, dx}+\lambda\int_{\tau}^{+\infty}\hat F(x) \, dx\}
$$
where $\lambda$ is a hyper-parameter that balance the data number and good data ratio.

For a specific submodel $k$, the algorithm can be further refined. We denote the estimated good data distribution for a specific data source as $\hat{f}^{(k)}(x)$:

$$
\hat f^{(k)}(x) = \Sigma_{z_i^* \in D^{(k)}_{anchor}} kernel(x-S(z^*_i))
$$
The threshold for the specific submodel can then be determined by:
% $$
% \hat g(\tau^*)=\hat f^{(k)}(\tau^*)
% $$

$$
\tau^*= argmax\{\frac{\int_{\tau}^{+\infty}\alpha\hat f^{(k)}(x) \, dx}{\int_{\tau}^{+\infty}(1-\alpha)\hat g(x) \, dx}+\lambda\int_{\tau}^{+\infty}\hat F(x) \, dx\}
$$

------

**3. Per-sample gradient calculation:**

We appreciate the opportunity to clarify our approach to calculating sample gradients from a checkpoint:
For a given checkpoint with parameters $\theta_t$, we compute the gradient of a single sample $z_i$ as follows:
$$
\nabla\ell\left(\boldsymbol{\theta}\_t;\boldsymbol{z}\_i\right) =\nabla\_{\theta_t} \text{LOSS}(f\_{\theta_t}(x\_i),y\_i)
$$
where $x_i$ and $y_i$ are the input and label of $z_i$, respectively.
To enhance computational efficiency, we can focus on specific parts of $\theta$, such as individual layers or LoRA modules (e.g., q, k, v, o). For instance, if we concentrate solely on the gradients of parameters in layer 0, we adjust our gradient calculation as:
$$
\nabla\ell\left(\boldsymbol{\theta}\_t;\boldsymbol{z}\_i\right) =\nabla\_{\theta\_t^{layer0}} \text{LOSS}(f\_{\theta\_t}(x\_i),y\_i)
$$
It's important to note that this gradient computation process is based on a single checkpoint, with no parameter updates occurring throughout. This allows us to process each training data point in parallel, significantly improving computational efficiency.


Regarding your concern about scalability, did you mean "micro-batching" here? In our scoring calculation step, we use a micro batch size of 1. Since there are no model updates during this process, scalability is not an issue.

We'd be glad to open-source our code implementation for more details.
First of all, we would like to clarify the following definition and notations: 


**Definition 1.1** (Data Quality on Specific Domain $k$)**.** Given a model architecture $\theta$, a training configuration (optimizer, etc.), and a validation set $D_{val}$ in a specific domain $k$, the quality of training data $z$ is defined as follows: for $z\_1, z\_2 \in \mathcal{D}\_{train}$, if $\mathcal{L}\_{val}(\theta(z\_1), D\_{val}) < \mathcal{L}\_{val}(\theta(z\_2), D\_{val})$, then the quality of $z_1$ is considered higher than that of $z_2$. Here, $\mathcal{L}_{val}$ denotes the validation loss. In other words, the lower the validation loss, the higher the data quality.


**Definition 1.2** (Data Quality on Collaborative Private Domains)**.** Given a model architecture $\theta$, a training configuration (optimizer, etc.), and a validation set $D\_{val}=\{\mathcal{D}^{(1)}\_{val}, \mathcal{D}^{(2)}\_{val}, \ldots, \mathcal{D}^{(K)}\_{val}\}$ for all $K$ tasks, the quality of training data $z$ is defined based on the validation loss of **the global model** $\theta_{merged}$ on $D_{val}$. Specifically, for $z\_1, z\_2 \in \\mathcal{D}^{(k)}\_{train}$, if $\mathcal{L}\_{val}(\theta\_{merged}(z\_1), D\_{val}) < \mathcal{L}{val}(\theta_{merged}(z\_2), D\_{val})$, then the quality of $z_1$ is considered higher than that of $z_2$. As in the single-domain case, lower validation loss indicates higher data quality.

**Remarks** (Enhancing Data Quality on Collaborative Private Domains)**.** In the collaborative learning framework, the ratio and distribution of low-quality data are unknown a priori. Only the server has access to the global distribution of both high-quality and low-quality data, while individual users cannot infer the global distribution from their local distributions due to statistical heterogeneity. The server can infer the distribution of high-quality data from public anchor data. Our objective is to select data points that most significantly reduce the validation loss of the global model, rather than optimizing for each local model independently. It is important to note that the scope of this study is not considering new models joining during training or continual learning paradigms. 

--------
Below are our responses to each follow-up question. We appreciate your insightful comments and the time and effort you've engaged in helping us clarify our proposed method. We look forward to continuing discussion with you on the OpenReview system if you have any remaining concerns, questions, or if you feel there's any misunderstanding on our part regarding the questions!

**1. Mixed Model Architectures:**

As per **Definition 1.1** above, data quality is specific to each model architecture. Different architectures naturally lead to the selection of different high-quality data points. Under the current definition of data quality, it is not a reasonable assumption to uniform data selection across models with varying configurations. For example, the data quality standard would differ significantly between LoRA models with rank $r=16$ and those with $r=4$. 

When it comes to individual models or groups of models from different domains with identical configurations, our proposed method is orthogonal and can be broadly applied. It is compatible with any model that allows gradient computation and can seamlessly integrate with state-of-the-art PEFT techniques such as AdaLoRA or Prompt Tuning.

Given that current collaborative learning and distributed training scenarios predominantly focus on models with homogeneous architectures, we believe our algorithm has broad applicability. While we acknowledge the potential for mixed fine-tuning with various PEFT methods, our current focus is on providing a SOTA solution for the most common use cases in the field.
Thank the author for providing further information and contributing great work to the community! I have read other reviewers' comments and the rebuttal information. I would like to remain my rating as 'Accept.'
Thank the authors for the response and clarification. The added new results certainly help us understand the proposed framework.

- Regarding W1, I wanted to highlight the potential use case for mixed fine-tuning with other PEFT methods and LoRA with different parameters (e.g. rank).
- Q1, I suggest that the authors incorporate the discussion of the necessity and justification for global thresholds and their implications. Also, the current conclusion is based on limited observation from the selected datasets.
- Q3 Here, do the authors use macro-batching to obtain the per-sample gradient? If that's the case, then I reserve the scalability concerns over the proposed framework.

In recognizing the effort of the authors for rebuttal, I increase my score to 5.
Finally, the data clarifications are _much_ appreciated and are critical to incorporate into the updated draft. It is important to clearly state the distributional relationship between the client training data, the client validation samples, the anchor data, and the final test data. (Correct me if I am still wrong but) In light of this new clarity, it seems that a simple explanation for how this method works is that the validation sets for each client are all known to be clean, and drawn from the test distribution. Therefore in expectation grad x grad products between local training samples and local validation samples will either be large when the training sample is clean, and small when the sample is corrupted. Critically though, this distinction will be relative, and this is a hypothesis for why a contextual threshold based on the current loss of the server model on the anchor data must be recomputed throughout the overall training process. 

Following this train of thought, was this need for a dynamic threshold ablated? What about the threshold update frequency? Would quantized threshold updates help or hurt the method? In general, was any analysis of the trajectory/statistics the score thresholds from the training runs performed? This would be a nice visual in an updated draft. Something like iterations on the x-axis, and the value of the prescribed global threshold on the y-axis. It could also show the fraction of client data that was above the current threshold at that step, and therefore trained on. Further/alternately, it could quantify the precision and recall of the local filtering - how many of the local samples for a client were correctly identified as clean (by passing the threshold) and how many of the corrupted samples were identified (by being omitted from the training round). It would be illuminating to see whether this threshold and the selections it causes have a stable or asymptotic dynamics or whether they are noisy throughout training.

I note that the rebuttal did not address weakness 3 in my original review. I believe that these ablations are interesting, but they are not clearly presented so I would be unable to summarize what we are supposed to learn from them in the manuscript's current form.

I am happy to bump my score up to a 4, but am unable to move it into acceptance territory at this time.
I appreciate the author's inclusion of the algorithm pseudocode. I have some followup questions on the order of operations. 
1) Is the new global threshold computed _before_ aggregating the client models from the previous round? This seems like introduces a lag, or gap between choosing the new threshold based on the state of the meta-model from the prior iteration, not the state of the model after incorporating the most recent client updates. 
2) Is there supposed to be a "for each checkpoint t in T" loop in the "On the Server" section? Section 3.2 "Step One" is the section in the manuscript that describes checkpoints being used, so it seems like this only applies in the local phase. Maybe I still don't understand the algorithm fully though.

The clarifications provided are encouraging. Allow me to reiterate them to confirm that my new understanding of the key points of the work.

A) (preamble in rebuttal) You create testbed scenario where clients have training datasets with some significant corruption/low quality data within them. You first show the impact of your intervention by comparing the results training on the raw data versus your corrupted data, and also include a baseline of training on just the remaining clean data, a reasonable upper bound on expected performance of an optimal filtering method. (This is made more clear as either s separate table as you show in the supplement, or by more clearly referring to "Raw Data", "Low-qual Data", and  "Oracle" as baseline measurements, or something other than "methods" because this is slightly confusing.

B) (point 4. in your rebuttal) You compare your data selection algorithm method to others in Table 1 in the original draft (Table 4 in the supplement) and show that it performs better than PPL, IDF, and DataInf in 2 out of 4 scenarios with the other strongest method being PPL. Please bold the strongest method in each column, even when it is not your own. Using the Table 1 in the manuscript (the horizontal line separating baselines from selection methods is good), this means bolding Ours in Mistral GPT-4 column, bolding PPL in Mistral Knowledge column, boldingOurs in Llama GPT-4 column, and bolding PPL in Llama Knowledge column. The caption should say "the best performing selection method (row) in the lower subtable is bolded for each benchmark evaluation (column)."

C) (points 1,2,3 in rebuttal) On Federated versus Model Merging... Can you explain how these are different with respect to your newly formalized algorithm in the supplement? I am not able to fully understand the reiterations of results in the rebuttal that discuss the differences because the draft does not make precise how the federated update or the model merging happens in context of the data selection process. (The model merging step, is now clarified in the supplement as a weighted average of the sent parameters $\theta'_k$ from clients. How would the federated update differ?) Point 2. in particular speculates that this is because of a key difference between how the data valuation scores are computed in the two settings - this is completely missing in the manuscript?