Instruction Mining: Instruction Data Selection for Tuning Large Language Models

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 02:01:41

Abstract: Large language models (LLMs) are initially pretrained for broad capabilities and then finetuned with instruction-following datasets to improve their performance in interacting with humans. Despite advances in finetuning, a standardized guideline for selecting high-quality datasets to optimize this process remains elusive. In this paper, we first propose InstructMining, an innovative method designed for automatically selecting premium instruction-following data for finetuning LLMs. Specifically, InstructMining utilizes natural language indicators as a measure of data quality, applying them to evaluate unseen datasets. During experimentation, we discover that double descent phenomenon exists in large language model finetuning. Based on this observation, we further leverage BlendSearch to help find the best subset among the entire datase. Experiment results show that InstructMining-7B achieves state-of-the-art performance on two of the most popular benchmarks: LLM-as-a-judge and OpenLLM benchmark.

Main paper: 
 introduction: 
 Large language models (LLMs) have demonstrated transformative capabilities, powering numerous applications with the strong ability in automatically generating responses according to human instructions (Ouyang et al., 2022; Peng et al., 2023; Chung et al., 2022; Scao et al., 2022) . However, it is hard sometimes for language models to capture the meaning of human instructions and respond to them even if they are pretrained with large amount of data. To counter this challenge, instruction tuning emerged as a paramount method in tailoring the behaviours of LLMs, which leverages instruction-following pairwise data (i.e., instruction data) during finetuning (Wei et al., 2021; Ouyang et al., 2022; Chung et al., 2022; Wang et al., 2022a) . A recent study LIMA has revealed that even a small amount of carefully selected high-quality instruction data can significantly improve model performance through instruction tuning (Zhou et al., 2023) . Nevertheless, LIMA still requires human experts to filter examples from extensive datasets, which is both time-consuming and expensive.In this paper, we propose INSTRUCTMINING, a novel method designed to automatically select high-quality instruction data for finetuning better LLMs. Achieving this objective necessitates a data evaluator capable of assessing the quality of instruction data without the intervention of human experts. Furthermore, the data selector is also indispensable for automatically identifying the most suitable subset of instruction data for finetuning LLMs. Nevertheless, quantifying the quality of instruction data without human expert is a non-trivial task. To address this problem, we employ the loss incurred by a finetuned model on the evaluation set a proxy for data quality. However, computing this inference loss necessitates the actual finetuning of a language model, a potentially time-consuming process. To overcome this obstacle, we introduce a set of selected natural language indicators (e.g., reward model score), capable of predicting the inference loss without the need for finetuning an LLM. This approach can efficiently provide an estimation of the dataset's quality.While our approach can assess and rank the entire dataset based on data quality, determining the most optimal subset for finetuning LLMs remains an unresolved challenge. A straightforward solution is to choose the top-K high-quality data samples, but selecting the appropriate K proves to be inherently difficult in practice. To address this complexity, we conducted a series of experiments exploring the relationship between data quantity and quality. Remarkably, as we continued to increase the subset size for finetuning language models, we observed the double descent phenomenon (Nakkiran et al., 2021) , as illustrated in Figure 1 . This observation signifies a transition in the primary determinant of Blue line refers to random selected data sizes w.r.t. model inference loss. Our optimization goal is to find the optimal point which results in the lowest inference loss.model performance from data quality to data quantity once the data size crosses a specific threshold. In such scenarios, focusing on an initial set of high-quality data points (e.g., K=10,000) is more efficient for identifying the optimal point than perusing the entire dataset. Given the cost-sensitive nature of pinpointing this optimal point, we employ BLENDSEARCH (Wang et al., 2021a) to automatically search for the best subset for our needs.We further substantiate the validity and scalability of INSTRUCTMINING by contrasting its performance with other state-of-the-arts across diverse benchmarks. Notably, INSTRUCTMINING enhances the performance of LLAMA-2-7B by 4.93 on the Huggingface OPENLLM benchmark. In addition, our finetuned models are able to generate equivalent or superior responses in 64.67% of instances, compared to VICUNA-7B-v1.5. Furthermore, INSTRUCTMINING contributes to heightened finetuning efficiency. The optimal INSTRUCTMINING model, finetuned on a mere 2.5% (i.e., 2,532 out of 100,000) of the highest-quality examples from the complete dataset, can achieve state-of-the-art on both LLM-AS-A-JUDGE (Zheng et al., 2023) and OPENLLM benchmarks.Our contributions are summarized as follows:• In this work, we pioneer the application of classical data mining techniques to enhance LLMs by autonomously selecting high-quality data. To realize this objective, we introduce INSTRUCTMINING, a method encompassing data assessment and selection processes.with an advanced searching algorithm, enabling the automatic assessment of data quality and identification of the optimal subset for finetuning language models. • Models finetuned with INSTRUCTMINING exhibit state-of-the-art performance on two of the most popular benchmarks: LLM-AS-A-JUDGE and Huggingface OPENLLM. Meanwhile, Utilizing less training data can effectively reduce both the training time and cost. 
methodology: 
 In this section, we provide a detailed description of our proposed method, INSTRUCTMINING. A procedure graph is provided in Figure 5 . Our method is composed of two parts, quality estimation and threshold search. We first introduce our method for estimating instruction data quality in Section 2.1. This is achieved by aligning the data quality with the inference loss of a fine-tuned model. Then we propose our evaluation rule along with the selected natural language indicators in Section 2.2. Finally, we present the observed double descent phenomenon and introduce a BLENDSEARCH-based data selector in Section 2.3.In this paper, we follow the superficial alignment hypothesis proposed by Zhou et al. (2023) that a model's knowledge is mostly learnt during pretraining, while instruction data teaches the model to follow a certain pattern when interacting with users. Hence, the quality of these instruction dataThe number of tokens in tokenized inputs.The number of tokens in tokenized outputs.The oasst-rm-pythia-1.4b reward model inference score of every pair in the dataset. (Köpf et al., 2023) Perplexity P P L The exponentiated average negative log-likelihood of response. (Dong et al., 2011) in SentenceBERT (Reimers & Gurevych, 2019) embedding space.The score of whether a response is like something a person would naturally say, provided by the UniEval (Zhong et al., 2022) dialogue model.The score of whether this response serves as a valid continuation of the previous conversation, provided by the UniEval (Zhong et al., 2022) dialogue model.The score of whether the response is understandable, provided by the UniEval (Zhong et al., 2022) dialogue model. could be viewed as its ability to efficiently steer language models in learning to generate responses in a particular manner. Based on this assumption, we further propose our instruction quality evaluation hypothesis as follows.Hypothesis 1 Instruction Quality Evaluation Hypothesis: Given an instruction dataset D, we finetune a language model on D, denoted as M f t . The instruction quality of D can be estimated through the inference loss of M f t on a evaluation dataset D eval .To ensure the inference loss provides a valid measure for evaluating data quality, the evaluation set should comprise a selected collection of unbiased and high-quality instruction-following samples.In particular, given an instruction-following dataset D, we finetune a base language model M using D with model training settings S. S normally refers to training batch size, epochs, etc. L refers to the loss function. The obtained finetuned language model is denoted as M f t . We define the dataset D's quality Q D|M,S as below, Q D|M,S ∝ -L(M f t , D eval )(1) where D eval refers to the high-quality and unbiased evaluation set, and ∝ means a direct proportion.According to Equation 1, we utilize the inference loss to evaluate instruction quality. However, finetuning an LLM for evaluation can be inefficient, since this process can take days of training. To solve this problem, we introduce a set of natural language indicators and use the indicators to predict the inference loss. In this paper, We have a set of indicators 1 . For a given instruction dataset D, we compute the corresponding indicator values I(D) = {I i (D), i = 0, • • • , n}. There exists a function F such that the aforementioned model inference loss L(M f t , D eval ) can be approximated using F (I(D)).I = {I i , i = 0, • • • , n}, summarized in TableThe relationship between the finetuned model inference loss L and these computed indicators can be formulated as in Equation 2. -Q !|#,In this paper, we assume that there exists a multivariate linear function of I that is proportional to the logarithmic loss. Consequently, Equation 2 can be reparameterized as Equation 3:log L(M f t , D eval ) ∝ L 0 + F {I(D)} ∝ L 0 + β 0 + β 1 I 1 (D) + β 2 I 2 (D) + • • • + β n I n (D) + ϵ (3)where β 0 denotes the linear constant, β i , i ∈ {1, • • • , n} represents a sequence of linear coefficients, and ϵ refers to the random error term.To investigate the relationship between these indicators and the overall dataset quality, it becomes necessary to accumulate experimental results to estimate the unknown parametersβ i , i ∈ {0, • • • , n}.In this study, we employ the Least Squares method (Björck, 1990) to estimate the parameters in the multivariate function. The Least Squares method is a standard approach in regression analysis for the approximate solution of overdetermined systems. The technique minimizes the sum of the square residuals, thus providing the optimal fit between the observed and predicted data in terms of reducing the overall prediction error. Our experimental results and analysis are detailed in Section 4.During experimentation, we observe that with larger data size, model performance first gets better and then gets worse. After the data size grows to a certain level, model performance gets better again. Further analysis are provided in Section 5.1. This phenomenon indicates that there is an optimal point where better performance can be obtained with a smaller amount of data. Hence, searching for the best data size is important for finetuning a better language model.To achieve our objective, we employ BLENDSEARCH (Wang et al., 2021a) in Flaml (Wang et al., 2021b) library to determine the optimal data size. BLENDSEARCH effectively combines global and local optimizations by Bayesian optimization and different local search threads, making it efficient for searching cost-related hyperparameters and complex search spaces with local optima. In our context, we leverage a logarithmic uniform distribution to randomly sample the dataset size, treating the dataset size as the experiment's cost since the training time scales proportionally with the dataset size. The search goal is to minimize the loss on the evaluation set. 
experiment settings: 
 Our experiments mainly focus on two goals. The first goal is to estimate the unknown parameters in the proposed INSTRUCTMINING rule. The second one is to evaluate and analyze the performance of INSTRUCTMINING over varied finetuning scenarios. Section 3.1 elaborates rule estimation empirical study design. Section 3.2 details the datasets we use for conducting both estimation and evaluation experiment. Section 3.3 elaborates the finetuning settings we used for estimation and evaluation.The general procedure of our rule estimation experiment is shown in Figure 5 . To estimate the correlation between the evaluation loss and indicators I, we need to get datasets of different indicator values. To achieve this, we first select several commonly used datasets with different presumed quality levels and fuse them together with randomly sampled percentages to create finetune datasets. These sampled finetune datasets should encompass varying proportions of preassumed high quality and low quality examples. For each of these sampled datasets D i , we compute its respective indicator values I(D i ) and finetune a base language model M using D i . Following Equation 1, the quality Q Di for dataset D i is approximated using the evaluation loss of finetuned model M f t,i on a fair evaluation dataset D eval . Following the collection of a range of results correlating Q Di with I(D i ), we undertake a statistical regression analysis to discern relationships within the dataset.Candidate datasets for rule fitting. In order to create diverse training datasets, we collect data from various sources. This approach ensures that the datasets exhibit differences in quality and maintain diversity among sources. For this purpose, we have selected the following datasets as candidate datasets: ALPACA (Taori et al., 2023) , OPEN-ASSISTANT (Köpf et al., 2023) , STACKEXCHANGE, and WIKIHOW. Due to the varying formats, sizes, and distributions of different datasets, we have applied distinct processing procedures to each dataset. Table 2 provides an overview of the candidate training datasets after preprocessing. As mentioned in section 3.1, we merged candidate training datasets, resulting in each dataset containing 1,000 instruction-output pairs. We generated a random number r i for each dataset and randomly selecting 1000 * r i / i r i samples from each dataset for combination. Besides, considering the significant size difference between ALPACA and other candidate datasets, we randomly sampled 2,000 data examples from ALPACA to maintain scale consistency across all the candidate datasets.Test set for rule fitting. To address real-world requirements, we use the SELF-INSTRUCT dataset Wang et al. (2022a) , which contains 252 instructions as rule-fitting test set. Considering evaluation efficiency, we randomly sampled 80 instructions from the whole dataset as our evaluation set. In our study, we employed gpt-4 from OPENAI to generate response for each instruction.Datasets for rule evaluation. We further test INSTRUCTMINING by using it to select high-quality examples from unseen datasets for finetuning large language models. During evaluation, we mainly use OPENORCA and DOLLY-15K as the two unseen candidate datasets. For OPENORCA, given its extensive size, we randomly select 50,000 examples from OPENORCA-GPT3.5 and 50,000 examples from OPENORCA-GPT4 for experimentation (henceforce refered to as OPENORCA). To make sure that our method does not overfit on the SELF-INSTRUCT evaluation set, we use the gpt-4 labeled MT-BENCH dataset (Zheng et al., 2023) as an unseen evaluation set. To be noticed, since our candidate and evalation datasets does not include multi-turn examples, when evaluating on MT-BENCH, we only use the first turn of MT-BENCH dataset.We conduct all finetuning experiments on the same base model LLAMA-2-7B (Touvron et al., 2023) . All finetuning datasets during estimation phase are of the same size, 1,000 examples in each. We run model finetuning for 3 epochs, with per step batch size set to 128. We use Adam with β 1 = 0.9, β 2 = 0.999, and cosine learning rate scheduler starts from 2e -5, decays to 0. Each model finetuned during estimation phase is evaluated on the evaluation dataset mentioned in section 3.2. We run all finetuning and evaluation experiments on a NVIDIA A100 80G GPU cluster, with 8 A100 GPUs used in each experiment. 
instructmining parameter fit: 
 Following Section 3.1, we randomly sampled 129 subsets from the entire data pool with different percentages. These subsets are then used to finetuned 129 corresponding language models. We collect the inference loss and indicator values for each subset. To select the optimal rule, we first choose regression results with the highest R 2 , and then prioritize the rule with the most significant p values. The detailed regression result is available in Table 9 . Based on this result, we delineate our estimated evaluation function, which is articulated as Equation 4. Accordingly, reward score and unieval scores appear to be the most significant indicators in the quality rule. This estimation result reveals that U nd is in negative correlation with data quality, while the other three indicators are of positive correlation with data quality. BlendSearch results. In response to the performance decline-rise phenomenon with increasing training data, we conducte a BLENDSEARCH within a range of data sizes from 512 to 10,000. Our prior empirical findings suggest that a maximum dataset size of 10,000 is sufficient for optimizing the data size. Figure 2 details the search procedure from the perspective of steps.EQUATIONLoss results. We first evaluate the finetuned models using inference loss on LLM assessment. We next compare our best model with other state-of-the-art models using LLM-AS-A-JUDGE. We let GPT-4 choose the better answer between two responses generated by two different models. According to the results presented in Figure 3 , our model is able to generate better or equal results in 64.67% of the cases compared to VICUNA-1.5-7B. We also let GPT-4 assess the model from different perspectives. According to Figure 3b , our model significantly improves the original LLAMA-2 model's ability in writing, roleplay, humanity, STEM, extraction and coding.OpenLLM benchmark results. Besides, we further test our finetuned models on the widely used OPENLLM benchmark (Gao et al., 2021) . OPENLLM benchmark is composed of four widely used general question answering benchmarks, ARC (Clark et al., 2018) , HELLASWAG (Zellers et al., 2019) , MMLU (Hendrycks et al., 2020) and TRUTHFULQA (Lin et al., 2021) In this section, we present further experimental findings on OpenOrca dataset. In previous experiments, we find out that a language model's performance can be influenced by both finetuning data quality and quantity. When data quantity increases, generative language models' performance does not promise to become better. This phenomenon suggests a balance between data quantity and data quality. Results are presented in Figure 4 . This reveals some interesting emergent phenomena when finetuning large language models. We detail the observed phenomena below.Phenomenon 1 Non-monotonicity exists in language model performance. As we increase the training data size, language model performance first becomes better then gets worse. When data size increases to a certain level, performance becomes better again.Based on Figure 4 , we observe that the performance first improves as the training data size grows. Then, after the data size grows to around 10,000, loss begins to increase, meaning that language model performance worsens. Finally, as data size continues to grow, language model's performance improves. This phenomenon is similar to the double descent phenomenon (Nakkiran et al., 2021 ) that non-monotonicity exists with varying numbers of training samples. In our experiment, we observe that this phenomenon not only exists in vanilla language model training but also in large generative language model finetuning.Phenomenon 2 Balance point exists between randomly selected and quality selected data. As data size grows, data quality becomes a less important factor for model performance.Given Figure 4 , we find out that when data size grows to a certain point, the performance curve of selected data will always intersect with the random one. Besides, the distance between the two decreases as the data size increases. This phenomenon indicates that data quality measure can help improve model performance at first. However, data quality becomes less important when data size grows to a certain level. 
robustness: 
 To further explore the effectiveness of INSTRUCTMINING, we evaluate it across three different finetuning settings: (1) Different base models. We change the original base model LLAMA-2-7B into LLAMA-1-7B to test whether our method is scalable to other models.(2) Different model sizes.We change the original 7B model size into 13B to test whether our method is scalable to other model sizes.(3) Parameter efficient settings. LORA (Hu et al., 2021) , a parameter efficient method, is widely used when finetuning a large language model to help save GPU memory usage. We also test our method with LORA settings to see whether INSTRUCTMINING is scalable to parameter efficient finetuning. Results are presented in Table 6 . As the data shows, INSTRUCTMINING rule can be applied to various base models, model sizes and parameter efficient settings.  
related work: 
 Instruction tuning. Recent studies have explored instruction tuning as a method for fine-tuning LLMs, enhancing their ability to generalize to unseen instructions (Wei et al., 2021) . Reinforcement learning from human feedback (RLHF) is popular method that aligns language models with human intent (Ouyang et al., 2022) . To further improve instruction tuning, some work choosed to increase the size of the data (Honovich et al., 2022; Wang et al., 2022a) . Besides, Zhou et al. (2023) demonstrated that utilizing a smaller volume of high-quality instruction data can still produce effective models.Instruction evaluation. The field has experienced growth with the publication of numerous instruction datasets (Taori et al., 2023; Köpf et al., 2023; Honovich et al., 2022) . Chung et al. ( 2022) first combined multiple datasets to augment both the quantity and diversity of instruction data, achieving notable performance gains. Some newest works suggest that enhancing instruction diversity can also significantly improve instruction tuning performance (Iyer et al., 2023; Wang et al., 2023; 2022b; Longpre et al., 2023) . Meanwhile, Gunasekar et al. (2023) To provide more details on natural language indicators, we present further descriptive analysis results on these indicators. We calculate the indicator values across the 129 sampled subsets. Figure 6 presents indicator distribution graphs.In addition, to make sure that statistical regression is valid in this paper, we perform Kolmogorov-Smirnov(KS) test on every indicator. Test results are provided in To investigate whether inference loss can serve as a suitable indicator of model capability and data quality, we conduct further finetuning experiments. We randomly select 1 1,000 examples from four datasets with different quality levels and finetune LLAMA-2-7B model on the selected datasets. We also finetune LLAMA-2-7B and LLAMA-2-13B models using 1,000 examples from ORCA-fused dataset. Results are provided in Table 8 . As shown in the table, GPT-4 labeled datasets tend to yield lower loss on the two evaluation sets. Finetuned models with larger model size also yield lower loss on the evaluation sets. Hence, we suppose that evaluation loss can serve as a suitable indicator of model capability and data quality. In this section, we present our analysis of other emergent phenomena in this paper. Except for regression test, we further conduct correlation test between indicator values and loss values. We plot regression analysis results in Figure 7 . We detail other discovered phenomena below.Phenomenon 3 Perplexity is negatively correlated with data quality.In general, a higher perplexity score corresponds to increased inference loss, suggesting a potential decline in data quality. Typically, elevated perplexity signifies that the data instance was infrequently encountered during pretraining. Such instances may fall outside the bounds of conventional natural language.Phenomenon 4 Reward score is positively correlated with data quality.In this paper, we employ the oasst-pythia reward model to assign reward scores to individual data examples. Reward models are typically trained using human preference labels, suggesting their capacity for human-like evaluations. Thus, examples preferred by the reward model are typically of high quality.Phenomenon 5 Output length is negatively correlated with data quality.When the number of tokens increases, loss tends to increase which means that data quality tends to decrease. This can be due to the maximum sequence length in language models. LLAMA-2-7B has 4096 maximum sequence length. For some very long instances, it is possible that the sentence is truncated in the middle during preprocessing.Phenomenon 6 Understandability is negatively correlated with data quality.Unieval-understandability normally represents the complexity of a sentence. When the complexity increases, data quality decreases. This is possibly due to some translation-related instructions in the dataset, which requires the unieval model to command multilingual assessment ability. However, these examples are not common in our evaluation set, which might result in the negative correlation between the two.As shown in Table 9 , we present our statistical regression result for our estimated rule of Equation 4in Table 9 . We select the indicators first through R 2 and then through p value, to make sure that our regression result is valid and effective. According to the regression result, Rew, N at and U nd appears to be the most prominent indicators in this rule. This estimation result reveals that U nd is in negative correlation with data quality, while the other three indicators are of positive correlation with data quality. 10 , Table 11 and Table 12 .Example 1Instruction: Teacher: Given reviews from Amazon, classify those review based on their content into two classes: Negative or Positive. Now, understand the problem?Input: This is not waterproof. Its GARBAGE where batteries go in broke, while trying to adj. DO NOT BUY. You turn the end to control it, that loosens the battery cover where water gets in so how is it waterproff????Output: Yes, I understand the problem. The task is to classify the given review as either negative or positive based on its content. In this instance, the review contains negative content. The reviewer is expressing dissatisfaction with the product and warning others not to buy it. The reviewer specifically mentions that the product is not waterproof and that the battery cover is broken, which allows water to get in. The use of all caps and exclamation marks also indicates a strong negative sentiment. Therefore, I would classify this review as negative.InstructMining score: 0.7902 (Lower, which means better) https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm 
 
 Appendix:  


Decision: This paper proposes Instruct Mining, an approach to identifying high-quality instruction tuning data instances. The idea is to use a model to select high-quality instructions as a function of features of the language (e.g., length). By selecting fine-tuning data under this model, good performance can be realized with less data. 

Overall, this is a nice contribution in the space of (pre-training) data centric LLM work, showing the critical importance of data quality.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: The paper introduces INSTRUCTMINING, a novel method that automates the selection of high-quality instruction-following data for fine-tuning large language models (LLMs). This innovation addresses the significant challenge of manually selecting instructional data, which is both time-consuming and costly.
strength_weakness: strength_weakness: Firstly, I'm somewhat skeptical about the necessity of the least squares method. Could the authors include additional ablation studies, such as simply combining individual indicator functions to evaluate the quality of each instruction dataset? Moreover, the authors should compare their method against some baselines, like employing the ChatGPT reward score [1], or using perplexity-based metrics [2].

[1]. AlpaGasus: Training a Better Alpaca with Fewer Data
[2]. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning
paper_summary: paper_summary: This paper propose to select the high-quality instruction-tuning data to better fine-tune LLMs. This is an important problem and the proposed methodology is very intuitive --- quality is sometimes over quantity. The authors developed an instantiation of the methodology by defining instruction quality and applying an existing method for selection. The experiment results show some remarkable performance.
strength_weakness: strength_weakness: - Only LLAMA-based LLMs are evaluated. More models (e.g., Mistral ones) will be helpful to justify that the proposed method can generalize well for different LLMs.
- The Hypothesis 1 in my perspective is basically saying that a validation set loss/performance shall be adopted for the evaluation of the training data quality. This is a common practice in hyper-parameter tuning. Selecting a subset of training data can be viewed as a kind of hyper-parameter tuning too. So this formulation is not very novel.
- Also, following this "validation set loss" idea, the double descent phenomenon is somewhat as expected?
- Breaking down the loss into a small set of pre-defined indicators following a linear combination seems overly simple?
- This paper itself doesn't propose a new selection method. Instead it applies an existing method BLENDSEARCH.
paper_summary: paper_summary: The paper proposes a method for automatically selecting high-quality instruction-following data for finetuning large language models. The method combines data assessment and selection processes, utilizing natural language indicators to evaluate data quality and identify the optimal subset for finetuning language models. The paper also discusses the double descent phenomenon observed in generative language models during finetuning.
strength_weakness: strength_weakness: Weakness:
1) While the paper presents results comparing the proposed method with other state-of-the-art models, a more detailed comparative analysis with a wider range of existing methods could provide additional insights into the method's performance and effectiveness.

Comments: comment: Thanks for your thoughtful feedback. Sorry for the late response. For your concerns, please find our responses below.

---

Q.1 Only LLAMA-based LLMs are evaluated. More models (e.g., Mistral ones) will be helpful to justify that the proposed method can generalize well for different LLMs.
 
Here are the ablation results on Falcon-7b based models.
| Model     | Data              | Loss@mt-bench | Loss@self-instruct |
| --------- | ----------------- | ------------- | ------------------ |
| Falcon-7B | Dolly-selected 1k | 1.1344        | 1.2780             |
| Falcon-7B | Dolly-random 1k   | 1.2396        | 1.3723             |
| Falcon-7B | Dolly-all         | 1.3028        | 1.3960             |


We also got results for the Mistral-7B models.

| Model      | Data              | Loss@mt-bench | Loss@self-instruct |
| ---------- | ----------------- | ------------- | ------------------ |
| Mistral-7B | Dolly-selected 1k | 1.0528        | 1.2483             |
| Mistral-7B | Dolly-random 1k   | 1.1107        | 1.3365             |
| Mistral-7B | Dolly-all         | 1.1494        | 1.3824             |


We finetuned these models using the same selected 1000 dolly data, which we used in the paper to finetune LLaMA-2-7b. 

---

Q.2 a validation set loss/performance shall be adopted for the evaluation of the training data quality which is not very novel.

We conducted multiple experiments using various validation sets to learn the relationship between indicators and validation set loss. As a result, we derived a linear function that can be applied to each individual data sample for data selection. We were pioneers in applying this method to the instruction tuning task, not just using validation loss to evaluate data quality.

---

Q.3  Following this "validation set loss" idea, the double descent phenomenon is somewhat as expected?

We are the first to acknowledge the existence of the double descent phenomenon in instruction tuning.

---


Q.4 Breaking down the loss into a small set of pre-defined indicators following a linear combination seems overly simple?
Our proposed Instruction mining can incorporate other functional assumptions or methods to obtain the function. In our paper, we use the least squares method to derive linear functions, providing a clearer understanding of the resulting functions. The function type is just one aspect of our approach. We also demonstrate potential linear relationships between certain metrics and losses.

---

Q.5 applies an existing method BLENDSEARCH.

We are the first to utilize BLENDSEARCH for data selection in instruction tuning. BLENDSEARCH serves as the search solution in our method.


---

About the novelty in our paper.

Our paper is proposing a framework to do data selection in instruction tuning which could help improve the model performance. Our method is the first to combine these to do data selection on instruction tuning.
Thank you for your review and for your thoughtful feedback, sorry for the delay in responding.

We have the result of LIMA-7b model (LLaMA-2-7B finetuned with LIMA data) and results from [1]. Its OpenLLM benchmark scores are shown in the table below. 

|Model | Datasize | loss@self-instruct| loss@mt-bench| OpenLLM Avg| ARC| Hellaswag| MMLU|TruthfulQA|
|---|---|---|---|---|---|---|---|---|
|LIMA-7b|1000|0.9947|0.7604|0.5533|0.5563|0.8009|0.4371|0.419|
|InstructMining-7b|1000| 0.9576 | 0.7114 | 0.562525  | 0.5589 | 0.7877 | 0.4299 | 0.4736 |
|15% Alpaca-7b[2]| * | * | * | 0.5637 | 0.5742 | 0.8068 | 0.4640 | 0.4095 |

reference:
[1]. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning
Thanks for your response. For your concerns, please find our responses below. Apologize for the delay in responding.

 --- 

Q.1 the necessity of the least squares method, and additional ablation studies, such as simply combining individual indicator functions to evaluate the quality of each instruction dataset

We use the least squares method to estimate parameters in multivariate functions due to its simplicity, explanatory power, and adaptability to various data types. Beyond the parameters detailed in the paper, the process of finding a usable function is also significant, allowing us to potentially replace the least squares method with other functions.

 --- 

Q.2 compare their method against some baselines
For, Alpagasus [1], However, It's expensive using chatgpt to give a score on large scale data. One of the advantages of the method proposed in our paper is that it is less costly.
We have the result of LIMA-7b model (LLaMA-2-7B finetuned with LIMA data) and results from [2]. Its OpenLLM benchmark scores are shown in the table below. 

|Model | Datasize | loss@self-instruct| loss@mt-bench| OpenLLM Avg| ARC| Hellaswag| MMLU|TruthfulQA|
|---|---|---|---|---|---|---|---|---|
|LIMA-7b|1000|0.9947|0.7604|0.5533|0.5563|0.8009|0.4371|0.419|
|InstructMining-7b|1000| 0.9576 | 0.7114 | 0.562525  | 0.5589 | 0.7877 | 0.4299 | 0.4736 |
|15% Alpaca-7b[2]| * | * | * | 0.5637 | 0.5742 | 0.8068 | 0.4640 | 0.4095 |

We also compared our model and LIMA model using LLM-as-a-judge. Results are listed in the table below
.
| Model | Win Rate | Lose Rate | Tie Rate | Adjusted Win Rate |
|---|---|---|---|---|
|InstructMining-7b(vs LIMA-7b)| 0.4875 | 0.1063 | 0.4062 | 0.6906 |


reference:
- [1]. AlpaGasus: Training a Better Alpaca with Fewer Data
- [2]. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning