G-DIG: Towards Gradient-based DIverse and hiGh-quality Instruction Data Selection for Machine Translation (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable abilities in general scenarios. Instruction finetuning empowers them to align with humans in various tasks. Nevertheless, the Diversity and Quality of the instruction data remain two main challenges for instruction finetuning. With regard to this, in this paper, we propose a novel gradient-based method to automatically select high-quality and diverse instruction finetuning data for machine translation. Our key innovation centers around analyzing how individual training examples influence the model during training. Specifically, we select training examples that exert beneficial influences on the model as high-quality ones by means of Influence Function plus a small high-quality seed dataset. Moreover, to enhance the diversity of the training data we maximize the variety of influences they have on the model by clustering on their gradients and resampling. Extensive experiments on WMT22 and FLORES translation tasks demonstrate the superiority of our methods, and in-depth analysis further validates their effectiveness and generalization.1

heading: 1 Introduction
text: Large Language Models (LLM) have revolutionized the field of Natural Language Processing with their strong abilities in general language understanding and generation (OpenAI, 2023; Achiam et al., 2023). To enable this strong ability, instruction finetuning has been proposed to better align language models (Wei et al., 2021; Chung et al., 2022; Ouyang et al., 2022). Significant progress has been made on collecting extensive instruction finetuning data for better aligning LLMs to produce helpful responses (Chung et al., 2022).
†The work was done when the author was an intern at ByteDance.
‡Corresponding author. 1Code is available at https://github.com/xypan0/
G-DIG
We argue that Diversity and Quality of the instruction data present a pair of challenges for instruction finetuning. Zhou et al. (2023) have demonstrated that a model trained with a limited, carefully curated dataset composed of high-quality and diverse examples outperforms models trained on larger, more extensive datasets during instruction finetuning. Subsequently, various methods have been proposed to automatically select high-quality and diverse training data from the large pool of instruction finetuning datasets (Chen et al., 2023a; Cao et al., 2023). Yet, these methods often rely on another model to decide quality or diversity, neglecting the inherent model behavior and strong ability of the LLM itself.
To this end, we propose G-DIG, a novel Gradient-based method to automatically select Diverse and hiGh-quality instruction finetuning data for machine translation. We use influence function (Koh and Liang, 2017), a gradient-based method that quantifies the impact made by individual training samples. Concretely, we (1) measure the response quality of each training sample with the influence score of the training sample on test instances and (2) enhance the diversity of the training data by maximizing the variety of influences they have on the model.
Specifically, we hypothesize that high-quality data should have positive influences on high-quality test samples. Hence, we first manually create a small set of high-quality seed data and then automatically select high-quality data that have positive influences on seed data. Moreover, we utilize Kmeans clustering algorithms to cluster training data with similar influences, using gradients representing their influences on the model. Unlike existing methods that introduce an external model to decide quality and diversity, our methods directly use model gradients, which capture the model behavior through learning algorithms and back to the training data.
15395
We conduct experiments on Zh ⇒ En and De ⇒ En translation tasks. Specially, We collect large candidate pools and manually construct two small sets of seed data. We then finetune different LLM backbones on various sizes of selected subsets and compare their performances with different selective methods and existing SOTA LLMs. Under a thorough comparison in a range from 1k to 64k selected samples, our proposed method not only surpasses the baseline selective methods but also achieves competitive performance against SOTA models. Extensive experiments and in-depth analysis emphasize the need for data selection and demonstrate the effectiveness and generalization of our proposed methods.

heading: 2 Related Work
text: LLM for Machine Translation. Due to their strong in-context learning and instructionfollowing abilities, powerful LLMs like GPT-4 have achieved remarkable progress in machine translation, with comparable performance to the top systems on the WMT translation task (Zhu et al., 2023; He et al., 2023; Raunak et al., 2023). To fully leverage LLMs’ translation ability, various methods have been proposed, including in-context translation exemplar selection (Garcia et al., 2023; Lin et al., 2022; Zhang et al., 2023a; Agrawal et al., 2022), prompt optimization (Jiao et al., 2023) and decoding strategies (Zeng et al., 2023a).
The aforementioned studies all focus on the inference stage optimization, while another line of work focuses on instruction tuning the LLMs for better translation performance. Xu et al. (2023) proposes to first finetune the model on monolingual data and then on high-quality parallel data. Li et al. (2024b) trains the model to elicit translation ability by multilingual instruction tuning. Li et al. (2024a) proposes to create high-quality instruction-tuning data from larger models by a patching mechanism. Chen et al. (2023b) improves the model instruction understanding by adding a global instruction representation and improves model faithfulness by comparing over-translation and misstranslation results with the correct translation. Zeng et al. (2023b) proposes a novel framework using examples in comparison to teach LLMs to learn translation. However, all these methods neglect the importance of instruction finetuning data quality and diversity in machine translation. And in this paper, we propose a novel approach to enhance the quality and
diversity of translation data.
Traning Data Quality and Diversity. Various studies evident that the quality and diversity of instruction finetuning data predominate the performance of LLMs (Zhou et al., 2023; Touvron et al., 2023; Li et al., 2023). For example, Zhou et al. (2023) manually curated a small, high-quality instruction set to elevate the model’s instruction following power. Although the methods in (Zhou et al., 2023) rely heavily on human effort, they motivate research aiming to automatically obtain high quality instructions. Cao et al. (2023) propose to score the quality of each instruction by combining several language indicators using a linear model. However, all these indicators rely on external models. Besides, Du et al. (2023) present a comprehensive approach for selecting high quality and diverse instruction based on reward model score and semantic diversity. Still, their methods rely heavily on external models and overlook the direct impact that finetuning data has on the model. Remarkably, Li et al. (2023) propose a self-guided approach to select difficult instructions with the guidance of the LLM itself. Admittedly, their methods are free of any external models. However, they select training examples more associated with necessity and complexity than quality, and they overlook the importance of finetuning data diversity. Despite the efforts these methods have made to automate instruction selection, they either overlook the importance of the quality and diversity of training data or rely on external models for judging. Significantly different from them, our methods take both quality and diversity into consideration and are free of external models.
Gradient-based Data Selection. Data selection with influence function has been widely studied in NLP. Lam et al. (2022) propose to identify erroneous training data by using synthetic noisy data, showing that vanilla influence functions are not sufficient for good retrieval performance. On the contrary, we select high-quality and diverse finetuning data with the aid of gradient information. Also, we show that our use of influence function is capable of selecting high-quality data. Akyurek et al. (2022) demonstrate the potential of using gradient information to trace factual knowledge in language models back to the training data. Nevertheless, their practical applications remain understudied. Methods for scaling up influence function (Schioppa et al., 2022) and explaining black
box predictions of NLP models have been proposed (Han et al., 2020). Remarkably, the concurrent work (Xia et al., 2024) proposes a similar method to select task-specific LLM finetuning data by estimating influences for training data. Different from them, we select high-quality finetuning data for machine translation. And we show that our method is capable of capturing a higher-level concept (i.e., the quality of training data).

heading: 3 Methods
text: In this section, we describe our gradient-based method to select high-quality and diverse training data for instruction finetuning, as displayed in Figure 1. Our methods consist of two parts: 1) high-quality data selection with the influence function (§ 3.1) and 2) diverse data selection with gradient-based clustering (§ 3.2). For high-quality data selection, importantly, we utilize the influence function to quantify the impact of individual training on the test sample. For diverse data selection, we use gradient distance to assess the overall diversity of the instruction training data.

heading: 3.1 High-quality Data Selection
text: In this section, we detail our approach for selecting high-quality training data for machine translation. Intuitively, if a training example significantly benefits the model to generate high-quality outputs, it is likely to be of high quality itself. Consequently, we first manually curate a small set of high-quality translation data that we refer to as the seed data to form a criteria for evaluating training data quality. Then we select training data that aids the model in generating high-quality seed data.
Concretely, we employ Influence Function (IF) (Koh and Liang, 2017) to quantify how a training example zm influences the model’s behavior on a test example zt. In our influence function setting, we start with the following finetuning objective:
θ∗ := argmin θ
1
n
n∑
i=1
L(zi|θ), (1)
where θ is the model parameter, zi = (xi, yi) is the i-th prompt-response pair and the loss L is simply
the language modeling loss of the response solely:
L(x, y) = − logP (y|x)
= − log T∏
j=1
p(yj |x, y<j), (2)
where xj denotes the j-th token of x. Then influence function calculates the influence of zm on zt by:
I(zm, zt) = −∇θ|θ∗L(zt)⊤H−1θ∗ ∇θ|θ∗L(zm), (3) where θ∗ is the optimal model parameter of finetuning in (1) and Hθ∗ = ∇2θ 1n ∑n i=1 L(zi|θ∗) is the Hessian of the training objective at θ = θ∗. Though the calculation of the influence function is complex, the most significant aspect for readers is that if the scalar I(zm, zt) is negative, training the model on zm reduces the model’s loss on zt. In this case, zm is considered to be helpful for the model to generate zt. In our implementation of IF, we use the gradients of model’s Multilayer Perceptron (MLP) parameters in {3, 6, 9, 12, 15, 18, 21, 24, 27, 30}-th layers to speed up calculation. And we average the gradients of each token to form a vector. Also, we use Kronecker-Factored Approximate Curvature (KFAC) (Martens and Grosse, 2015) to approximate Hessian for reducing memory consumption. We detail the derivation of IF and our modification for fitting it into LLM fining tuning in Appendix A.
Correspondingly, as depicted in Figure 1, our proposed high-quality data selection method consists of two steps: (1) calculating influences and (2) selection. We select training examples in our candidate pool Draw that exert beneficial impact on all samples in the seed data, i.e., example zm ∈ Draw meets:
∀zt ∈ Dseed, I(zm, zt) < 0. (4)
Thus, we select training data that contributes to the model’s high-quality generation.
In practice, we find that the seed instruction dataset Dseed of size 256 suffices for selecting high quality instructions. Hence, we set the size of Dseed to 256 in our implementation. As the focus of this paper is to select high-quality finetuning data for translation task, we randomly select 256 parallel texts from the validation set and have them revised by human translators. We select high-quality data from our candidate pool, for
which we collect publicly available WMT22’s parallel texts. Throughout our implementations, we use the prompt template: Translate the following text into {trg_lang}.\n\nText:\n“{src_text}”, where {trg_lang} are target languages such as "English" and {src_text} are the source text. And the response is simply the target text. We detail our data preparation in Section 4.

heading: 3.2 Diverse Data Selection
text: After obtaining high-quality translation pairs, we further ensure the diversity of the training data. To ensure coverage of different translation patterns, we propose to use gradient similarity to assess the diversity. Specifically, we consider the gradient of the response loss in equation (2) with respect to the final MLP layer and average them out on all the tokens. We utilize the Euclidean distance as the similarity measure.
To maximize the diversity of training data influences, we cluster on the gradients of training examples to obtain different patterns. Then we sample uniformly from the clustering result to ensure the diversity of training data. Moreover, we employ K-means as the clustering algorithm due to its ability to process large-scale datasets. Furthermore, to speed up and reduce memory, we use Random Projection as the dimensionality reduction method to reduce the dimension of the gradients to 400 (Bingham and Mannila, 2001). In practice, we cluster the training data into 512 clusters.

heading: 4 Experiment Setup
text: Datasets. We conduct experiments on Zh ⇒ En and De ⇒ En tasks. We collect separate candidate pools for different translation directions. Specifically, our Zh ⇒ En pool is composed of 1.9 million examples sampled from WMT22’s ParaCrawl v9, News Commentary v16, UN Parallel Corpus V1.0, WikiMatrix and Back-translated news and our De ⇒ En pool contains 256k examples sampled from WMT22’s Europarl v10, ParaCrawl v9, News Commentary v16 and WikiMatrix. We split 512 examples to validation sets for evaluation. We test our methods on the latest WMT22 test sets from the news translation track of WMT22 competition2 and Flores-101 dev-test split (Goyal et al., 2022). The WMT22 Zh ⇒ En and De ⇒ En test sets contain 1875 and 1984 samples, respectively.
2https://github.com/wmt-conference/wmt22-newssystems
And the Flores-101 dev-test split contains 1012 samples for each Zh ⇒ En and De ⇒ En.
Implementation Details. We use Baichuan2-7B (Baichuan, 2023) for Zh ⇒ En and Llama2-7B for De ⇒ En. For all finetuning experiments, we adopt the same setting. The finetuning process lasts for 3 epochs with an initial learning rate of 1e− 5 and a global batch size of 64. The instruction template we use is Translate the following text into {trg_lang}.\n\nText:\n“{src_text}”, where {trg_lang} are target languages such as "English" and {src_text} are the source text. The models are evaluated every 10 steps. We use the checkpoint with the smallest loss on the valid set for the final test. During the inference, we use beam search as the decoding strategy with a beam size of 4. The training and inference of 7B size models are conducted on 16 NVIDIA A100 80GB GPUs with DeepSpeed ZeRO-3 Offload.
Evaluation. For automatic evaluation, we use the widely used metrics BLEU (Papineni et al., 2002), BLEURT (Sellam et al., 2020), and COMET (Rei et al., 2020). We use ScareBLEU3, BLEURT-20 (Pu et al., 2021) and Unbabel/wmt22-comet-da4 in the evaluation implementations.
3https://github.com/mjpost/sacrebleu 4https://huggingface.co/Unbabel/wmt22-comet-da
Baselines and Comparisons. In order to demonstrate the superiority and effectiveness of our methods, we compare our G-DIG to validate the performance of our overall approach. Also, we assess its variant G-DIG w/o Diversity (G-DIG without enhancing the data diversity) to solely investigate our high-quality data selection module. For comparisons, we consider the following baselines:
• Bayling-13B (Zhang et al., 2023b), an English / Chinese LLM based on Llama with superior translation capabilities.
• BigTranslate-13B (Yang et al., 2023), a multilingual LLM based on Llama with the capability of translating over 100 natural languages.
• TIM (Zeng et al., 2023b). We present the results of BLOOMZ-7b-mt and Llama2-7b trained with TIM-Full-LM-based Bad Output for WMT22 test sets and FLORES respectively.
• Model trained on the random subset. To emphasize the need for instruction selection, we choose k random instructions to form a finetuning subset. We finetune the LLM on the selected subsets and report their performances.
• Model trained on the reward subset. We also compare our method with the existing
in red .
selection method. We use the commonly used reward model-based method for selecting high-quality training data (Du et al., 2023; Cao et al., 2023). Specifically, we follow (Du et al., 2023) to use the reward-modeldebertav3-large-v25 to score each instruction and select the top k instructions with the highest score to form the finetuning subset.

heading: 5 Experimental Results

heading: 5.1 Main Results
text: In this section, we present our main experimental findings. We start with comparing our methods with baselines on various amounts of training data in Figure 2. Then we compare our best results with SOTA models and baselines in Table 1. Furthermore, we conduct human evaluation and present the results in Table 2. We show that our approach is superior in terms of effectiveness and robustness.

heading: Our Methods Improve the Translation Perfor-
text: mance Across Various Amount of Training Data. In order to demonstrate the scalability of our method in terms of the amount of selected training data, we present the results as a function of the amount of training data from 1000 to 64,000 in Figure 2 for Zh ⇒ En and De ⇒ En translation. Notably, our G-DIG model consistently surpasses the random model across all metrics and dataset sizes for Zh ⇒ En translation. Also, for De ⇒ En translation, our G-DIG outperforms the random
5https://huggingface.co/OpenAssistant/reward-modeldeberta-v3-large-v2
baseline in almost all cases. Statistical analysis results in Appendix C further suggest our methods excel the random baseline. These results demonstrate the efficacy and robustness of our methods in terms of the amount of selected data. Meanwhile, we also observe that the quality and diversity of finetuning data dominate the performance of LLM. Specifically, for both Zh ⇒ En and De ⇒ En translations we can see that models trained on 1000 examples selected by our G-DIG outperform the models trained with 64k randomly selected examples.

heading: Our Methods Surpass Baselines and Achieve Comparable Results with SOTA Models. To
text: see how our method performs compared with baselines and SOTA models, in Table 1 we present the
results of SOTA models, our best and corresponding baselines. Specifically, our G-DIG achieves its best results at 4000 training examples for Zh ⇒ En translation and 64k for De ⇒ En translation. For baselines comparisons, we observe that our methods surpass baselines in almost all cases, demonstrating the effectiveness of our approach. For SOTA models comparisons, our 7B models achieve comparable results with TIM-7B and even better results compared with Bayling-13B, BigTranslate13B and NLLB-54B.
Our Methods Align the Model Better Compared with Random Baseline. We further conduct human evaluation to analyze the translation quality. We respectively randomly pick 100 sentences from Zh ⇒ En and De ⇒ En test sets, and recruit three human judges for evaluation. For each sentence, the judges read the source sentence and two candidate translations, which are from the random subset model and G-DIG subset model. The judges are required to rate each candidate on a scale of 1 to 5 and pick the better one.
From Table 2, we can see our methods enable the model to translate better with a higher average score in both Zh ⇒ En and De ⇒ En translations. Also, our G-DIG subset model is more frequently rated as better translation than the random subset model on both Zh ⇒ En and De ⇒ En translations,
indicating our methods better align the model than the random selection method.
The Fewer the Instructions, the Greater the Significance of Diversity. To see the role that diversity plays during finetuning, we compare the results of G-DIG and G-DIG w/o Diversity on Zh ⇒ En and De ⇒ En translation in Figure 2. Remarkably, our diversity enhancement makes significant advancements in enhancing the translation performance when there are only few training data provided (k ≤ 2000). For Zh ⇒ En translation with 1000 training examples, our G-DIG further improves G-DIG w/o Diversity by 1.7 in terms of COMET on WMT and by 2.17 in terms of BLEU on FLORES. In addition, for De ⇒ En translation with 1000 training examples, our G-DIG improves the BLEU by 2.11 and 1.24 on WMT and FLORES respectively compared with G-DIG w/o Diversity. However, as the amount of instructions increases, this effect fades away, since large instruction sets are already coupled with high diversity. As shown in Figure 2, the G-DIG curves almost coincide with the G-DIG w/o Diversity curves in all metrics when the amount of instructions goes beyond 4000 and 8000 for WMT and FLORES, respectively.

heading: 5.2 Analysis
text: In this section, we conduct ablation studies on our high-quality data selection module and our diversity enhancement module. Specifically, to see the superiority of our high-quality data selection module, we compare G-DIG w/o Diversity with the reward model-based baseline in Figure 3. In order to demonstrate the advantage of our gradient-based diverse data selection module, we compare our G-DIG with its embedding-based counterparts in Figure 4.

heading: Our High-quality Data Selection Module Surpasses the Reward Model-based Method In
text: Figure 3, we show that our high-quality data selection module G-DIG w/o Diversity achieves superior results compared with the existing reward model-based method in Figure 3. Specifically, for Zh ⇒ En translation our G-DIG achieves overall better results compared with the reward modelbased method, falling behind in only few cases. For De ⇒ En translation, it is hard to distinguish between ours and reward models-based baselines in COMET and BLEURT. However, our G-DIG w/o Diversity significantly outperforms the reward model in terms of BLEU on both WMT22 and FLORES.
Our Diversity Enhancement Module Improves the Training Data Diversity. In this part, we substantiate the superiority of our gradient-based diversity enhancement module. We compare our G-DIG
with its embedding-based counterpart. Specifically, embedding-based methods measure the similarity of training examples based on embeddings. Their objective is to maximize the semantic richness of the training data. We follow the model used in (Du et al., 2023) for diversifying the training data. We use BERT (Devlin et al., 2018) to extract embeddings. Then we run Kmeans to cluster these embeddings and use our sampling procedure to enhance the training data diversity. As illustrated
in Figure 4, our proposed method outperforms its embedding-based counterpart in almost all cases, which validates the advantage of our gradient-based method. For instance, with 1000 instructions provided, our methods surpass the embedding-based method by 2.11 of BLEURT and 1.91 of BLEU on WMT and FLORES respectively. Moreover, in almost all scenarios, the curves of embeddingbased method coincide with G-DIG w/o Diversity curves with trivial improvements, indicating that the existing methods that enhance semantic coverage contribute little to the finetuning data diversity.

heading: Our Methods Select Highly Parallel Texts. In
text: this part, we showcase our selected high-quality data and randomly selected data in the form of parallel texts in Table 3. Remarkably, not only is our selected data accurate and coherent in the target text space, but it is also natural and of the correct format and grammar in terms of the source text.

heading: 5.3 Discussion on Hyperparameters
text: There are two primary hyperparameters in our methods: the number of seed data |Dseed| in highquality data selection and the number of clusters for K-means in diverse data selection. Intuitively, the more seed data we use, the more strict the criteria in (4) would be. Therefore, increasing the number of seed data improves the quality of selected data. And using less seed data can be regarded as a relaxation to the criteria in (4). We experiment with seed dataset sizes of 128, 256, and 512 and note that further enlarging the seed dataset contributes little to improving the quality of selected seed data. Considering the high cost of obtaining human-annotated high-quality data, we ultimately use 256 seed data after balancing annotation costs and model performance benefits.
The reasonable choice of the number of clusters in diverse data selection is crucial for our methods. Since our candidate pool is quite large, we cluster the data into 512 clusters to ensure the variety of training data.

heading: 6 Conclusion
text: In this paper, we propose G-DIG, a novel gradientbased method for selecting high-quality and diverse LLM finetuning data for machine translation. Specifically, using Influence Function and a seed dataset we select high-quality training data that have beneficial influence on the model. Furthermore, we enhance the training data diversity by
clustering on their gradients and resampling. Extensive experiments prove that our methods improve the LLM in terms of translation ability. Also, human evaluation results demonstrate that our methods better align the LLM compared with the baseline model. We hope this work facilitates the research on LLM finetuning data selection.

heading: Limitations

heading: Computational Cost for Influence Function. In
text: this paper, we utilize Influence Function to measure the influence of a training sample on a test sample’s prediction. However, the computational cost for calculating this influence can be large for LLMs. The asymptotic computational complexity for calculating Hessian is O(P 2) where P is the number of model parameters. And the computational complexity for calculating pair-wise influence between the seed data and candidate pool is O(MNP ) where M and N denote the number of seed data and candidates, respectively. How to reduce the computational cost of influence function still remains a challenge. We leave this for future work.

heading: Ethical Considerations
text: All the data sources are publicly available and do not involve privacy issues. For all human evaluations mentioned in the paper, we hire full-time professional translators and pay them with market wage. All of our translators are graduates.

heading: B Annotation Guidelines
text: We hire full-time translators who are fluent in both Chinese and English and translators who are fluent in both German and English. They are recruited to conduct human evaluations. The translators are shown the source sentence and two candidate translations, which are from the random subset model and G-DIG subset model. Then the translators are required to rate on the translation quality from 1 to 5 with 1 the worst and 5 the best and pick the better one. All our translators are Chinese.

heading: C Statistical Analysis of Experimental Results
text: We conduct statistical analysis on our experimental results as shown in Table 4. Specifically, we use t-test to analyze the results of our G-DIG across various dataset sizes from Figure 2 compared with random baseline. Our experimental results have demonstrated statistical significance, as evidenced by the p-values less than 0.05 in almost all cases.
6https://github.com/tfjgeorge/nngeometry

heading: Future Work
text: We leave this for future work