DELIFT: Data Efficient Language model Instruction Fine-Tuning

Authors: Ishika Agarwal, Krishnateja Killamsetty, Lucian Popa, Marina Danilevsky
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Data Efficient Instruction Fine-Tuning;  Data Subset Selection; Submodular Functions

Abstract:
Fine-tuning large language models (LLMs) is crucial for task specialization but often becomes resource-intensive due to redundant or uninformative data. Existing data selection methods typically rely either on computationally expensive gradient-based metrics or static embeddings that fail to adapt dynamically to the model’s evolving state, thus limiting their practical effectiveness. To address this,
we propose DELIFT (Data Efficient Language model Instruction Fine-Tuning), leveraging a novel, computationally efficient utility metric inspired by In-Context Learning (ICL). Our ICL-based metric measures the informational value of each data sample by quantifying its effectiveness as an in-context example in improving model predictions for other samples, reflecting its actual contribution relative to the model’s current state. Integrated with tailored submodular optimization methods, DELIFT systematically selects diverse, informative subsets optimized specifically for each fine-tuning stage: instruction tuning, task-specific adaptation, and continual fine-tuning. Experimental results across multiple datasets and model scales show DELIFT reduces fine-tuning data requirements by up to 70% without compromising performance, consistently outperforming existing methods by up to 26% in effectiveness and efficiency.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730283533363

thread_type:
official_review

<thread_object>
creator: reviewer_qaun

<content>
summary:
In this paper, the authors propose a data selection method named DELIFT. DELIFT first employs a pairwise utility metric to quantify how beneficial a data sample is for improving the model's responses to other samples. Basing on the pairwise utility metric, DELIFT leverages 3 specially designed submodular functions to conduct diverse subsets that are useful across the instruction tuning, task-specific fine-tuning and continual fine-tuning. Experiments across various tasks and model scales demonstrate that DELIFT can reduce the fine-tuning data size by up to 70% without compromising performance, offering significant computational savings and outperforming existing methods in both efficiency and efficacy.

strengths:
1. **Effective performance of DELIFT** In this paper, the authors design a utility-based kernel to quantify the informativeness of data. Based on this, the authors further design submodular functions for instruction data selection, targeting Instruction Tuning, Task-specific Fine-Tuning, and Continual Fine-Tuning. This method has been proven effective in all three stages and outperforms other model-dependent baselines.

weaknesses:
1. **Related Work not Discussed** In Entropy Law [1], the authors also addressed the instruction data selection problem from the perspective of informativeness (entropy) and quantifies the relationship between data compression ratio and model performance; However, [1] is not discussed in this paper.

[1] Entropy Law: The Story Behind Data Compression and LLM Performance

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730594711905

thread_type:
official_review

<thread_object>
creator: reviewer_hvnc

<content>
summary:
This paper addresses the challenge of selecting optimal data for fine-tuning large language models (LLMs). The authors propose a pairwise utility metric to assess the value of each sample based on the model's current performance and the relevance of other samples in the dataset. They introduce three variations of a facility location (FL) function to guide data selection at different fine-tuning stages, including (i) a vanilla FL function to prioritize diversity in general instruction fine-tuning, (ii) a mutual information-based FL function to select data that is diverse and relevant to a target task for task-specific fine-tuning, and (iii) a conditional gain-based FL function to select data that complements previously used data for continual fine-tuning. Experiments on two instructional models, Phi-3-mini and Qwen2-72B, demonstrate that the proposed methods outperform existing data selection techniques across all three fine-tuning stages.

strengths:
- Data selection for LLM fine-tuning is a timely research problem, particularly in light of the growing model sizes, data availability, and resource constraints.
- The paper discusses the varying criteria for data selection across fine-tuning stages. The proposed method can be customized for different fine-tuning objectives.

weaknesses:
- The utility metric in Equation 1 is calculated based on in-context learning (ICL) gains, which may not directly translate to improvements in fine-tuning performance. Is there supporting literature to justify the use of ICL-derived utility in this context?
- It remains unclear how the utility gain measured through ICL reflects sample diversity in the semantic space. Since utility values are computed only once at the beginning of the data selection process, they may not capture changes in the model's performance after fine-tuning on the current selected subset.
- The experimental design would benefit from comparisons with additional diversity-based baselines (e.g., clustering). Analysis of the selected examples across different methods would provide insights into the strengths and weaknesses of each approach.
- A more thorough ablation analysis on the FL functions is needed. Do the additional terms for each fine-tuning stage significantly contribute to the results? Would the vanilla FL perform adequately across all stages? Evaluating this would help clarify the need for distinct data selection strategies at each fine-tuning stage.

questions:
- In equation 3-5, what is $s_ij$? Is it the utility-based kernel $UF_ij$? Is it recomputed after each selection round? 
- For Equation 1, how is the ground truth distribution determined? The description of "a vector of ones for each token to signify perfect predictions" needs further clarification.
- The latex template of the submitted manuscript appears modified, as it lacks line numbers in the left margin.
- Why are instruction-finetuned models used for the experiments rather than pre-trained models, especially given that the first use case is instruction fine-tuning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730905052

thread_type:
official_review

<thread_object>
creator: reviewer_gadz

<content>
summary:
This paper presents DELIFT, Data Efficient Language model Instruction Fine-Tuning, a method to select and prune fine-tuning data for three stages - initial instruction tuning, task-specific fine-tuning, and continual fine-tuning. One of the contributions of the paper is a utility-based kernel to quantify the importance of a data point in the presence of another data point used as an in-context learning (ICL) sample. The authors further use various submodular functions (Facility Location - FL, Facility Location Mutual Information - FLMI, and Facility Location Conditional Gain - FLCG) for the three fine-tuning stages defined above. The main core of DELIFT comprises of a greedy algorithm to iteratively select data points that provide the maximum marginal gain with respect to the chosen submodular function. DELIFT shows clear gains over various baselines for all three stages of fine-tuning using the Phi-3 (3.8B params) and Qwen-2 (72B params).

strengths:
- The paper proposes a novel utility metric function to measure mutual information of data points and their relevance. The authors further highlight the benefit of this choice by comparing against sentence embedding based metrics. DELIFT performs better compared to DELIFT used with SE.
- The authors build targeted modular functions for each stage (FL for standard IFT, FLMI for task-specific FT, and FLCG for continual FT), catering to the optimization objectives for these 3 stages.
- DELIFT is able to retain and sometimes improve performance over the full fine-tuning by pruning out 70% of the full data.

weaknesses:
- Since the authors use post-trained models like Phi-3-mini-128k-instruct and Qwen2-72B-Instruct as the starting point, aren't the three stages described and the followup experiments basically a part of continual fine-tuning? I would be interested to see the effect of DELIFT when starting from base models and see if the method is able to prune out fine-tuning data for IFT starting from scratch.
- The authors present the results on ICL and QLoRA fine-tuning but not the full parameter fine-tuning. Are there diminishing gains with full model fine-tuning? I would like to see the results corresponding to those too.
- What's the computational cost for the data selection algorithm since it involves doing forward passes over the language model and not use cheap embedding models to compute the probability distributions. How does it compare it with the fine-tuning stage involving QLoRA. 
- Weakness 1 is further enhanced when results for use case 2 are presented, where fine-tuning actually leads to drop in performance on MMLU, where it doesn't make sense to do any task-specific fine-tuning at all. And I don't trust the results on MT-Bench since it's a very small dataset and the error bars would be huge. To mitigate this weakness, I would suggest the authors to try task-specific fine-tuning on say some reasoning dataset and then track the performance on benchmarks like MATH/GSM8K/ARC-Challenge etc.

questions:
Other than the questions listed in the weaknesses, here are few clarifying questions for the authors:

1. For computing the metric in equation 2, are the probability vectors ($d$ dimension) from all predicted tokens (say $x$ tokens) concatenated to form a vector of dimension $x * d$ and then the equation 2 is applied?
2. Minor typo in Figure 2 (should be LAJ scores instead of LJA scores).
3. Can some other baselines be also included like [1] and [2].


[1] DEFT-UCS: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection, Das et al., 2024.

[2] Text Quality-Based Pruning for Efficient Training of Language Models, Sharma et al., 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731498013120

thread_type:
public_comment

<thread_object>
creator: ~junyu_luo2

<content>
title:
Nice Work of Data-efficient LLM

comment:
I greatly enjoyed reading the authors' impressive work. The research on improving LLM performance in data-efficient settings is both timely and inspiring. Given our shared research interests, I would like to humbly share our recent exploration in this direction[1]. We find our work is relevant to this research and would be deeply grateful for any discussion of our work in the future revisions.

Congratulations on the authors' excellent contribution to the field.

Best regards,

Junyu

---
[1]  SemiEvol: Semi-supervised Fine-tuning for LLM Adaptation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731960909696

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Experiments with base models

comment:
We thank the reviewers for their time and effort in reviewing our work. We especially appreciate the fact that the reviewers found our work to be novel while (1) targeting specific use cases with a general framework, and (2) showcasing good results with DELIFT's subset selection methodology. We wish to address some common concerns that reviewers posed.

---
Below are the results of various subset selection baselines on Use Case 1: Instruction Tuning with Mix-Instruct as the dataset, and is equivalent to the setting in Table 1 of the paper. This is performed on the base model Llama-3.2-3B. As per reviewer gaDz's suggestion, we also add DEFT-UCS as a baseline. We see that **the takeaways are consistent with those mentioned in the paper**. Similar to DELIFT (SE), DEFT-UCS relies on sentence embeddings for sampling representative subsets. One key issue with DEFT-UCS is that it doesn't select representative samples useful for the model but rather rely on the semantics of the data samples. Similarly, our results show that **DELIFT (SE) outperforms DEFT-UCS** even though both rely on sentence embeddings for clustering highlighting the effectiveness of facility location as the subset selection objective. Finally, **DELIFT outperforms all the baselines**, highlighting the effectiveness of our utility kernel in capturing the important samples for the model.

Table 10: Llama-3.2-3B (base) on Use Case 1: Mix-Instruct (equivalent to Table 1).
| Method      |       | ICL   |      |       | QLoRA |      |
|-------------|-------|-------|------|-------|-------|------|
|             | ROUGE | BGE   | LAJ  | ROUGE | BGE   | LAJ  |
| Initial     | 25.88 | 61.97 | 1.9  | 28.51 | 73.14 | 2.48 |
| Random      | 28.64 | 77.14 | 2.59 | 38.79 | 79.89 | 2.53 |
| SelectIT    | 42.67 | 81.54 | 2.67 | 45.87 | 84.67 | 2.61 |
| DEFT-UCS    | 41.55 | 80.12 | 2.63 | 41.03 | 81.86 | 2.59 |
| LESS        | 44.99 | 82.48 | 2.69 | 50.54 | 84.14 | 2.78 |
| DELIFT (SE) | 51.19 | 83.54 | 2.72 | 55.32 | 85.92 | 2.79 |
| DELIFT      | 54.58 | 88.29 | 2.83 | 58.57 | 90.98 | 2.98 |
| Full Data   | 55.46 | 92.71 | 3.31 | 61.23 | 95.52 | 3.10 |

---
Below are the results of various subset selection baselines on Use Case 2: Task-Specific Fine-Tuning where examples from Mix-Instruct that are similar to those in GSM-8k are chosen in the subsets. This is equivalent to Table 4 in our paper, and is performed on the base model Mistral-7B-v0.1. As with Table 10, **the takeaways here are also consistent with those from the paper**. As per reviewer hvNC's suggestion, we conducted an ablation study comparing two variants of our method: **DELIFT with Facility Location Mutual Information (FLMI)** and **DELIFT with only Facility Location (FL)**. The results demonstrate that **DELIFT with FLMI achieves superior performance**, yielding a 2.61\% improvement over the FL-only variant, validating our choice of FLMI for task-specific fine-tuning. Notably, even **DELIFT with FL alone outperforms all other baselines**, underscoring the fundamental strength of our utility kernel and facility location objective for subset selection.

Table 11: Mistral-7B-v0.1 (base) on Use Case 2: Mix-Instruct and GSM-8k (equivalent to Table 4).
| Method           |       | ICL   |      |       | QLoRA |      |
|------------------|-------|-------|------|-------|-------|------|
|                  | ROUGE | BGE   | LAJ  | ROUGE | BGE   | LAJ  |
| Initial          | 17.06 | 55.13 | 1.35 | 27.95 | 65.92 | 1.36 |
| Random           | 19.18 | 56.03 | 1.38 | 33.85 | 81.74 | 1.85 |
| SelectIT         | 30.27 | 77.23 | 1.45 | 42.29 | 88.17 | 2.49 |
| DEFT-UCS         | 30.06 | 76.99 | 1.59 | 41.45 | 87.84 | 2.08 |
| LESS             | 31.69 | 77.87 | 2.26 | 43.86 | 88.22 | 2.60 |
| DELIFT (SE)      | 31.84 | 78.62 | 2.44 | 43.04 | 90.53 | 2.54 |
| DELIFT (FL Only) | 32.30 | 78.54 | 2.54 | 44.62 | 91.04 | 2.63 |
| DELIFT           | 33.25 | 79.10 | 2.56 | 46.12 | 92.97 | 2.71 |
| Full Data        | 35.33 | 81.59 | 2.79 | 49.10 | 94.57 | 2.85 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary: 
The paper introduces DELIFT (Data Efficient Language model Instruction Fine-Tuning), an algorithm designed to optimize data selection efficiently across three crucial stages of fine-tuning large language models (LLMs). DELIFT utilizes a pairwise utility metric, which uses in-context learning to quantify the significance of each data sample in enhancing the model's responses to other samples, enabling informed data selection. By applying different facility-location-based submodular functions tailored to each fine-tuning stage, DELIFT strategically selects diverse and optimal subsets, reducing fine-tuning data requirements by up to 70% without compromising performance. Experiments across various tasks and model scales validate DELIFT's computational efficiency and superior efficacy compared to existing methods, demonstrating clear gains in both efficiency and effectiveness throughout the fine-tuning process.

## Strengths:
1. DELIFT studies the important data selection problem for all three fine-tuning stages of LLMs using submodular maximization of facility location functions, which is a novel application of submodular optimization. 
1. The in-context learning-based utility metric is novel: it measures the contribution of sample-j to predicting sample-i in order to quantify the mutual information and relevance between the two samples. 
1. It introduces three facility location functions to measure the utility of selected subsets of data for the three finetuning stages, respectively, by providing intuitive explanations. 
1. DELIFT demonstrates the ability to reduce fine-tuning data requirements by up to 70% while maintaining or even improving performance.

## Weaknesses:
1. The effectiveness of the proposed method needs to be demonstrated in more general cases, e.g., finetuning base models, full-model finetuning, and comparisons to more recent data selection baselines. The rebuttal provides some relevant and new experiments. 
1. It lacks a thorough comparison of the computational cost for data selection. Most existing data selection approaches for LLM finetuning only require $O(n)$ forward inferences for $n$ samples and evaluating $n$ scores followed by top-$k$ selection. But the proposed method requires $O(n^2)$ forward inferences to estimate the pairwise utility and $O(kn)$ evaluations for selecting $k$ samples. Hence, the proposed method can be much more expensive. This is the diversity tax that may undermine the practical applicability of the method. 
1. Although the goal is ambitious and targets three finetuning stages, the comparison for each stage does not cover some widely-used datasets, e.g., Alpaca/UltraChat/MAGPIE for instruction tuning, FLAN/SuperGLUE for task-specific finetuning (MMLU and MT-Bench are evaluation benchmarks with a small data size per task). The presented GSM-8k is better but a thorough report is needed. 
1. The choice of submodular functions and hyperparameters is usually highly important to their practical performance. The paper only focuses on facility location-based functions but lacks discussion and comparison to other widely studied ones in submodular optimization literature. 
1. The paper does not discuss the relationship and difference between the proposed ICL-based utility in Eq. 1 and the influence function (Koh & Liang, ICML'17), both aiming to measure the gain of one sample bringing to another in training. 

## Decision:
Two reviewers raised their ratings from 5 to 6 after the rebuttal and confirmed some important concerns in their original comments have been successfully addressed by the rebuttal. The authors are encouraged to provide more thorough and complete evaluations on more widely used benchmarks and keep improving the paper based on the discussion. Based on the positive feedback of all reviewers, the meta-reviewer recommends acceptance of the paper.