Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities

Authors: Qirun Dai, Dylan Zhang, Jiaqi W. Ma, Hao Peng
Primary area: foundation or frontier models, including LLMs
Keywords: Instruction Tuning, Data Selection, Influence Estimation

Abstract:
Selecting appropriate training data is crucial for successful instruction fine-tuning, which aims to (1) elicit strong capabilities from pretrained large language models (LLMs), and (2) achieve balanced performance across a diverse range of tasks. Algorithms based on influence estimation have shown promise in achieving (1) through estimating the contribution of each training example to model's prediction on a downstream task, but they often struggle with (2). Through systematic experiments, we attribute their underperformance to an inherent bias---certain tasks intrinsically have greater influence than others. Directly comparing influence scores across different tasks would thus bias the selected data towards these tasks, hurting the LM's performance not only on other capabilities, but also, surprisingly, on the tasks for which the selected data has high influence.

To address this issue, we propose BIDS, a Balanced and Influential Data Selection algorithm.  BIDS first normalizes influence scores of the training data with respect to each downstream task at an instance level. It then applies an iterative process to further balance the selection of influential training data. At each step, BIDS selects the training example that bears the highest influence on the most underrepresented capability by the currently selected data. We perform comprehensive experiments using both Llama-3 and Mistral-v0.3 on seven evaluation benchmarks spanning five diverse capabilities. Results demonstrate that BIDS consistently outperforms both state-of-the-art influence-based data selection algorithms and other non-influence-based selection frameworks under various budgets. Surprisingly, training on a 15% subset selected by BIDS can even outperform full-dataset training with a much more balanced performance across different tasks. Our analysis further highlights the importance of both instance-level normalization and iterative optimization of selected data for balanced learning of diverse capabilities.

Decision: reject
Reviewer scores: 8, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730151605023

thread_type:
official_review

<thread_object>
creator: reviewer_xlpb

<content>
summary:
The paper proposes BIDS, a data-selection algorithm designed to address the inherent bias in existing influence-based data selection methods for instruction tuning of large language models. Build upon the previous work LESS, they first apply instance-level normalization on the influence scores, then utilize an iterative selection algorithm to identify influential samples while maintaining a balance between tasks.

strengths:
- Overall, the writing is clear and easy to follow, and the problem setup is well-defined.
- The paper provides a detailed analysis of the limitations of previous work.
- Experimental results demonstrates the propose method's effectiveness across the selected baselines (however, see weaknesses).

weaknesses:
- This proposed is somewhat incremental, with limited technical contributions. It heavily relies on the previous work LESS, with the primary contributions being instance-level normalization and an “iterative” selection algorithm. However, normalization is already well-explored in ML, and the iterative selection is very similar to “maximal marginal relevance"[1].

  [1] Carbonell, Jaime, and Jade Goldstein. "The use of MMR, diversity-based reranking for reordering documents and producing summaries." *Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval*. 1998.

- Experiments are insufficient. There is no direct comparison between the proposed method and its previous work, LESS, or other data selection methods. The selected baselines are naive. Additionally, experiments are limited to LLaMA-3-8B, leaving the generalization capacity of the proposed method unproven.

- The experimental results are not very convincing. The performance of the proposed method is worse than the random baseline on some benchmarks, and there is no clear clarification or explanation in the paper. Besides, there are some presentation issues in table 2. It is misleading to "bold" the results if it outperforms two baselines (even if it is specified in the caption). In addition, results for the 5% budget and 15% budget on MMLU should not be bolded.

questions:
- Why do you call it "instance-level" normalization? Can you provide some more comparisons between different normalization methods?
- Can you provide additional results on different LLMs (e.g. Mistral-7B, LLama-2-7B/13B as used in LESS)? It might better demonstrates the generalization of the proposed method.
- Can you provide some examples (LLM response) to show that the proposed method is better for balancing across tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730486567242

thread_type:
official_review

<thread_object>
creator: reviewer_uuab

<content>
summary:
This paper proposes a new data selection method for instruction tuning of LLMs to achieve better performance while using less data. The method proposed is BIDS, which normalizes influence scores per task and iteratively selects the most influential data for underrepresented capabilities, ensuring a balanced selection across different tasks. Experiments on LLama3 show that BIDS achieves better performance on UltraInteract benchmark over several baselines.

strengths:
1. The writing is clear and easy to follow.
2. After using the proposed data selection algorithm, the performance is better than using all data on some benchmark.

weaknesses:
1. The algorithm design lacks enough novelty. It simply modifies the original LESS algorithm by adding a instance-level normalization to the attribution matrix.
2. The paper does not well explain why using a balanced data selection can leads to better performance. And why using less data can leads to better performance than using all data for training. (Why excluding less "Influence" data can leads to better performance.)
3. The algorithm needs to maintain a |d| &times; |V| matrix, which is not efficiency when the dataset is large. The computation cost of the proposed algorithm needs to be discussed in the paper.
4. Authors should compare their results with other SOTA data selection algorithms to prove their effectiveness, instead of just comparing different variant of Influence-based instruction tuning data selection.

questions:
What is the ratio for dividing the data into training, validation, and test sets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730582949878

thread_type:
official_review

<thread_object>
creator: reviewer_kszn

<content>
summary:
This paper studies selecting suitable training data for supervised instruction fine-tuning (SFT) of pretrained LLMs. Specifically, this paper focuses on the strategy of iterative balanced selection based on the influence score between training and validation samples, named BIDS. BIDS first normalizes influence scores per task and iteratively selects training data that maximally enhances underrepresented capabilities. They provide several analyses and ablations to validate the effectiveness of core components and motivations of their approach.

strengths:
Pointing out the necessity of balanced sampling for recent instruction tuning datasets, which usually consist of multiple distinctive tasks.

weaknesses:
The reviewer agrees that the balanced selection should be meaningful for the multitask instruction tuning dataset.

However, the observations and discussions provided in the paper seem to be limited and less rigorous than the reviewer expected.

1. limited contribution: I understand this work as a LESS method, but I draw each sample iteratively for balance, which sounds more like an ad-hoc engineering technique for LESS. Evaluation with a single LLM backbone and single 'meaningful' baseline doesn't give me meaningful insights throughout the paper.

2. limited analysis: The most straightforward setting for balanced sampling should be to select the same number of influential samples of each training dataset (i.e., task) from the corresponding validation set (i.e., task-specific LESS selection). However, there is no comparison or discussion. Also, the impact of influence score from validation sets from a task to training samples from other tasks is not discussed, which may provide further insights on balanced sampling for underrepresented tasks.

3. marginal performance improvement: performance improvement is limited. For example, in Table 2, under a 5% budget, there is a 0.4% improvement from simple *sum*. Under 15% sample budget, it also shows 0.4% improvement against *random*. Considering the randomness of selecting validation samples and stochastic training, it looks almost like random fluctuation within the variance.  In addition, in Table 1, I wonder why the gap between the random selection and LESS has a marginal performance difference, which is quite hard to understand considering the report of the original LESS paper showing significant performance gain from random selection. This paper uses LLAMA3 (LESS uses llama2-7B, llama2-13B, and mistral-7B), but not sure this is the only reason for this. 

4. limited experiments: this paper uses only one LLM backbone (llama3-7B) and uses only one meaningful baseline - LESS. More extensive comparison under the various LLM backbones following LESS paper and including various types of data pruning/selection papers, including a balanced or density-based selection approach, should be needed.

questions:
Please see the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696563061

thread_type:
official_review

<thread_object>
creator: reviewer_6hj9

<content>
summary:
The paper introduces BIDS (Balanced Influence-based Data Selection), an algorithm for improving the balance and performance of LLMs across diverse tasks during supervised instruction tuning. Traditional influence-based methods often skew data selection toward tasks with inherently higher influence scores, leading to unbalanced model capabilities. BIDS addresses this by normalizing influence values at the instance level and applying an iterative selection process that prioritizes data for underrepresented tasks. Experimental results show that BIDS achieves more balanced performance across tasks like coding, logic, math, and instruction-following.

strengths:
1. strong results with only 15% of the dataset, keeping up with full-dataset training, demonstrating its potential for resource-efficient tuning of LLMs.
2. Comprehensive across diverse tasks (coding, logic, math, and instruction-following), showing consistent improvements over baseline methods, which strengthens the validity of its balanced selection approach.

weaknesses:
The paper primarily focuses on specific tasks and one model (Llama-3-8B). Testing BIDS on a broader range of models (different families and sizes) and larger task sets would help assess its scalability and generalization.

questions:
How robust is BIDS to noise or inaccuracies in influence score estimation? Could variations in influence scoring across tasks impact the balance achieved, and have any measures been taken to address potential score biases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732306626753

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/3)

comment:
We thank all the reviewers for their constructive feedback! A revision of the original paper has been uploaded, where we fix typos and update additional experiments and discussions in response to the reviewers’ questions and concerns. Most of the revisions are highlighted in blue for better presentation, except for the revision of mild typos or formatting issues in tables/figures/algorithms that don’t change the meaning of the original content. Here we specifically address some of the shared concerns and questions.


## 1. Validating the generalizability of BIDS with different base models (Appendix A.5)

All the four reviewers highlighted the need for validating the effectiveness of BIDS on other base models than Llama-3-8B. Our revision compares BIDS with other baseline data selection algorithms using Mistral-7B-v0.3 as the backbone for both selection and training. The additional results are updated in Appendix A.5, and also shown below.

| **Budget** |    **Method**   | **Coding** |          | **Logic** | **STEM** | **Math** |          | **Ins-Following** | **Macro Avg** |
|:----------:|:---------------:|:----------:|:--------:|:---------:|:--------:|:--------:|:--------:|:-----------------:|:-------------:|
|            |                 |  HumanEval |   MBPP   |    BBH    |   MMLU   | GSM-Plus |   MATH   |       IFEval      |               |
|     5%     |      Random     |    36.8    |   44.3   |  **59.5** |   61.7   |   37.0   | **19.9** |        22.2       |      40.2     |
|            |   Unnormalized  |    33.3    | **45.0** |    59.3   |   61.6   |   38.0   |   18.7   |        22.0       |      39.7     |
|            |    Normalized   |    36.8    |   44.1   |    59.1   |   61.5   | **38.2** |   19.6   |      **27.5**     |    **41.0**   |
|            |       BIDS      |  **37.7**  |   44.4   |  **59.5** | **61.8** |   38.0   |   19.8   |        26.1       |    **41.0**   |
|     10%    |      Random     |    37.7    |   44.8   |    59.8   | **61.8** |   40.0   | **21.2** |        22.0       |      41.0     |
|            |   Unnormalized  |    36.0    |   43.8   |    59.7   |   61.5   | **41.6** |   20.8   |        24.6       |      41.1     |
|            |    Normalized   |    37.7    |   45.0   |    59.7   |   61.6   |   40.2   |   20.2   |        26.7       |      41.6     |
|            |       BIDS      |  **40.4**  | **46.1** |  **60.5** |   61.7   |   40.5   |   21.0   |      **27.1**     |    **42.5**   |
|     15%    | BIDS (epochs=4) |    40.4    |   47.0   |  **58.9** | **61.1** |   44.1   |   23.5   |      **28.1**     |    **43.3**   |
|    100%    | Full (epochs=4) |    41.2    |   49.3   |    54.6   |   59.4   |   48.1   |   30.1   |        19.6       |      43.2     |

We would like to highlight three observations from these results:

1. For Mistral-v0.3 under different selection budgets, BIDS still outperforms all other three baselines with a better macro-avg and more balanced task-specific performance.
2. Specifically, the improvements of Unnormalized → Normalized → BIDS are consistent with our prior observation on Llama-3-8B (Section 6 of main text). We also carried out the same set of analysis of the influence distribution of selected data (Section 6.2 of main text) by probing into AID and THI metrics. These analysis figures are presented in Appendix A.5. Both AID and THI metrics suggest a balancing trend, though the degree of the original imbalance in influence scores for Mistral-v0.3 is not as high as Llama-3.
3. Notably, the top 15% subset selected by BIDS again outperforms full dataset training in macro average. With the increase of the scale of training data, BIDS not only continues to improve its performance on Coding and Math, but also maintains its remarkable instruction-following ability, which is severely degraded in the case of full dataset training.

These results confirm the generalizability of BIDS to different base models, especially given that Llama-3 and Mistral-v0.3 belong to different backbone families, and thus probably have divergent pre-training data compositions.
</content>
</thread_object>

</thread>


Metareview: