MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space
Authors: Yicheng Chen, Yining Li, Kai Hu, Ma Zerun, HaochenYe, Kai Chen
Accepted at: acl25

MIG: Automatic Data Selection for Instruction Tuning by Maximizing Information Gain in Semantic Space

Yicheng Chen1,2, Yining Li1†, Kai Hu1,3, Zerun Ma1, Haochen Ye1, Kai Chen1†
1Shanghai AI Laboratory 2Fudan University 3Carnegie Mellon University

Project page: https://yichengchen24.github.io/projects/mig

Abstract

Data quality and diversity are key to the construction of effective instruction-tuning datasets. With the increasing availability of open-source instruction-tuning datasets, it is advantageous to automatically select high-quality and diverse subsets from a vast amount of data. Existing methods typically prioritize instance quality and use heuristic rules to maintain diversity. However, this absence of a comprehensive view of the entire collection often leads to suboptimal results. Moreover, heuristic rules generally focus on distance or clustering within the embedding space, which fails to accurately capture the intent of complex instructions in the semantic space. To bridge this gap, we propose a unified method for quantifying the information content of datasets. This method models the semantic space by constructing a label graph and quantifies diversity based on the distribution of information within the graph. Based on such a measurement, we further introduce an efficient sampling method that selects data samples iteratively to Maximize the Information Gain (MIG) in semantic space. Experiments on various datasets and base models demonstrate that MIG consistently outperforms state-of-the-art methods. Notably, the model fine-tuned with 5% Tulu3 data sampled by MIG achieves comparable performance to the official SFT model trained on the full dataset, with improvements of +5.73% on AlpacaEval and +6.89% on Wildbench.

1 Introduction

Large Language Models (LLMs) have shown remarkable capabilities in following human instructions in a wide range of tasks (Wang et al., 2023a). Typically, LLMs first acquire general knowledge through large-scale pretraining and are subsequently refined through instruction tuning to better align with diverse human intentions (Brown et al., 2020; Taori et al., 2023; Touvron et al., 2023). Instruction tuning utilizes instruction-response pairs to guide base models toward more accurate and contextually appropriate responses. Recent studies (Zhou et al., 2023a; Chen et al., 2024) emphasize the critical role of data engineering in instruction tuning, highlighting data quality rather than quantity as the key to effective instruction tuning. Notably, LIMA (Zhou et al., 2023a) demonstrates that just 1000 high-quality, human-curated instructions can achieve performance comparable to substantially larger datasets. However, manual curation of such datasets is inherently time-consuming and labor-intensive (Chiang et al., 2023).

More recently, a line of work (Chen et al., 2024; Lu et al., 2024; Liu et al., 2024b; Bukharin et al., 2024) proposes automatic selections of optimal subsets from extensive data pools by defining desirable data characteristics. These approaches (Bukharin et al., 2024; Yu et al., 2024b) posit that quality and

![Comparison with different data selection methods on Tulu3, evaluated on black and red benchmarks.](page_1012_370_420_420.png)

Figure 1: Comparison with different data selection methods (Lu et al., 2024; Liu et al., 2024b) on the Tulu3 (Lambert et al., 2024) pool using Llama3.1-SB (Touvron et al., 2023), evaluated on (black) knowledge-based benchmarks and (red) human-preference benchmarks. See details in Sec. 4.2.
diversity are crucial for an effective instruction-tuning dataset. Data quality is defined from multiple perspectives, such as instruction complexity (Lu et al., 2024; Zhao et al., 2024), model perplexity and uncertainty (Li et al., 2024b), or scores assigned by advanced external models (Chen et al., 2024; Liu et al., 2024b). However, diversity remains less explicitly quantified, often addressed via heuristic methods such as maximizing label set coverage (Lu et al., 2024), reducing redundancy through diversity filters (Liu et al., 2024b), or enforcing fixed sample distributions per cluster (Ge et al., 2024; Yu et al., 2024b). This narrow focus on diversity only during later selection stages, without a comprehensive view of the entire dataset, diminishes the global diversity and representativeness of the sampled data. Alternative methods (Bukharin et al., 2024) employing embedding-based facility location functions (Cornuéjols et al., 1983) quantify diversity but require computationally intensive iterative pairwise distance calculations, making them impractical for large datasets. Additionally, distance-based clustering in the embedding space may fail to capture the semantic intent of complex instructions accurately. To solve these issues, several essential questions are raised: 1) How can we effectively quantify diversity in semantic space while balancing quality and diversity in dataset evaluation? 2) How can we efficiently select data based on such evaluations?

To this end, we propose an information-based measure for instruction-tuning datasets and introduce an efficient data selection algorithm that aims to Maximize the Information Gain (MIG). We model the semantic space as a label graph, with nodes representing labels and edges capturing semantic relationships. Information in the dataset is distributed across this graph, with the total information being the sum of each label’s information. Each data point contributes to its associated labels in proportion to its quality. Thus, the information of each data point measures local data quality, while the total of all label information measures the global diversity of the dataset. To balance quality and diversity, we apply a monotonically increasing but marginally diminishing function to compute label information, thereby promoting diversity and preventing excessive data concentration on particular labels. To better model information distribution in semantic space, we propagate information along label graph edges to address semantic correlations and annotation biases. Leveraging the submodularity of our proposed information-based dataset measurement, we implement an efficient greedy algorithm that iteratively selects data points that maximize the information gain according to the current state of the label graph.

Through extensive experiments across data pools (Liu et al., 2024a; Teknium, 2023; Lambert et al., 2024) of varying quality and sizes, and LLMs of different families (Touvron et al., 2023; Jiang et al., 2023; Yang et al., 2024), MIG consistently achieves superior performance on both human-preference and knowledge-based evaluations. As shown in Fig 1, on the Tulu3 (Lambert et al., 2024) pool with Llama3.1-8B as the base model, MIG achieves average improvements of \(+1.49\%\) on six knowledge-based benchmarks (Clark et al., 2018; Suzgun et al., 2022; Hendrycks et al., 2021; Chen et al., 2021; Cobbe et al., 2021; Zhou et al., 2023b) and \(+1.96\%\) on three human-preference benchmarks (Zheng et al., 2023; Dong et al., 2024; Lin et al., 2024) compared to previous state-of-the-art data selection methods (Liu et al., 2024b; Bukharin et al., 2024). When combining both evaluations, MIG achieves average improvements of \(+2.20\%\) compared to the second-best method (Bukharin et al., 2024). Notably, the model fine-tuned with 5% Tulu3 data sampled by MIG outperforms the official SFT model trained on the full dataset by \(+1.73\%\) (average on nine benchmarks), with a substantial boost of \(+4.59\%\) in human-preference evaluations. MIG also outperforms existing methods on the Openhermes2.5 (Teknium, 2023) and \(X_{sota}\) (Lu et al., 2024; Liu et al., 2024b), further demonstrating its generalizability across different settings. Additionally, MIG significantly enhances sampling efficiency, reducing sampling time by over 100-fold on the Tulu3 data pool compared to embedding-based methods.

In summary, our contributions are as follows:

• We propose an information-based measurement for instruction-tuning datasets in semantic space. It quantifies quality and diversity within the information distributed across the semantic label graph.
• We introduce MIG, an efficient data selection algorithm that maximizes the information gain on the label graph iteratively.
• Extensive experiments on various data pools, base models, and benchmarks demonstrate the effectiveness and generalizability of MIG. The correlation between parameters in MIG and the attributes of sampled data is well studied.
2 Related Work

Data Selection for Instruction Tuning. Recent studies (Zhou et al., 2023a; Chen et al., 2024) indicate that increasing data quality and diversity rather than quantity effectively boosts instruction-following performance. Consequently, data selection methods aim to identify optimal subsets that meet such characteristics and generally fall into three categories: (1) Quality-based approaches prioritize high-quality data points, where quality is defined through various perspectives, such as instruction complexity and response quality. INSTRUCTMINING (Cao et al., 2024b) identifies natural language metrics indicative of high-quality instruction data. Instruction-Following Difficulty (IFD) (Li et al., 2024b) highlights inconsistencies between a model’s anticipated responses and its self-generated outputs. Nuggets (Li et al., 2023b) measures quality based on the disparity between one-shot and zero-shot performance. LESS (Xia et al., 2024a) uses gradient features to select samples based on their similarity to a few representative examples. SelectIT (Liu et al., 2024a) selects high-quality data based on intrinsic uncertainty from token, sentence, and model levels. Additionally, some methods employ external LLMs to assess data quality, such as ALPAGASUS (Chen et al., 2024), which uses a well-designed prompt applied to ChatGPT to assess the quality of each data tuple. (2) Diversity-based approaches aim to select data subsets with broad coverage of the data pool. DiverseEvol (Wu et al., 2023) iteratively selects samples distant from previously selected data in the embedding space. ZIP (Yin et al., 2024) prioritizes subsets with low compression ratios, implicitly favoring diversity. (3) Comprehensive approaches strive to balance quality and diversity. #InsTag (Lu et al., 2024) employs ChatGPT to generate detailed open-ended tags for instructions and prioritize complex data with more tags while maximizing topic coverage. DEITA (Liu et al., 2024b) prioritizes high-quality data points while avoiding duplicates in the embedding space. CaR (Ge et al., 2024) and kMQ (Yu et al., 2024b) cluster data and sample high-quality points from each cluster. However, these methods typically rely on heuristic rules rather than a unified quantitative metric to balance quality and diversity.

Submodular function for Diversity Measurement. Traditional submodular functions, such as facility location, graph cut, and log determinant, effectively quantify dataset diversity by identifying representative, non-redundant subsets. Leveraging this property, QDIT (Bukharin et al., 2024) measures diversity using the facility location function (Cornuéjols et al., 1983), combining it linearly with quality scores. Similarly, DPP (Wang et al., 2024) employs the log determinant distance to quantify subset diversity. Although this NP-hard problem can be approximated with a greedy algorithm following submodularity (Nemhauser et al., 1978; Minoux, 2005), embedding-based methods are inefficient at scale due to the high storage and computational costs of calculating high-dimensional pairwise distances. To mitigate this issue, our custom dataset measurement is submodular, which justifies the use of a greedy strategy, while MIG samples data in a high-level semantic space, substantially reducing computational overhead.

3 Method

As shown in Fig. 2(a), we begin by annotating the raw data pool with a tagger and scorer. Next, MIG constructs a label graph to measure dataset information (Sec. 3.2) and selects a subset for subsequent SFT by maximizing the information gain (Sec. 3.3).

3.1 Preliminary

Task. Given a data pool \( D_P \), a budget \( N \), and an information measure \( E(D) \) over any dataset \( D \), the goal is to select a subset \( D_S \subset D_P \) of size \( N \) that maximizes \( E(D) \). Formally,

\[
D_S = \underset{D \subset D_P, |D|=N}{\operatorname{argmax}}\ E(D)
\]

Data. Each data point is formed as:

\[
d_i = \{(q_i^j, r_i^j)\}_{j=1}^M, L_i, s_i\}
\]

where \( (q_i^j, r_i^j)_{j=1}^M \) represents \( M \) rounds of query-response pairs used for training, \( L_i \) is the set of labels (e.g., task category, knowledge domain, and other meta information) associated with \( d_i \), and \( s_i \) is the quality score.

3.2 Information Measurement

Label Graph. Previous studies (Lu et al., 2024; Ge et al., 2024; Yu et al., 2024b) assume that labels (including embedding-based clusters) are independent, ignoring the semantic relationships among them. However, such label associations are crucial for accurately capturing the information distribution in semantic space. Intuitively, we can
Figure 2: Illustration of (a) Data Selection Pipeline and (b) MIG Sampler. Given the raw data pool, our pipeline first applies a tagger and scorer to annotate data. Next, MIG constructs the label graph based on the label set and iteratively selects the data point that maximizes the information gain within the graph. The selected data are used for supervised fine-tuning (SFT) of LLMs.

model labels as nodes, their associations as edges, and the intensity of associations as edge weights, thus modeling semantic space as an undirected weighted graph \( G_L = (L, E_L) \), where \( L \) represents the label set with a size of \( K \) and \( E_L \) represents edges. Specifically, we use label similarities as edge weights and remove edges whose weights are below a threshold \( T \) to ensure computational efficiency. Therefore, \( E_L \) can be formed as a weighted adjacency matrix \( W_L \in \mathbb{R}^{K \times K} \) with elements:
\[
w_{pq} = \sigma[w(l_p, l_q) \geq T] \cdot w(l_p, l_q)
\]
where \( w(l_p, l_q) \) represents textual similarity between label \( l_p \) and \( l_q \), and \( \sigma(\cdot) \) yields 1 when the input is evaluated as True.

Data Point Information. Under the label set \( L \), a data point \( d_i \) can be formed as a binary label vector with its associated labels \( L_i \):
\[
v_i = \{ v_k^i = \sigma(l_k \in L_i) \}_{k=1}^K
\]
The information of \( d_i \) is distributed over \( L_i \) and is proportional to its quality score \( s_i \). Thus, the raw information of \( d_i \) can be formed as:
\[
e_i = s_i \cdot v_i
\]
Semantic overlaps between labels and annotation-induced bias can lead to inaccurate information distribution. To address this, we introduce information propagation along the edges of the label graph, enabling a more accurate modeling of information distribution across the semantic space. Formally, the propagation from \( l_p \) to \( l_q \) is:
\[
a_{pq} = \frac{\alpha w_{pq}}{w_p + \alpha \sum_{k,k \neq p} w_{pk}}
\]
where \( w_p \) equals 1 and \( \alpha \) is a hyperparameter controlling the intensity of information propagation. Let \( A \) be the propagation matrix, then the propagated information vector of \( d_i \) is:
\[
\hat{e}_i = Ae_i
\]

Dataset Information. To balance quality and diversity within the label graph, we apply a monotonically increasing yet upper-convex function \( \phi \) to compute the label information. The marginally diminishing information gain is negatively correlated with the existing label information. Thus, information gains on labels with less information are prioritized. Formally, the dataset information is:
\[
E(D) = \Phi(\sum_{i \in D} Ae_i) = \Phi(A \sum_{i \in D} s_i v_i)
\]
where \( \Phi \) is a nonlinear transformation that applies \( \phi \) element-wise to the input and then aggregates the results by summation.

3.3 MIG Sampling

Directly selecting \( D_S \) from \( D_P \) is computationally infeasible as the combination \( C_{|D_P|}^N \) grows quickly.
Algorithm 1: MIG Sampling

Data: Initial Data Pool \( D_P \), Label Sets \( L \), Sample Budget \( N \)
Result: The Sampled Dataset \( D_S \)

1 Initialize Empty \( D_S \);
2 Initialize Propagation Matrix \( A \);
3 while \( |D_S| < N \) do
4   \( G \leftarrow A\Phi'(A \sum_{k \in D_S} E_k) \);
5   \( d_i \leftarrow \operatorname{argmax}_{d \in D_P} GE_d \);
6   \( D_S \leftarrow D_S \cup \{d_i\} \);
7   \( D_P \leftarrow D_P \setminus \{d_i\} \);
8 return \( D_S \)

Thus, as shown in Fig 2(b), we follow the submodularity of \( E(D) \) (detailed in Appx. C) and propose a greedy strategy, iteratively selecting the data point that yields the maximum information gain:

\[
d_k = \operatorname{argmax}_{d \in D_P^k} \{E(D_S^k \cup \{d\}) - E(D_S^k)\}
\]

where \( D_S^k \) and \( D_P^k \) denote the selected subset and remaining candidate pool at iteration \( k \). We approximate the information gain in Eq. 9 via a gradient-based approach:

\[
G_k = \frac{\partial E(D_S^k)}{\partial E} = A\Phi'(A \sum_{i \in D_S^k} e_i)
\]

where \( \Phi' \) represents the derivative of \( \Phi \). Thus, the selection process can be formed as:

\[
d_k = \operatorname{argmax}_{d \in D_P^k} G_k e_d
\]

The sampling is detailed in Alg. 1. Refer to Appx. A.2 for more implementation details of MIG.

4 Experiments

4.1 Setups

Datasets. To investigate data selection across various scenarios and demonstrate the robustness of MIG, we use three distinct data pools:
• Tulu3 (Lambert et al., 2024): A large-scale, real-world SFT dataset presented by Ai2, containing million-level records across a wide variety of subjects, including mathematics, programming, and user dialogues.
• Openhermes2.5 (Teknium, 2023): A dataset with over 1 million data points, sourced from 16 distinct origins, including MetaMath (Yu et al., 2024a), CamelAI (Li et al., 2023a), and others.
• Xsota (Lu et al., 2024; Liu et al., 2024b): A combined data pool consisting primarily of high-quality conversations from datasets such as WizardLM (Alpaca), WizardLM (ShareGPT), UltraChat (Ding et al., 2023), and ShareGPT (Chiang et al., 2023), totaling 300K data points.

Benchmarks. We use both human-preference and knowledge-based benchmarks to evaluate model performance comprehensively. The evaluation is conducted using OpenCompass (Contributors, 2023), with the average results reported as normalized scores on a percentage scale. Detailed evaluation settings are provided in Appx. A.4.
• Human-preference Benchmarks. We evaluate open-ended dialogue abilities using model-based evaluation metrics on three benchmarks: AlpacaEvalv2 (Dubois et al., 2024), MTBench (Zheng et al., 2023), and WildBench (Lin et al., 2024).
• Knowledge-based Benchmarks. We assess the factual knowledge, reasoning, coding, mathematical, and instruction-following abilities using automatic metrics on six benchmarks: ARC (Clark et al., 2018), Big-Bench-Hard(BBH) (Suzgun et al., 2022), MMLU (Hendrycks et al., 2021), HumanEval (Chen et al., 2021), GSM8k (Cobbe et al., 2021), and IFEval (Zhou et al., 2023b).

Baselines. We compare our methods against strong data selection approaches: random selection (Xia et al., 2024b), IFD (Li et al., 2024b), ZIP (Yin et al., 2024), #InsTag (Lu et al., 2024), DEITA (Liu et al., 2024b), CaR (Ge et al., 2024), and QDIT (Bukharin et al., 2024). To replicate baselines on Tulu3 and Openhermes2.5, we adjust certain parameters to fit the large-scale datasets, as detailed in Appx A.1.

Training. We use LLaMA3.1-8B (Touvron et al., 2023), Mistral-TB-v0.3 (Jiang et al., 2023), and Qwen2.5-7B (Yang et al., 2024) as our base models and fine-tune them using the Llama-Factory framework (Zheng et al., 2024). Please refer to Appx. A.3 for detailed training setup.

4.2 Main Results

Main Comparison. Table 1 presents the performance of MIG and baselines across benchmarks. All methods select 50K samples based on the grid search (Sec 4.3). With Llama3.1-8B, MIG outperforms all baselines on most tasks, with average improvements of +1.49% and +1.96% over previous state-of-the-art selection methods on knowledge-based and human-preference evaluations, respectively. MIG surpasses QDIT, the second-best
Table 1: Comparison with data selection methods on the Tulu3 pool. HE denotes HumanEval, AE denotes AlpacaEvalv2, MT denotes MTBench, and Wild denotes WildBench. Avg_obj and Avg_sub represent the average of the normalized knowledge-based and human-preference benchmark scores, respectively. Avg is the mean of Avg_obj and Avg_sub. MIG achieves the best performance on Avg_obj*, Avg_sub*, and Avg on all base models.

<table>
  <tr>
    <th rowspan="2">Base Model</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Data Size</th>
    <th colspan="4">ARC</th>
    <th colspan="4">BBH</th>
    <th colspan="4">GSM</th>
    <th colspan="4">HE</th>
    <th colspan="4">MMLU</th>
    <th colspan="4">FEval</th>
    <th colspan="4">Avg_obj</th>
    <th colspan="4">AE</th>
    <th colspan="4">MT</th>
    <th colspan="4">Wild</th>
    <th colspan="4">Avg_sub</th>
    <th colspan="4">Avg</th>
  </tr>
  <tr>
    <th>BBH</th><th>GSM</th><th>HE</th><th>MMLU</th><th>FEval</th><th>Avg_obj</th><th>Avg_sub</th><th>Avg</th><th>BBH</th><th>GSM</th><th>HE</th><th>MMLU</th><th>FEval</th><th>Avg_obj</th><th>Avg_sub</th><th>Avg</th><th>BBH</th><th>GSM</th><th>HE</th><th>MMLU</th><th>FEval</th><th>Avg_obj</th><th>Avg_sub</th><th>Avg</th><th>BBH</th><th>GSM</th><th>HE</th><th>MMLU</th><th>FEval</th><th>Avg_obj</th><th>Avg_sub</th><th>Avg</th>
  </tr>
  <tr>
    <td rowspan="7">Llama3.1-8B</td>
    <td>Pool</td>
    <td>939K</td>
    <td>69.15</td><td>63.88</td><td>83.40</td><td>63.41</td><td>65.77</td><td>67.10</td><td>68.79</td><td>8.94</td><td>6.86</td><td>-24.66</td><td>38.40</td><td>53.59</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>50K</td>
    <td>74.24</td><td>64.80</td><td>70.36</td><td>51.22</td><td>63.86</td><td>61.00</td><td>64.25</td><td>8.57</td><td>7.06</td><td>-22.15</td><td>39.36</td><td>51.81</td>
  </tr>
  <tr>
    <td>ZIP</td>
    <td>50K</td>
    <td>77.63</td><td>63.00</td><td>52.54</td><td>35.98</td><td>65.00</td><td>61.00</td><td>59.19</td><td>6.71</td><td>6.64</td><td>-32.10</td><td>35.69</td><td>47.44</td>
  </tr>
  <tr>
    <td>IFD</td>
    <td>50K</td>
    <td>75.93</td><td>63.56</td><td>61.03</td><td>49.39</td><td>64.39</td><td>53.60</td><td>61.32</td><td>12.30</td><td>7.03</td><td>-20.20</td><td>40.83</td><td>51.08</td>
  </tr>
  <tr>
    <td>#InsTag</td>
    <td>50K</td>
    <td>72.54</td><td>64.80</td><td>69.83</td><td>48.17</td><td>63.50</td><td>65.99</td><td>64.14</td><td>6.58</td><td>6.84</td><td>-20.70</td><td>38.21</td><td>51.17</td>
  </tr>
  <tr>
    <td>DEITA</td>
    <td>50K</td>
    <td>78.98</td><td>66.11</td><td>74.07</td><td>49.39</td><td>64.00</td><td>64.33</td><td>66.15</td><td>10.19</td><td>6.83</td><td>-19.95</td><td>39.50</td><td>52.83</td>
  </tr>
  <tr>
    <td>CaR</td>
    <td>50K</td>
    <td>78.98</td><td>69.04</td><td>71.42</td><td>52.44</td><td>65.15</td><td>56.75</td><td>65.63</td><td>12.55</td><td>6.95</td><td>-20.67</td><td>40.57</td><td>53.10</td>
  </tr>
  <tr>
    <td>QDIT</td>
    <td>50K</td>
    <td>79.66</td><td>63.42</td><td>70.74</td><td>53.05</td><td>65.06</td><td>57.30</td><td>65.21</td><td>15.78</td><td>6.76</td><td>-20.56</td><td>41.03</td><td>53.12</td>
  </tr>
  <tr>
    <td></td>
    <td>MIG</td>
    <td>50K</td>
    <td>80.00</td><td>66.39</td><td>72.02</td><td>57.93</td><td>64.44</td><td>65.06</td><td>67.64</td><td>14.66</td><td>7.32</td><td>-17.77</td><td>42.99</td><td>55.32</td>
  </tr>
  <tr>
    <td rowspan="7">Mistral-7B-v0.3</td>
    <td>Random</td>
    <td>50K</td>
    <td>67.80</td><td>56.90</td><td>66.34</td><td>42.07</td><td>60.34</td><td>65.43</td><td>59.81</td><td>5.84</td><td>6.84</td><td>-25.20</td><td>37.21</td><td>48.51</td>
  </tr>
  <tr>
    <td>ZIP</td>
    <td>50K</td>
    <td>72.88</td><td>56.73</td><td>33.21</td><td>3.05</td><td>61.68</td><td>63.03</td><td>48.43</td><td>5.34</td><td>6.57</td><td>-36.17</td><td>34.32</td><td>41.37</td>
  </tr>
  <tr>
    <td>#InsTag</td>
    <td>50K</td>
    <td>76.27</td><td>57.15</td><td>66.34</td><td>40.85</td><td>61.80</td><td>63.22</td><td>60.94</td><td>8.20</td><td>6.91</td><td>-21.66</td><td>38.82</td><td>49.88</td>
  </tr>
  <tr>
    <td>DEITA</td>
    <td>50K</td>
    <td>75.93</td><td>57.72</td><td>64.82</td><td>11.59</td><td>61.41</td><td>64.51</td><td>56.00</td><td>8.82</td><td>6.96</td><td>-20.51</td><td>39.39</td><td>47.69</td>
  </tr>
  <tr>
    <td>CaR</td>
    <td>50K</td>
    <td>64.41</td><td>58.65</td><td>63.76</td><td>9.15</td><td>61.95</td><td>55.64</td><td>52.26</td><td>11.93</td><td>7.03</td><td>-17.82</td><td>41.11</td><td>46.58</td>
  </tr>
  <tr>
    <td>QDIT</td>
    <td>50K</td>
    <td>54.92</td><td>58.68</td><td>59.97</td><td>42.68</td><td>62.46</td><td>58.23</td><td>56.16</td><td>15.03</td><td>6.84</td><td>-17.74</td><td>41.52</td><td>48.84</td>
  </tr>
  <tr>
    <td></td>
    <td>MIG</td>
    <td>50K</td>
    <td>75.25</td><td>56.19</td><td>66.94</td><td>45.12</td><td>60.23</td><td>64.70</td><td>61.41</td><td>13.66</td><td>7.17</td><td>-18.39</td><td>42.05</td><td>51.73</td>
  </tr>
  <tr>
    <td rowspan="7">Qwen2.5-7B</td>
    <td>Pool</td>
    <td>939K</td>
    <td>90.51</td><td>65.01</td><td>85.29</td><td>78.05</td><td>75.15</td><td>64.88</td><td>76.31</td><td>9.07</td><td>7.04</td><td>-23.98</td><td>39.16</td><td>57.74</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>50K</td>
    <td>85.42</td><td>63.87</td><td>80.74</td><td>79.27</td><td>73.81</td><td>58.04</td><td>75.53</td><td>10.56</td><td>7.18</td><td>-18.08</td><td>41.11</td><td>57.32</td>
  </tr>
  <tr>
    <td>ZIP</td>
    <td>50K</td>
    <td>85.76</td><td>63.43</td><td>83.24</td><td>72.56</td><td>73.60</td><td>58.23</td><td>72.80</td><td>7.45</td><td>7.33</td><td>-27.83</td><td>38.94</td><td>55.87</td>
  </tr>
  <tr>
    <td>#InsTag</td>
    <td>50K</td>
    <td>88.81</td><td>63.03</td><td>84.61</td><td>81.10</td><td>73.50</td><td>61.00</td><td>75.34</td><td>9.07</td><td>7.52</td><td>-18.80</td><td>41.62</td><td>58.48</td>
  </tr>
  <tr>
    <td>DEITA</td>
    <td>50K</td>
    <td>89.15</td><td>63.22</td><td>86.13</td><td>79.27</td><td>74.27</td><td>58.78</td><td>75.14</td><td>10.31</td><td>7.28</td><td>-19.71</td><td>41.09</td><td>58.11</td>
  </tr>
  <tr>
    <td>CaR</td>
    <td>50K</td>
    <td>91.86</td><td>60.60</td><td>87.64</td><td>77.44</td><td>73.97</td><td>50.28</td><td>74.47</td><td>13.66</td><td>7.39</td><td>-20.77</td><td>42.39</td><td>58.43</td>
  </tr>
  <tr>
    <td>QDIT</td>
    <td>50K</td>
    <td>89.83</td><td>69.34</td><td>87.04</td><td>81.10</td><td>74.72</td><td>50.83</td><td>75.48</td><td>13.79</td><td>7.10</td><td>-20.46</td><td>41.52</td><td>58.50</td>
  </tr>
  <tr>
    <td></td>
    <td>MIG</td>
    <td>50K</td>
    <td>90.51</td><td>67.39</td><td>84.46</td><td>79.88</td><td>73.85</td><td>61.74</td><td>76.30</td><td>11.80</td><td>7.54</td><td>-14.49</td><td>43.32</td><td>59.81</td>
  </tr>
</table>

Table 2: Results on different data pools, Openhermes2.5 and X_sota, based on Llama3.1-8B. MIG outperforms all baselines across both data pools. Please refer to Table 5 in Appx. D for detailed scores on all benchmarks.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">Openhermes2.5</th>
    <th colspan="4">X_sota</th>
  </tr>
  <tr>
    <th>Data Size</th>
    <th>Avg_obj</th>
    <th>Avg_sub</th>
    <th>Avg</th>
    <th>Data Size</th>
    <th>Avg_obj</th>
    <th>Avg_sub</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Pool</td>
    <td>1M</td>
    <td>36.91</td><td>61.49</td><td>49.20</td>
    <td>306K</td>
    <td>31.51</td><td>52.88</td><td>42.19</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>50K</td>
    <td>32.99</td><td>55.69</td><td>44.34</td>
    <td>6K</td>
    <td>29.94</td><td>49.69</td><td>39.81</td>
  </tr>
  <tr>
    <td>#InsTag</td>
    <td>50K</td>
    <td>36.23</td><td>54.12</td><td>45.17</td>
    <td>6K</td>
    <td>31.89</td><td>46.19</td><td>39.04</td>
  </tr>
  <tr>
    <td>DEITA</td>
    <td>50K</td>
    <td>36.80</td><td>57.36</td><td>47.08</td>
    <td>6K</td>
    <td>31.60</td><td>48.70</td><td>40.15</td>
  </tr>
  <tr>
    <td>CaR</td>
    <td>50K</td>
    <td>37.51</td><td>55.57</td><td>46.54</td>
    <td>6K</td>
    <td>31.86</td><td>48.43</td><td>40.15</td>
  </tr>
  <tr>
    <td>QDIT</td>
    <td>50K</td>
    <td>37.98</td><td>57.71</td><td>47.80</td>
    <td>6K</td>
    <td>32.52</td><td>49.10</td><td>40.81</td>
  </tr>
  <tr>
    <td>MIG</td>
    <td>50K</td>
    <td>38.12</td><td>58.30</td><td>48.21</td>
    <td>6K</td>
    <td>32.98</td><td>50.63</td><td>41.80</td>
  </tr>
</table>

method, by **+2.20%** on overall Avg score. Notably, the model trained on 5% data sampled by MIG outperforms the model trained on the full Tulu3 pool by **+4.59%** on human-preference benchmarks while maintaining comparable knowledge-based performance. Additionally, MIG significantly outperforms embedding-based methods in sampling efficiency due to reduced computational overhead. Please refer to Table 4 in Appx. B for detailed sampling times and efficiency analysis.

**Transferability on Models.** Table 1 presents results for Mistral-7B and Qwen2.5-7B. MIG consistently surpasses baselines with Avg improvements of **+1.85%** and **+1.31%**, respectively, demonstrating its robustness. Notably, the second-best selection method varies among different base models, further demonstrating the generalizability of MIG.

**Transferability on Data Pools.** Table 2 presents results on different data pools with varying sizes and quality. MIG consistently outperforms all baselines, achieving Avg improvements of **+0.41%** and **+0.99%** over previous best methods, further demonstrating its generalizability. Notably, on *X_sota*, all baselines exhibit performance degradation on knowledge-based evaluations, consistent with the findings in (Xia et al., 2024b). We hypothesize that quality metrics, such as DEITA scores and tag counts, are biased toward multi-round, long samples that enhance subjective chat abilities. However, samples in specific domains, such as math and code, are typically single-turn. MIG mitigates this bias by effectively balancing quality and diversity.

**Data Scaling.** We compare MIG with baseline methods across varying data budgets on the Tulu3 pool, using Llama3.1-8B as the base model. As

![Data scaling experiments on Tulu3 using Llama3.1-8B. The score reported here is the Avg score.](page_1092_1342_377_246.png)
Figure 4: (a) Derivative of Information Score Functions. (b) Avg\(_{obj}\) on Different Information Score Functions. (c) Avg\(_{subj}\) on Different Quality Scores.

![Three line plots showing derivative of information score functions, average objective scores, and average subjective scores for different information score functions](page_120_120_1208_370.png)

Figure 5: Quantitative results on different quality metrics. DEITA scores achieve the best performance on both human-preference and knowledge-based evaluations.

![Bar chart comparing IFD, TagNum, and DEITA quality scores](page_120_600_1208_370.png)

shown in Fig 3, MIG consistently delivers superior performance at each data budget, demonstrating its robust scalability. Remarkably, MIG achieves comparable performance to the full dataset with only 20K samples, underscoring its efficiency. The observed initial increase and subsequent plateau in performance align closely with findings from previous works (Li et al., 2024b; Liu et al., 2024b), highlighting the importance of data selection.

4.3 Analysis

Information Score Function \( \Phi \). The information score function \( \Phi \) is crucial in MIG sampling as it balances quality and diversity. Based on the principles outlined in Sec. 3.2, \( \Phi \) is expected to be monotonically increasing with a diminishing rate of increase. In our experiments, we evaluate two candidate functions:

\[
\Phi(x) = 1 - e^{-\alpha x} \quad (\alpha > 0) \tag{12}
\]
\[
\Phi(x) = x^{\alpha x} \quad (0 < \alpha < 1) \tag{13}
\]

Fig. 4(a) compares the decreasing rate in the derivative of these functions under varying parameter settings. Functions that decay rapidly tend to favor diverse label distributions as the information on any given label converges quickly. Fig. 4(b)(c) present the performance on different evaluations, with \( \Phi(x) = x^{0.8} \) achieving the best results on human-preference and knowledge-based benchmarks, effectively balancing quality and diversity.

Quality Metrics. We implement three alternative quality measurement approaches: the number of tags (Lu et al., 2024), the IFD score (Li et al., 2024b), and the DEITA score (Liu et al., 2024b), to investigate their impact on information measurement. Fig. 5 compares these three quality metrics with a baseline score that assigns a constant value to all samples. The DEITA scores consistently outperform the other quality metrics in both evaluation settings. Therefore, we adopt the DEITA scores as the default quality measurements for MIG.

Label Graph. An essential question in MIG is how to determine an appropriate label graph, including its nodes (label set) and edges (label relationships). Increasing the number of nodes leads to a more granular label set, thereby providing broader coverage of knowledge topics. However, excessively large label sets inevitably include outliers or low-quality labels. Similarly, increasing edge density between labels enhances the comprehensiveness of label relationships, but overly dense graphs may result in computational inefficiencies and noise from the embedding model. There is no universally optimal solution, as the ideal label graph depends on the characteristics of the data pool and potentially other parameters in MIG. To explore the relationship between the label graph and the downstream performance of trained models, we conduct an empirical experiment on the Tulu3 pool. Fig. 6(a) shows the downstream performance from a set of node counts in the label graph, ranging from 839 to 6738, while Fig. 6(b) presents performance across varying edge densities, with thresholds between 0.8 and 0.94. The observed trends align with our
Figure 6: Analysis of Parameters in the Label Graph. The reported score is the average of Avg_subh and Avg_subh'. Please refer to Table 7 8 9 in Appx. D for detailed scores on all evaluated benchmarks. (a) Comparison of various node counts (label set size) in the label graph. (b) Comparison of different edge thresholds, with a lower threshold indicating a dense graph. (c) Comparison of different propagation weights, where a smaller weight corresponds to weak propagation.

Table 3: Grid search of appropriate data size and training epochs on the Tulu3 pool. We report the AVG score here.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">Random</th>
    <th colspan="3">MIG</th>
  </tr>
  <tr>
    <th>Epoch2</th>
    <th>Epoch3</th>
    <th>Epoch4</th>
    <th>Epoch2</th>
    <th>Epoch3</th>
    <th>Epoch4</th>
  </tr>
  <tr>
    <td>10K</td>
    <td>46.76</td>
    <td>49.42</td>
    <td>50.39</td>
    <td>49.13</td>
    <td>52.36</td>
    <td>51.11</td>
  </tr>
  <tr>
    <td>20K</td>
    <td>48.23</td>
    <td>50.36</td>
    <td>51.08</td>
    <td>51.18</td>
    <td>53.71</td>
    <td>53.14</td>
  </tr>
  <tr>
    <td>50K</td>
    <td>49.78</td>
    <td><b>51.81</b></td>
    <td>50.68</td>
    <td>52.88</td>
    <td><b>55.32</b></td>
    <td>55.14</td>
  </tr>
</table>

initial analysis, showing an unimodal performance curve in both experiments. For the Tulu3 pool, the optimal label graph is achieved with a label set size of 4531 and an edge similarity threshold of 0.9.

Information Propagation. We conduct a series of experiments to study the impact of information propagation intensity in MIG sampling. Appropriate information propagation results in accurate information distribution in the semantic space. Specifically, we experiment with various values of \( \alpha \) in Eq. 6, where \( \alpha \) is proportional to the intensity of information propagation. Fig 6(c) shows that \( \alpha = 1.0 \) yields the best performance, with Avg improvement of 2.76 over the non-propagation. It indicates that information propagation effectively improves the accuracy of information measurement on the label graph.

Grid Search. To identify an appropriate data bucket and training epoch on the Tulu3 pool for the main comparison, we perform a grid search. Results in Table 3 indicate 50K samples with three training epochs as optimal for Tulu3, consistently maximizing performance for MIG and random selection.

5 Conclusion

In this paper, we propose a novel method for measuring instruction-tuning datasets in semantic space. We model the semantic space as a label graph and jointly evaluate data quality and diversity. We introduce an upper-convex information score function to balance quality and diversity, and propose information propagation to capture the information distribution accurately Building on the submodularity of such measurement, we propose MIG, an efficient sampling algorithm that iteratively selects samples to maximize the information gain on the label graph. Extensive experiments across diverse data pools and base models validate the effectiveness and generalizability of MIG. Our research bridges the gap between instance-level quality assessment and global dataset-level evaluation, offering a unified approach to dataset measurement. We hope our results can inspire dataset measurement-guided data selection in the future.

Limitation. Currently, the parameters in MIG are static and depend on grid search to identify the optimal values, which can not be extensively explored. Future work could focus on developing methods to automatically determine the parameters in MIG, such as customizing the information score function for each label, to enhance the flexibility and scalability of MIG.

Acknowledgements

This work was supported by National Key R&D Program of China 2022ZD0161600.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NIPS.

Alexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang, Bing Yin, Xian Li, Chao Zhang, Tuo Zhao, and Haoming Jiang. 2024. Data diversity matters for robust instruction tuning. In EMNLP.

Maosong Cao, Alexander Lam, Haodong Duan, Hongwei Liu, Songyang Zhang, and Kai Chen. 2024a. Compassjudger-1: All-in-one judge model helps model evaluation and evolution. arXiv preprint arXiv:2410.16256.

Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. 2024b. Instruction mining: Instruction data selection for tuning large language models. In COLM.

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024. Alpagaus: Training a better alpaca with fewer data. In ICLR.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chatzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.

Gérard Cornuéjols, George Nemhauser, and Laurence Wolsey. 1983. The uncapacitated facility location problem. Technical report, Cornell University Operations Research and Industrial Engineering.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP.

Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. How abilities in large language models are affected by supervised fine-tuning data composition. In ACL.

Yann Dubois, Balázs Galambosi, Percy Liang, and Tat-sunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475.

Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Mahong Xia, Zhang Li, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, and JingBo Zhu. 2024. Clustering and ranking: Diversity-preserved instruction selection through expert-aligned quality estimation. In EMNLP.

Michael Hahsler, Matthew Piekenbrock, and Derek Doran. 2019. dbscan: Fast density-based clustering with r. Journal of Statistical Software.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In ICLR.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. 2024. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124.

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizhullin, and Bernard Ghanem. 2023a. CAMEL: Communicative agents for “mind” exploration of large language model society. In NIPS.
Ming Li, Yong Zhang, Shuai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024a. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. In ACL.

Ming Li, Yong Zhang, Zhitao Li, Juhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024b. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In NAACL.

Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxyi Yang, Min Yang, Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et al. 2023b. One shot learning as instruction data prospector for large language models. arXiv preprint arXiv:2312.10302.

Bill Yuchen Lin, Yuntian Deng, Khvathi Chaudu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770.

Liangxin Liu, Xuebo Liu, Derek F. Wong, Dongfang Li, Ziyi Wang, Baotian Hu, and Min Zhang. 2024a. SelectIT: Selective instruction tuning for LLMs via uncertainty-aware self-reflection. In NIPS.

Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024b. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In ICLR.

Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2024. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In ICLR.

Michel Minoux. 2005. Accelerated greedy algorithms for maximizing submodular set functions. In Optimization Techniques: Proceedings of the 8th IFIP Conference on Optimization Techniques Würzburg, September 5–9, 1977.

George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. 1978. An analysis of approximations for maximizing submodular set functions—i. Mathematical programming.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP.

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Teknium. 2023. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023a. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023b. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368.

Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. 2024. Diversity measurement and subset selection for instruction tuning datasets. arXiv preprint arXiv:2402.02318.

Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. 2023. Self-evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182.

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024a. LESS: Selecting influential data for targeted instruction tuning. In ICML.

Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, and Junyang Lin. 2024b. Rethinking data selection at scale: Random selection is almost all you need. arXiv preprint arXiv:2410.09335.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Mingjia Yin, Chuhuan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. 2024. Entropy Law: The Story Behind Data Compression and LLM Performance. arXiv preprint arXiv:2407.06645.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024a. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR.
Simon Yu, Liangyu Chen, Sara Ahmadian, and Marzieh Fadaee. 2024b. Diversify and conquer: Diversity-centric data selection with iterative refinement. arXiv preprint arXiv:2409.11378.

Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Minghao Li, Fei Huang, Nevin L. Zhang, and Yongbin Li. 2024. Tree-instruct: A preliminary study of the intrinsic relationship between complexity and alignment. In COLING.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In NIPS.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyen Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In ACL.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023a. Lima: Less is more for alignment. In NIPS.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023b. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
A  Details of Experiments Setup

A.1 Baseline Settings

The specific baseline settings in our experiments are as follows:

• Random. A fixed random seed of 42 is used to ensure reproducibility.
• IFD (Li et al., 2024b). We follow the setting from (Li et al., 2024a) to directly compute IFD scores using base LLMs for efficiency.
• ZIP (Yin et al., 2024). The default setting is applied across all data pools.
• #InsTag (Lu et al., 2024). The open-released InsTagger is used to tag data from the pool, followed by tag normalization, which includes frequency filtering and semantic aggregation. The frequency threshold is set to 2, as InsTagger is fully trained on valid normalized tags, resulting in 9471 tags. For semantic aggregation, we use E5-Mistral-7B-Instruct (Wang et al., 2023b) to generate tag embeddings, and the DBSCAN algorithm (Hahsler et al., 2019) is applied with a semantic similarity threshold of 0.05, yielding 6738 tags.
• DEITA (Liu et al., 2024b). For \( X_{sota} \), we use the released sampled dataset. For the Openhermes2.5 (Teknium, 2023) and Tulu3 (Lambert et al., 2024) pools, we reproduce its method. Quality assessment is conducted using the released quality and complexity scorers. For Repr Filter, we utilize Llama3.1-8B to obtain instance embeddings. A threshold of 0.9 is applied to the Tulu3 pool, and 0.95 is used for the Openhermes2.5 pool, as the 0.9 threshold results in insufficient samples for the latter.
• CaR (Ge et al., 2024). We follow the default setting, using all-mpnet-base-v2 (Reimers and Gurevych, 2019) to obtain data embeddings, PCA to retain 95% of dimensions, and k-Means clustering with the number of clusters as \( k = \sqrt{n/2} \). For ranking, we use the released IQS model. We maintain the original ratio between \( n_1 \) and \( kn_2 \) for different pools.
• QDIT (Bukharin et al., 2024). Embedding computation follows the default setting with all-mpnet-base-v2. Quality assessment uses the DEITA scores instead of ChatGPT. The hyperparameter \( \alpha \) for balancing quality and diversity is set to 0.9 for \( X_{sota} \) and 0.7 for the Tulu3 and Openhermes2.5 pools.

A.2 Implementation Details

The label set in MIG follows #InsTag (Lu et al., 2024), with variations in the label graph across different data pools. Specifically, 3059 tags are used for \( X_{sota} \), 4531 for Tulu3, and 5166 for Openhermes2.5, with label set size positively correlated to pool size. E5-Mistral-7B-Instruct is used as the embedding model to compute label similarity, with a threshold set to 0.9. Quality assessment in MIG uses the DEITA scores. The information score function is set to \( \phi(x) = x^{0.8} \), and the information propagation weight \( \alpha \) is set to 1.

A.3 Training Recipes

For experiments on \( X_{sota} \), we follow the default settings from (Liu et al., 2024b), using a batch size of 128, a learning rate of 2e-5, a warm ratio of 0.1, and a maximum input length of 2048. For the Tulu3 pool, we adopt the settings from (Lambert et al., 2024), with a batch size of 128, a learning rate of 5e-6, a warm ratio of 0.03, and a maximum input length of 4096. For the Openhermes2.5 pool, we follow the settings from (Xia et al., 2024b), setting the batch size to 128, learning rates to 7e-6, a warm ratio of 0.01, and a maximum input length of 4096.

A.4 Evaluation Setup

The evaluation of our experiments is implemented using OpenCompass (Contributors, 2023) with greedy inference to ensure uniform evaluation across all models.

Human-preference Evaluations. We use the open-source CompassJudger-1-32B (Cao et al., 2024a) for human-preference evaluation. As different benchmarks use different scoring metrics and ranges, we normalize the scores according to the following mapping:
• AlpacaEvalv2 (Dubois et al., 2024): The score range is 0-100, requiring no special adjustment.
• MTBench (Zheng et al., 2023): The score range is 0-10, which is mapped by multiplying by 10.
• WildBench (Lin et al., 2024): The score range is -100-100, which is normalized by adding 100 and dividing by 2.

Knowledge-based Evaluations. We conduct evaluations using the following settings: three-shot evaluation on BBH (Suzgun et al., 2022), five-shot evaluation on MMLU (Hendrycks et al., 2021), and zero-shot evaluation on ARC (Clark et al., 2018) and GSM8K (Cobbe et al., 2021). For Hu-
Table 4: Efficiency comparison of different methods for 50K sampling from the Tulu3 pool, with timing measured on a single NVIDIA-L20Y.

<table>
  <tr>
    <th>Method</th>
    <th>GPU</th>
    <th>Time</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>x</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>ZIP (Yin et al., 2024)</td>
    <td>x</td>
    <td>53.99</td>
  </tr>
  <tr>
    <td>IFD (Li et al., 2024b)</td>
    <td>x</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>#InsTag (Lu et al., 2024)</td>
    <td>x</td>
    <td>2.33</td>
  </tr>
  <tr>
    <td>DEITA (Liu et al., 2024b)</td>
    <td>✓</td>
    <td>81.56</td>
  </tr>
  <tr>
    <td>QDIT (Bukharin et al., 2024)</td>
    <td>✓</td>
    <td>86.17</td>
  </tr>
  <tr>
    <td>CaR (Ge et al., 2024)</td>
    <td>✓</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>MIG</td>
    <td>✓</td>
    <td>0.45</td>
  </tr>
</table>

manEval (Chen et al., 2021), we report pass@1 results, and for IFEval (Zhou et al., 2023b), we provide strictly followed scores.

B Efficiency Analysis

Table 4 presents the time used for 50K sampling on the Tulu3 pool. Among methods that balance quality and diversity, MIG demonstrates the highest efficiency. Notably, MIG outperforms QDIT (Bukharin et al., 2024) and DEITA (Liu et al., 2024b) significantly, as it eliminates the need for iterative pairwise similarity computations in the embedding space.

C Theoretical Analysis of Greedy Strategy in MIG

C.1 Submodularity Analysis

We first prove that our dataset measurement function \( E(D) \), defined in Eq. 8, is submodular. Specifically, for all \( D \subseteq T \), and for all elements \( e \notin T \), the following inequality holds:

\[
E(D \cup e) - E(D) \geq E(T \cup e) - E(T)
\] (14)

Proof. Let \( z(D) = A \sum_{i \in D} s_i v_i \). The marginal gain from adding an element \( e \) to \( D \) is:

\[
\Delta(D, e) = \sum_{k=1}^n [\phi(z_k(D) + \Delta_k(e)) - \phi(z_k(D))]
\] (15)

where \( \Delta_k(e) = s_e (A v_e)_k \geq 0 \) represents the incremental contribution of \( e \) to the \( k \)-th component. Since \( \phi \) is monotonically increasing and concave, for any \( \delta \geq 0 \) and \( z' \geq z \), we have:

\[
\phi(z + \delta) - \phi(z) \geq \phi(z' + \delta) - \phi(z')
\] (16)

Given \( D \subseteq T \), we have:

\[
z_k(T) = z_k(D) + \sum_{i \in T \setminus D} s_i (A v_i)_k \geq z_k(D)
\] (17)

Thus, by the concavity property of \( \phi \):

\[
\begin{align*}
\phi(z_k(D) + \Delta_k(e)) - \phi(z_k(D)) &\geq \\
\phi(z_k(T) + \Delta_k(e)) - \phi(z_k(T))
\end{align*}
\] (18)

Summing over all components \( k \), we get:

\[
\Delta(D, e) \geq \Delta(T, e)
\] (19)

Thus, \( E(D \cup e) - E(D) \geq E(T \cup e) - E(T) \), which satisfies the definition of submodularity. \( \square \)

C.2 Cardinality-Constrained Submodular Maximization

Given that \( E(D) \) is submodular, our data selection task, defined in Eq. 1, constitutes a cardinality-constrained submodular maximization problem that is NP-complete. However, a greedy algorithm provides a well-established approximation guarantee, ensuring that \( E(D^{greedy}) \geq (1 - \frac{1}{e})E(D^*) \), where \( D^* \) represents the optimal solution (Nemhauser et al., 1978). Assuming \( P \neq NP \), this guarantee is the best achievable for polynomial-time algorithms.

D Detailed Results on Benchmarks

We provide detailed scores on full benchmarks in Table 5 6 7 8 9.
Table 5: Full Results on the Openhermes2.5 Pool.

<table>
  <tr>
    <th>Method</th>
    <th>ARC</th>
    <th>BBH</th>
    <th>GSM8K</th>
    <th>HumanEval</th>
    <th>MMLU</th>
    <th>IFEval</th>
    <th>Avg_obj</th>
    <th>AlpacaEval</th>
    <th>MTbench</th>
    <th>Wildbench</th>
    <th>Avg_sub</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>Pool</td>
    <td>72.88</td>
    <td>60.53</td>
    <td>70.51</td>
    <td>51.22</td>
    <td>64.99</td>
    <td>48.80</td>
    <td>61.49</td>
    <td>5.47</td>
    <td>7.10</td>
    <td>-31.51</td>
    <td>36.91</td>
    <td>49.20</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>75.25</td>
    <td>60.20</td>
    <td>51.40</td>
    <td>50.00</td>
    <td>51.23</td>
    <td>46.03</td>
    <td>55.69</td>
    <td>4.72</td>
    <td>6.63</td>
    <td>-44.12</td>
    <td>32.99</td>
    <td>44.34</td>
  </tr>
  <tr>
    <td>InsTag</td>
    <td>70.85</td>
    <td>68.64</td>
    <td>56.25</td>
    <td>43.90</td>
    <td>45.70</td>
    <td>49.35</td>
    <td>54.12</td>
    <td>5.09</td>
    <td>7.14</td>
    <td>-35.60</td>
    <td>36.23</td>
    <td>45.17</td>
  </tr>
  <tr>
    <td>DEITA</td>
    <td>69.83</td>
    <td>61.85</td>
    <td>60.96</td>
    <td>46.95</td>
    <td>58.01</td>
    <td>46.58</td>
    <td>57.36</td>
    <td>7.83</td>
    <td>6.94</td>
    <td>-33.69</td>
    <td>36.80</td>
    <td>47.08</td>
  </tr>
  <tr>
    <td>CaR</td>
    <td>62.71</td>
    <td>63.73</td>
    <td>55.42</td>
    <td>44.51</td>
    <td>64.37</td>
    <td>42.70</td>
    <td>55.57</td>
    <td>7.33</td>
    <td>7.09</td>
    <td>-31.43</td>
    <td>37.51</td>
    <td>46.54</td>
  </tr>
  <tr>
    <td>QDIT</td>
    <td>66.44</td>
    <td>62.45</td>
    <td>58.61</td>
    <td>50.00</td>
    <td>63.64</td>
    <td>45.10</td>
    <td>57.71</td>
    <td>9.19</td>
    <td>6.99</td>
    <td>-30.78</td>
    <td>37.90</td>
    <td>47.80</td>
  </tr>
  <tr>
    <td>MIG</td>
    <td>78.98</td>
    <td>63.33</td>
    <td>51.55</td>
    <td>45.73</td>
    <td>63.81</td>
    <td>46.40</td>
    <td>58.30</td>
    <td>7.83</td>
    <td>7.17</td>
    <td>-30.34</td>
    <td>38.12</td>
    <td>48.21</td>
  </tr>
</table>

Table 6: Full Results on the X_sota Pool.

<table>
  <tr>
    <th>Method</th>
    <th>ARC</th>
    <th>BBH</th>
    <th>GSM8K</th>
    <th>HumanEval</th>
    <th>MMLU</th>
    <th>IFEval</th>
    <th>Avg_obj</th>
    <th>AlpacaEval</th>
    <th>MTbench</th>
    <th>Wildbench</th>
    <th>Avg_sub</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>Pool</td>
    <td>73.22</td>
    <td>54.12</td>
    <td>40.49</td>
    <td>45.12</td>
    <td>61.05</td>
    <td>43.25</td>
    <td>52.88</td>
    <td>3.85</td>
    <td>6.78</td>
    <td>-54.21</td>
    <td>31.51</td>
    <td>42.19</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>61.02</td>
    <td>58.12</td>
    <td>32.07</td>
    <td>42.69</td>
    <td>62.31</td>
    <td>41.96</td>
    <td>49.69</td>
    <td>3.60</td>
    <td>6.34</td>
    <td>-54.39</td>
    <td>29.94</td>
    <td>39.81</td>
  </tr>
  <tr>
    <td>InsTag</td>
    <td>64.07</td>
    <td>51.82</td>
    <td>36.62</td>
    <td>28.66</td>
    <td>55.11</td>
    <td>40.85</td>
    <td>46.19</td>
    <td>5.22</td>
    <td>6.56</td>
    <td>-50.28</td>
    <td>31.89</td>
    <td>39.04</td>
  </tr>
  <tr>
    <td>DEITA</td>
    <td>71.86</td>
    <td>50.82</td>
    <td>27.67</td>
    <td>40.24</td>
    <td>63.36</td>
    <td>38.26</td>
    <td>48.70</td>
    <td>4.22</td>
    <td>6.48</td>
    <td>-48.44</td>
    <td>31.60</td>
    <td>40.15</td>
  </tr>
  <tr>
    <td>CaR</td>
    <td>72.88</td>
    <td>48.90</td>
    <td>20.92</td>
    <td>46.95</td>
    <td>62.68</td>
    <td>38.26</td>
    <td>48.43</td>
    <td>5.22</td>
    <td>6.51</td>
    <td>-49.46</td>
    <td>31.86</td>
    <td>40.15</td>
  </tr>
  <tr>
    <td>QDIT</td>
    <td>71.53</td>
    <td>51.48</td>
    <td>29.95</td>
    <td>41.46</td>
    <td>63.22</td>
    <td>36.97</td>
    <td>49.10</td>
    <td>5.09</td>
    <td>6.55</td>
    <td>-46.05</td>
    <td>32.52</td>
    <td>40.81</td>
  </tr>
  <tr>
    <td>MIG</td>
    <td>74.58</td>
    <td>51.93</td>
    <td>31.54</td>
    <td>43.90</td>
    <td>62.24</td>
    <td>39.56</td>
    <td>50.63</td>
    <td>5.34</td>
    <td>6.72</td>
    <td>-47.18</td>
    <td>32.98</td>
    <td>41.80</td>
  </tr>
</table>

Table 7: Full Results across Different Node Numbers.

<table>
  <tr>
    <th>Node Number</th>
    <th>ARC</th>
    <th>BBH</th>
    <th>GSM8K</th>
    <th>HumanEval</th>
    <th>MMLU</th>
    <th>IFEval</th>
    <th>Avg_obj</th>
    <th>AlpacaEval</th>
    <th>MTbench</th>
    <th>Wildbench</th>
    <th>Avg_sub</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>839</td>
    <td>73.00</td>
    <td>66.35</td>
    <td>73.31</td>
    <td>52.44</td>
    <td>64.42</td>
    <td>61.18</td>
    <td>65.27</td>
    <td>16.02</td>
    <td>7.18</td>
    <td>-19.59</td>
    <td>42.67</td>
    <td>53.97</td>
  </tr>
  <tr>
    <td>1629</td>
    <td>76.27</td>
    <td>66.22</td>
    <td>72.18</td>
    <td>56.71</td>
    <td>64.71</td>
    <td>58.96</td>
    <td>65.84</td>
    <td>15.16</td>
    <td>7.40</td>
    <td>-18.15</td>
    <td>43.36</td>
    <td>54.60</td>
  </tr>
  <tr>
    <td>3070</td>
    <td>78.31</td>
    <td>66.80</td>
    <td>72.25</td>
    <td>55.49</td>
    <td>65.03</td>
    <td>63.22</td>
    <td>66.85</td>
    <td>14.04</td>
    <td>7.25</td>
    <td>-17.63</td>
    <td>42.58</td>
    <td>54.71</td>
  </tr>
  <tr>
    <td>4531</td>
    <td>80.00</td>
    <td>66.39</td>
    <td>72.02</td>
    <td>57.93</td>
    <td>64.44</td>
    <td>65.06</td>
    <td>67.64</td>
    <td>14.66</td>
    <td>7.32</td>
    <td>-17.77</td>
    <td>42.99</td>
    <td>55.32</td>
  </tr>
  <tr>
    <td>5683</td>
    <td>83.73</td>
    <td>66.47</td>
    <td>72.55</td>
    <td>55.49</td>
    <td>64.22</td>
    <td>64.14</td>
    <td>67.77</td>
    <td>12.55</td>
    <td>7.35</td>
    <td>-20.15</td>
    <td>41.99</td>
    <td>54.88</td>
  </tr>
  <tr>
    <td>6738</td>
    <td>76.27</td>
    <td>65.92</td>
    <td>70.13</td>
    <td>52.44</td>
    <td>64.78</td>
    <td>64.14</td>
    <td>65.61</td>
    <td>11.30</td>
    <td>7.36</td>
    <td>-19.87</td>
    <td>41.65</td>
    <td>53.63</td>
  </tr>
</table>

Table 8: Full Results across Different Edge Densities.

<table>
  <tr>
    <th>Node Number</th>
    <th>ARC</th>
    <th>BBH</th>
    <th>GSM8K</th>
    <th>HumanEval</th>
    <th>MMLU</th>
    <th>IFEval</th>
    <th>Avg_obj</th>
    <th>AlpacaEval</th>
    <th>MTbench</th>
    <th>Wildbench</th>
    <th>Avg_sub</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>0.8</td>
    <td>77.97</td>
    <td>65.69</td>
    <td>70.96</td>
    <td>54.88</td>
    <td>64.59</td>
    <td>63.96</td>
    <td>66.34</td>
    <td>11.3</td>
    <td>7.24</td>
    <td>-21.45</td>
    <td>40.99</td>
    <td>53.67</td>
  </tr>
  <tr>
    <td>0.86</td>
    <td>80.68</td>
    <td>65.72</td>
    <td>71.49</td>
    <td>55.49</td>
    <td>64.15</td>
    <td>62.85</td>
    <td>66.73</td>
    <td>14.04</td>
    <td>7.21</td>
    <td>-19.77</td>
    <td>42.08</td>
    <td>54.41</td>
  </tr>
  <tr>
    <td>0.88</td>
    <td>76.95</td>
    <td>67.77</td>
    <td>72.02</td>
    <td>57.93</td>
    <td>64.99</td>
    <td>62.85</td>
    <td>67.09</td>
    <td>13.42</td>
    <td>7.04</td>
    <td>-20.17</td>
    <td>41.25</td>
    <td>54.17</td>
  </tr>
  <tr>
    <td>0.9</td>
    <td>80.00</td>
    <td>66.39</td>
    <td>72.02</td>
    <td>57.93</td>
    <td>64.44</td>
    <td>65.06</td>
    <td>67.64</td>
    <td>14.66</td>
    <td>7.32</td>
    <td>-17.77</td>
    <td>42.99</td>
    <td>55.32</td>
  </tr>
  <tr>
    <td>0.92</td>
    <td>83.73</td>
    <td>66.39</td>
    <td>71.80</td>
    <td>58.54</td>
    <td>64.55</td>
    <td>63.43</td>
    <td>68.37</td>
    <td>12.55</td>
    <td>6.97</td>
    <td>-19.29</td>
    <td>40.87</td>
    <td>54.62</td>
  </tr>
  <tr>
    <td>0.94</td>
    <td>78.51</td>
    <td>65.86</td>
    <td>70.56</td>
    <td>56.10</td>
    <td>64.81</td>
    <td>64.14</td>
    <td>66.08</td>
    <td>10.68</td>
    <td>7.15</td>
    <td>-19.42</td>
    <td>40.82</td>
    <td>53.71</td>
  </tr>
</table>

Table 9: Full Results across Different Propagation Weights.

<table>
  <tr>
    <th>Node Number</th>
    <th>ARC</th>
    <th>BBH</th>
    <th>GSM8K</th>
    <th>HumanEval</th>
    <th>MMLU</th>
    <th>IFEval</th>
    <th>Avg_obj</th>
    <th>AlpacaEval</th>
    <th>MTbench</th>
    <th>Wildbench</th>
    <th>Avg_sub</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>0</td>
    <td>74.24</td>
    <td>65.39</td>
    <td>71.04</td>
    <td>49.39</td>
    <td>64.62</td>
    <td>63.29</td>
    <td>64.71</td>
    <td>10.81</td>
    <td>7.09</td>
    <td>-21.01</td>
    <td>40.40</td>
    <td>52.56</td>
  </tr>
  <tr>
    <td>0.6</td>
    <td>80.68</td>
    <td>66.78</td>
    <td>72.33</td>
    <td>54.88</td>
    <td>64.66</td>
    <td>63.40</td>
    <td>67.12</td>
    <td>12.29</td>
    <td>7.13</td>
    <td>-19.87</td>
    <td>41.87</td>
    <td>54.34</td>
  </tr>
  <tr>
    <td>0.8</td>
    <td>78.78</td>
    <td>66.46</td>
    <td>73.54</td>
    <td>54.88</td>
    <td>64.83</td>
    <td>64.14</td>
    <td>67.14</td>
    <td>14.04</td>
    <td>7.14</td>
    <td>-17.71</td>
    <td>42.19</td>
    <td>54.67</td>
  </tr>
  <tr>
    <td>1.0</td>
    <td>80.00</td>
    <td>66.39</td>
    <td>72.02</td>
    <td>57.93</td>
    <td>64.44</td>
    <td>65.06</td>
    <td>67.64</td>
    <td>14.66</td>
    <td>7.32</td>
    <td>-17.77</td>
    <td>42.99</td>
    <td>55.32</td>
  </tr>
  <tr>
    <td>1.2</td>
    <td>80.00</td>
    <td>67.42</td>
    <td>73.39</td>
    <td>52.44</td>
    <td>64.89</td>
    <td>64.14</td>
    <td>67.04</td>
    <td>12.42</td>
    <td>7.20</td>
    <td>-17.72</td>
    <td>41.85</td>
    <td>54.45</td>
  </tr>
  <tr>
    <td>2.0</td>
    <td>81.36</td>
    <td>67.03</td>
    <td>70.36</td>
    <td>54.88</td>
    <td>64.86</td>
    <td>63.96</td>
    <td>67.08</td>
    <td>13.54</td>
    <td>7.06</td>
    <td>-20.49</td>
    <td>41.30</td>
    <td>54.19</td>
  </tr>
</table>

Reviewer paper summaries: The authors of this paper propose an automatic data selection method, MIG, for optimizing instruction fine-tuning datasets. MIG models the semantic relationships of data points through label graphs and introduces information dissemination to measure data quality and diversity. The greedy strategy is used for data selection, and the samples that can maximize the information gain are selected for each iteration to construct an efficient training subset. Experiments were conducted on Tulu3, OpenHermes2.5, and Xsota datasets, and the results showed that MIG outperformed the baseline approach on multiple tasks and remained competitive with reduced data volumes.
Summary of strengths: Compared with the traditional heuristic methods based on embedded space, MIG can consider data diversity more comprehensively, reduce redundancy, and improve the representativeness of datasets. Experiments were conducted across multiple large-scale data pools and validated on different language models, and the results showed that the small amount of data selected by MIG could achieve fine-tuning performance similar to that of the full dataset. In addition, MIG computational overhead is low, which avoids the high embedding similarity calculation in some baseline methods (e.g., QDIT, DEITA), and improves the scalability of data selection.
Summary of weaknesses: 1.There is a lack of theoretical support for the measurement of information gain
The information gain proposed in this paper is based on the weighted information propagation of label graphs, but it lacks rigorous mathematical derivation. The information measurement method is empirical, and it is not theoretically compared with the existing information theory methods such as entropy and mutual information, and it is difficult to evaluate the rigor of the MIG method.
    2. Greedy strategies may lead to suboptimal solutions and lack of optimality analysis
MIG uses a greedy strategy for data selection, each time selecting the sample that contributes the most to the current state. However, greedy methods are difficult to guarantee globally optimal and can lead to limited coverage of the selected subset of data. The paper does not provide convergence analysis, nor does it compare with other optimization methods, which cannot justify the MIG sampling strategy.
    3. Insufficient experimental evaluation and questionable generalization ability
This paper mainly evaluates MIG methods based on benchmarks such as AlpacaEval and WildBench, but does not test the performance of MIGs on zero-shot or few-shot learning tasks, nor does it explore the generalization ability of MIGs on cross-domain tasks. In addition, the experiment did not analyze the sensitivity of the MIG method to changes in the quality of the data pool, i.e., whether the performance in the low-quality or high-noise data pool remained stable.
Comments, suggestions and typos: There are a few issues in the paper that could be improved:
1.In the definition of information gain, the explanation lacks mathematical formalism. While the paper introduces weighted information propagation, it does not provide a precise formulation or compare it with standard measures like entropy or mutual information. Clarifying this would improve the theoretical rigor.
2.The description of the greedy sampling strategy does not address its potential limitations. Since greedy methods do not guarantee global optimality, a brief discussion on its trade-offs or a comparison with alternative selection strategies would be beneficial.
3.Some sentences in the results section are overly complex, making them harder to follow. Simplifying these statements or breaking them into shorter sentences could improve readability.
Overall assessment scores: 3
Metareview score: 3
Metareview: This paper proposes a data selection method, MIG (Maximizing Information Gain), for instruction fine-tuning. The method constructs a label graph where nodes represent semantic tags derived from instruction labels, and edges represent semantic similarity. Information gain is computed by propagating information over this graph, and a greedy strategy is used to select data points that maximize the global information gain. The method is applied to several instruction tuning datasets, and experiments show competitive or improved results compared to strong baselines, using significantly less data.
Summary of reasons to publish: - The paper proposes a data selection method based on semantic label graphs and information propagation, which differs from embedding-based heuristics used in prior work.
- Experiments on Tulu v2.5, OpenHermes-2.5, and X-SOTA using Mistral-7B and LLaMA2-7B show that MIG achieves competitive or better performance with significantly less data.
- Evaluation covers multiple capabilities (reasoning, factual knowledge, coding, math) via AlpacaEval and WildBench. 
- The authors addressed key concerns in the rebuttal, including the intuition behind Equations (6) and (7), and the motivation for the greedy strategy.