SMART: Submodular Data Mixture Strategy for Instruction Tuning (2024)

Abstract: Instruction Tuning involves finetuning a language model on a collection of instructionformatted datasets in order to enhance the generalizability of the model to unseen tasks. Studies have shown the importance of balancing different task proportions during finetuning, but finding the right balance remains challenging. Unfortunately, there’s currently no systematic method beyond manual tuning or relying on practitioners’ intuition. In this paper, we introduce SMART (Submodular data Mixture strAtegy for instRuction Tuning) — a novel data mixture strategy which makes use of a submodular function to assign importance scores to tasks which are then used to determine the mixture weights. Given a fine-tuning budget, SMART redistributes the budget among tasks and selects non-redundant samples from each task. Experimental results demonstrate that SMART significantly outperforms traditional methods such as examples proportional mixing and equal mixing. Furthermore, SMART facilitates the creation of data mixtures based on a few representative subsets of tasks alone and through task pruning analysis, we reveal that in a limited budget setting, allocating budget among a subset of representative tasks yields superior performance compared to distributing the budget among all tasks. The code for reproducing our results is open-sourced at https://github.com/kowndinyarenduchintala/SMART.

heading: 1 Introduction
text: “Your ability to juggle many tasks will take you far.”
One of the main goals of artificial intelligence (AI) research is to build machines that can communicate (Turing, 1950), and an essential part of communication is to understand and follow instructions. Large Language Models (LLMs), which are pre-trained over massive text corpora on next-tokenprediction objective, can perform a wide range of
NLP tasks via “prompting” (Brown et al., 2020; Kojima et al., 2022; Almazrouei et al., 2023; Liu et al., 2023; Touvron et al., 2023).
Instruction Tuning (Wei et al., 2021; Sanh et al., 2021; Chung et al., 2022) is an approach that further enhances the instruction-following ability and generalizability of pre-trained LLMs to unseen tasks. It involves fine-tuning an LLM on a collection of instruction-formatted instances (encompassing multiple tasks) - each consisting of an instruction (or task description), an optional input, the corresponding output (the ground truth) and optionally a few demonstrations/examples. It is a special case of multitask learning where the LLM is finetuned on a collection of instruction-formatted multitask datasets (Chung et al., 2022). Finetuning on multiple tasks simultaneously, allows the model to share and transfer information across tasks, resulting in a better common internal representation that is preferred by all tasks while suppressing taskdependent noise (Caruana, 1997). Consequently, the model learns to generalize to unseen tasks by discerning helpful cues from both implicitly and explicitly related tasks that it has previously seen.
The performance enhancement from instruction tuning is heavily contingent on data quality, data quantity, and task composition (Wang et al., 2023b). Studies by Iyer et al. (2022) and Longpre et al. (2023) have shown that while scaling the number of tasks is important, the relative proportion of various tasks (mixture weighting) merits as much attention for optimal instruction tuning. Intuitively, we want the model to see enough data for a given task that it can perform well on it, but not to see so much data that it memorizes the training set (Raffel et al., 2020). Iyer et al. (2022) performed manual tuning of various benchmark proportions and decided on a final mixture, whereas Longpre et al. (2023) studied the impact of removing each benchmark from the finetuning mixture and relied on their practioners’ intuition from there on, to decide
12916
on the exact proportions of benchmarks. In this work, we would like to explore a more systematic approach to mixture weighting. Specifically, we are motivated by the fact that in a large multitask dataset like FLAN 2022 (Longpre et al., 2023), which has 1840 tasks, there will likely be many similar tasks leading to redundancies and not all of them may require sampling in equal proportions. For instance, there might be many tasks of the type Natural Language Inference (NLI), and it might be enough to sample relatively more instances from a few representative NLI tasks and less from the others. Furthermore, which samples we select from each task is also crucial because the samples should faithfully represent the task at hand. A random subset may fail to do this as it can miss out on essential corner cases.
With this context, we focus on the following two fundamental research questions (RQs) that form the basis for our subsequent inquiry:
• (RQ1) Given a huge multitask instructiontuning dataset and a limited fine-tuning budget which is defined by the total number of (prompt, response) instances that can be used for fine-tuning, how do we divide this budget among thousands of tasks present in the dataset? i.e., how many instances to sample from each task? and which instances to sample from each task? • (RQ2) Can we go a step further and strategically prune some tasks altogether and only fine-tune on a small subset of representative tasks without hurting the performance? If yes, what is the nature of this subset?
To the best of our knowledge, there’s currently no principled approach to determining task compositions for instruction tuning, other than manual tuning and/or practioners’ intuition.
As a first step towards addressing both of the above RQs, we first define a common subset selection problem (more formally stated in Section 3) as follows - Given a huge collection of M instructionformatted task datasets, a task budget M ′ ≤M and a total budget (N ′) of (prompt, response) pairs, which M ′ tasks to select? and how many instances to select from each of these M ′ tasks and which instances to select? Note that RQ1 is an instance of this problem where M ′ = M .
Constrained Submodular Maximization (Section 2) proves to be a good model for discovering representative subsets (or coresets) of a massive training dataset (or ground set) that acts as
surrogate (i.e., achieves similar performance) and are much better than uniformly-at-random subsets. Intuitively, this is because submodular functions model information in subsets, and hence maximizing a submodular function subject to a constraint yields non-redundant subsets of the ground set. An essential feature of this model is that it returns weighted subsets, i.e., each sample in the coreset comes with an associated score, which indicates how important the sample is.
Inspired by submodular functions, we propose our solution (Section 3) to the above subset selection problem for instruction tuning that works in two stages. In the first stage, we select a weighted subset of tasks from the full dataset where the weights will determine how many samples to select from each task. In the next stage, we select samples from each task based on the assigned task budgets. Note that the submodular functions used in each stage are not necessarily identical (Section 4.8).
The main contributions of our work can be summarized as follows:
• We introduce SMART — a novel data mixture strategy for instruction tuning that models the data mixture problem (Section 3) as a sequence of two cardinality-constrained submodular maximization problems and offer empirical evidence that it outperforms both examples proportional and equal mixing baselines (Section 4) as well as the mixture weights proposed by Longpre et al. (2023). • Existing works like Longpre et al. (2023) have reported a continuous increase in performance upon increasing the number of tasks (though the gains themselves may be diminishing). However, we posit that this depends on the order in which new tasks are incorporated and show empirically that in the case of SMART mixtures, a performance peak is observed with an initial addition of few representative tasks and upon adding more and more tasks, the performance is not sustained (Section 4.6). • We find that the nature of instances that should be selected in each task (i.e, whether a representative or diverse subset) also depends on the total task budget, M ′ (Section 4.8). For higher M ′s, each task on average gets the relatively low budget and selecting representative samples is more important; however for lower M ′s, when there is sufficient enough budget for each task, the need for diversity dominates that of representation.

heading: 2 Background: Submodularity
text: Notations. Let f : 2V → R be a set function that assigns a value to every subset of the ground set V . We use the notation f(v|X) as a shorthand for f(X ∪ {v})− f(X) i.e., the incremental value gain of v in the context of X .
Definition 1. (Submodular Function) A given set function f : 2V → R is submodular if for all X,Y ⊆ V , where X ⊆ Y and for all v /∈ Y , the following inequality holds true:
f(v|X) ≥ f(v|Y ) (diminishing gains property)
Intuitively, the definition states that — Adding an element to a smaller set X yields more value gain than adding the same element to a superset Y of X . Table 1 contains examples of three submodular functions — Facility Location (models representation), Log Determinant (models diversity), and Graph Cut (models a trade-off between representation and diversity controlled by the parameter λ).
Definition 2. (Cardinality Constrained Submodular Maximization) Given a submodular function f : 2V → R defined over the subsets of the ground set V , the constrained submodular maximization problem involves finding S∗ such that
S∗ = argmax X⊆V
|X|≤N ′ f(X)
The above cardinality-constrained submodular maximization problem is NP-complete (Feige, 1998). However, if f is monotone submodular (i.e., f(X) ≤ f(Y ) whenever X ⊆ Y ), Nemhauser et al., 1978; Fisher et al., 1978 show that a simple greedy algorithm described in Algorithm 1 can be
used to find an approximate solution Sgreedy, with a guarantee that f(Sgreedy) ≥ (1− 1/e)f(S∗).1
Algorithm 1 The Naïve Greedy Input: Ground Set (V), Budget (N ′) X0 ← ∅; S ← [ ]; Gains← [ ]; for i = 0 to (N ′ − 1) do e∗ = argmax
v∈V\Xi f(v|Xi);
gi+1 = f(e ∗|Xi); Xi+1 = Xi ∪ {e∗}; S.append(e∗); Gains.append(gi+1);
end for return S, Gains;
Remark. The algorithm produces a weighted subset where the value gains themselves are the weights (i.e., max
v∈V\Xi f(v|Xi) is the weight of
argmax v∈V\Xi
f(v|Xi)).
Algorithm 1 (also known as Naïve Greedy) requires O(N ′.|V|) function evaluations which is costly in practice. Accelerated Greedy (Minoux, 2005), also known as Lazy Greedy, can be used instead, which leverages submodularity and offers a more efficient heap-based implementation of the same algorithm.

heading: 3 Approach
text: Inspired by submodularity (Section 2), in this section, we introduce a novel data mixture strategy, SMART, as a technique to solve the following subset selection problem introduced in Section 1:
Consider a collection D = {T1, . . . , TM} of M instruction-formatted task datasets where each Ti = {(promptij , responseij)} NTi j=1 consists of NTi (prompt, response) pairs. Let ∑M
i=1NTi = N . Given an M ′ ≤M and an N ′ ≤ N , how do we select a subset of tasksD′ = {T ′1 , . . . , T ′M ′}(where D′ ⊆ D), and subsequently S = {S1, . . . ,SM ′} (where Sj ⊆ T ′j ) and ∑M ′ j=1 |Sj | = N ′) such that efficiently fine-tuning on the subset S alone is (nearly) as effective as fine-tuning on the entire collection D?
SMART models the above problem as a sequence of two cardinality-constrained submodular
1Assuming P ̸= NP , this is the best approximation ratio that can be achieved by any polynomial time algorithm.
maximization problems. The first one is to select a weighted subset of M ′ tasks and the second one is to select instances - a total of N ′ instances from these tasks. The weights obtained in the first stage will be used to determine how many instances we sample from each task.

heading: 3.1 The Algorithm
text: We now give a detailed description of the two stages of SMART (summarized in Algorithm 2):

heading: 3.1.1 Stage-1: Weighted Task Subset Selection
text: In this first stage, given the instruction-tuning dataset D = {T1, . . . , TM}, our goal is to find D′ = {T ′1 , . . . , T ′M ′} where D′ ⊆ D, along with the instance budgets, {N ′1, . . . , N ′M ′}, such that∑M ′
j=1 |N ′j | = N ′. If f1 is the submodular function that we use in
this stage, D′ is given by:
D′ = argmax X⊆D
|X|≤M ′ f1(X)
To find the instance budgets (N ′js), we use the second-order Taylor-softmax operation (De Brebisson and Vincent, 2015) on the value gains obtained from the greedy algorithm, to compute a probability distribution which determines the probability with which instances will be sampled from a given task i.e., if {g1, . . . gM ′} are the value gains returned by the greedy algorithm, corresponding to the tasks {T ′1 , . . . , T ′M ′}, the instance budgets are given by
N ′j = (1 + gj + 0.5g 2 j )∑M ′
k=1(1 + gk + 0.5g 2 k) ×N ′

heading: 3.1.2 Stage-2: Instance Subset Selection
text: In this stage, given the subset of tasks, {T ′1 , . . . , T ′M ′}, and the instance budgets {N ′1, . . . N ′M ′} from the first stage, the goal is to actually select those many samples from each task. If f2 is the submodular function used, the final subset S is given by
S = M ′⋃
j=1 argmax Xj⊆T ′j |Xj |≤N ′ j f2(Xj)
Algorithm 2 The SMART Data Mixture Strategy Input: Task datasets D = {T1, . . . , TM}, Task Budget (M ′), Instance Budget (N ′), f1, f2 # Each Ti = {(promptij , responseij)} NTi j=1
encoder ← SentenceEncoder() prompt_embeddings← dict() task_embeddings← dict() for i = 1 to M do V← [ ] for j = 1 to NTi do v← encoder.encode(promptij) V.append(v)
end for prompt_embeddings[Ti] = V
task_embeddings[Ti] = 1
NTi NTi−1∑ j=0 V[j]
end for Ktask = cos_sim(task_embeddings) D′, gains = Greedy(f1,D,Ktask,M ′) # D′ = {T ′1 , . . . T ′M ′} probs← Taylor_Softmax(gains) S ′ ← [ ] for j = 1 to M ′ do N ′j = probs[j]×N ′ KT ′j = cos_sim(prompt_embeddings[T ′ j ])
ST ′j , gains = Greedy(f2, T ′ j ,KT ′j , N ′ j)
S ′.append(ST ′j ) end for S = M ′⋃ j=1 S ′[j] return S

heading: 3.2 Obtaining Similarity Measures
text: The three submodular functions listed in Table 1 require computation of similarity measures (sij) between items in the ground set. So, all prompts in the dataset are first encoded using a sentence encoder. For instance subset selection (i.e., Stage-2), cosine similarity between prompt embeddings is used as the similarity measure and for weighted task subset selection (i.e., Stage-1), cosine similarity between task embeddings is computed, where the task embeddings are computed as the average prompt embeddings, following Vu et al. (2020). Although there are other sophisticated methods (Achille et al., 2019; Zhou et al., 2022; Xi et al., 2023; Vu et al., 2021) for obtaining task embeddings, most of them depend on the LLM at hand.

heading: 3.3 Choosing f1 and f2
text: Each submodular function in Table 1 captures a different property: facility-location emphasizes representation, graph-cut balances representation and diversity, and log-determinant prioritizes diversity. We treat f1 and f2 as hyperparameters in our grid search in Section 4.8, exploring three options for each function. Determining the optimal f1 and f2 is part of our research question (RQ2). We discuss the findings of grid search and their qualitative implications for instruction tuning in Section 4.10.

heading: 4 Experiments

heading: 4.1 Finetuning Data
text: In all the experiments, the underlying ground set (D) is FLAN 2022 (Longpre et al., 2023; Chung et al., 2022). The collection consists of the following five sub mixtures adding up to a total of 1840 tasks and 17,591,640 (∼17.5M) instructionformatted (prompt, response) pairs:
• FLAN 2021 (Wei et al., 2021) • T0 (Sanh et al., 2021) • NIV2 (Wang et al., 2022) • CoT (several chain-of-thought datasets) • Dialog (a few dialog datasets)
Each of the tasks comes in a variety of templates - zeroshot with and without answer options, fewshot with and without answer options.
SMART Data Mixture Creation We encode prompts in the collection with GTE-large (Li et al., 2023b), a light-weight (340M parameters) BERTbased effective (Muennighoff et al., 2022) sentence encoder for semantic textual similarity. Task embeddings are obtained by averaging the corresponding prompt embeddings (Section 3.2). We use SUBMODLIB (Kaushal et al., 2022), which has the necessary algorithms implemented, to obtain the weighted task subsets (Section 3.1.1) and the instances (Section 3.1.2). We distribute the instance budgets equally among task templates based on findings by Longpre et al. (2023) which showed that an equal number of zero-shot and few-shot templates yield the best performance on held-out tasks.

heading: 4.2 Finetuning Procedure
text: We evaluate SMART on three 7B parameter LLMs: Llama-2 (Touvron et al., 2023), Falcon (Almazrouei et al., 2023), and Mistral (Jiang et al., 2023). We fine-tune each model for 1 epoch on a
given data mixture with a learning rate of 2e − 5 for Llama-2 and Falcon, and 5e − 6 for Mistral. A batch size of 64, weight decay of 0.1, and cosine learning rate decay with a linear warmup for the initial 1% training steps are employed. The experiments all ran on 8 NVIDIA A100-SXM480GB GPUs, utilizing Flash Attention (Dao, 2023) for memory efficiency and speeding up finetuning. Our code is open-sourced here.

heading: 4.3 Baselines
text: We mainly compare SMART with two baseline mixture strategies: Examples Proportional Mixture, Equal Mixture (Raffel et al., 2020).
Examples Proportional Mixture (Baseline-1) Instances are sampled in proportion to the size of each task’s dataset. This is equivalent to randomly sampling from the combined datasets.
Equal Mixture (Baseline-2) Instances are sampled from each task with equal probability i.e., by dividing the total budget equally among tasks, and then uniformly sampling from each task.
We also compare SMART with mixture weights used by Longpre et al. (2023) but since their mixture weights apply to the five sub mixtures listed in Section 4.1 rather than individual tasks, we analyse this separately in Section 4.7.

heading: 4.4 Evaluation Protocol
text: We evaluate the fine-tuned models on two benchmark datasets: MMLU (Hendrycks et al., 2020) with 57 tasks assessing world knowledge and problem-solving, and BBH (Suzgun et al., 2022) with 23 challenging tasks from Big-Bench (Srivastava et al., 2022). MMLU covers STEM, Humanities, Social Sciences, and Other (business, medical, and misc.) categories, while BBH includes both NLP and Algorithmic tasks2. Evaluation involves prompting the LLM directly and using Exact Match as the scoring metric. Responses are generated using the greedy decoding approach and they undergo basic post-processing steps (removing punctuation and lower-casing) before calculating exact match. For baseline data mixtures, 3 mixtures with different random seeds are created and the mean exact match of the 3 fine-tuning runs is reported.
2The Algorithmic subcategory is so named because these tasks (e.g., 2-digit arithmetic) do not require an LLM to be solved.

heading: 4.5 Addressing RQ1 (M ′ = M )
text: RQ1 is an instance of the subset selection problem defined in Section 3, where M ′ = M , allowing all tasks for finetuning; however we still have a constraint (N ′) on total number of (prompt, response) pairs. We use Graph-Cut and Facility Location functions (Table 1) in Stage1 (Section 3.1.1) and Stage-2 (Section 3.1.2) of SMART respectively. This choice of f1 and f2 is determined via grid search discussed in Section 4.8. Table 2 contains comparison of SMART mixtures with baseline mixtures, on MMLU and BBH benchmarks, upon instruction fine-tuning Llama-2-7b on data mixtures generated by varying N ′ in {25000, 50000, 100000, 200000, 400000}. SMART data mixtures consistently perform better than both examples proportional mixtures and equal mixtures baseline.

heading: 4.6 Addressing RQ2 (M ′ < M )
text: RQ2 is an instance of the subset selection problem defined in Section 3, where M ′ < M i.e., studying the effect of pruning some tasks altogether, on both SMART and the baseline data mixtures. We finetune Llama-2-7b on both the baseline and SMART data mixtures by varying the number of tasks (M ′) in {8, 16, 32, 64, 128, 256, 512, 1024, 1840} and the total number of instances (N ′) in {25000, 50000, 100000, 200000, 400000}. Figure 1 depicts the task-scaling plots for each N ′. Baseline data mixtures’ performance steadily improves upon increasing the number of tasks, even
though the gains are diminishing after a point. In contrast, SMART data mixtures yield optimal performance at an in-between point and this performance does not seem to sustain upon adding more and more tasks, suggesting that rather than increasing the tasks, focusing on representative tasks and sampling more instances from these might be more beneficial in low-budget scenarios. Even with an ample budget, scaling tasks should be done judiciously, as close to 97% of performance achievable by using the entire FLAN collection can be attained with just a subset of 16 representative tasks alone with N ′ = 200000.

heading: 4.7 Comparison with FLANv2 Mix
text: We now compare SMART with FLANv2-mix i.e., by using the mixture weights suggested by Longpre et al. (2023) where the weights 40%, 32%, 20%, 5% and 3% are assigned to the 5 sub mixtures of FLAN 2022 — FLANv1, T0, NIV2, CoT
and Dialog respectively and instances are randomly sampled from each sub mixture. Since this method only prescribes weights on sub mixtures and not on individual tasks, we only consider the case where M ′ = M . Table 3 contains comparison of SMART vs FLANv2-mix for Llama-2-7b. FLANv2-mix seems to perform better than EPM (Baseline-1) and EM (Baseline-2) in some cases although SMART almost always outperforms FLANv2-mix.

heading: 4.8 Ablation Study: Submodular Function
text: Both Stage-1 and Stage-2 of SMART require us to choose submodular functions f1 and f2 - which are best treated as hyperparameters because of uncertainty with respect to which functions are best suited for instruction tuning (Section 3.3). We conduct a grid search on f1 and f2 using three functions from Table 1, by setting M ′ = M and varying
N ′ in {25000, 50000, 100000, 200000}. In this work, we use λ = 0.4 for Graph-Cut. The grid search (summarized in Table 4) suggests that Graph Cut is optimal for task subset selection, while Facility Location is best for instance selection when M ′ = M . The task scaling curves for each combination of f1 and f2 (for different N ′s) are present in Appendix B. However, in this section, we highlight the case where f1 is Graph Cut and f2 is varied. We find that optimal f2 in this case also depends on M ′. For instance, in Figure 2 where N ′ = 200000, Facility Location performs the best at higher M ′, highlighting the importance of representation, while Graph Cut and Log Determinant show better performance at lower M ′, highlighting the importance of diversity. We hypothesize this is because — for higher M ′s, each task on average gets a relatively low budget and the need for representation dominates the need for diversity; however when there is sufficient enough budget for each task, i.e., for lower M ′s, the need for diversity takes over.

heading: 4.9 Ablation Study: LLM
text: We also test the effectiveness of SMART strategy on two other LLMs - Falcon-7B and Mistral7B. Table 5 presents a comparison of SMART mixture with the baseline mixtures for Falcon and Mistral when M ′ = M . For Falcon, we only report MMLU-zero-shot since Falcon gets an exact match of 0.0 for BBH-zero-shot even after finetuning. The task-scaling curves for these models are present in Appendix C.

heading: 4.10 Discussion
text: In Section 4.8, we saw that Graph Cut proves to be the most effective for selecting the weighted task subsets. Figures 6, 7, 8 of Appendix D contain t-SNE visualizations for task subsets selected by the three submodular functions listed in Table 1. Facility Location being a sum-max formulation, a single point is sufficient to represent a cluster and hence it gives more weightage to cluster centers and very less weightage to others. As a result, the submodular gains for first few selected points are very high and then the gains quickly become very small for other points in case of facility location. Log Determinant on the other hand predominantly selects diverse points not taking representation into account. Graph Cut, which models both, hence performs better than both Facility Location and Log Determinant for selecting tasks.
Further, in Figure 1 of Section 4.6 we saw that, at around 16 tasks, there is a peak in performance,
after which there is a slight decline. To facilitate more investigation into what tasks are assigned more weightage, Table 6 and Table 7 list down the top-128 and last-128 tasks in the submodular ordering obtained using graph cut. While the former set contains more traditional NLP tasks like Natural Language Inference, Next Sentence Prediction, Question Answering, Summarization, etc ., the latter set mostly consists of tasks like Program Execution.

heading: 5 Related Work

heading: 5.1 Data for Instruction Tuning
text: Following Wang et al. (2023b), we summarize the related work in three categories — data quality, data quantity, and task composition.
Data Quantity Research diverges on scaling instruction data quantity, with some advocating for limited data (Zhou et al., 2023; Chen et al., 2023a) to expose pretraining knowledge, while others argue for scaling up (Wei et al., 2021; Sanh et al., 2021). According to Ji et al. (2023); Dong et al. (2023); Yuan et al. (2023); Song et al. (2023), the impact of scaling varies across tasks and model abilities. AlShikh et al. (2023) also introduce a metric for instruction following ability, and suggest an early stopping criterion for instruction-tuning.
Data Quality High-quality data is crucial in instruction tuning (Chia et al., 2023; Ding et al., 2023; Zhou et al., 2023). Wang et al. (2023a) use perplexity to select suitable instructions generated by models, while Li et al. (2023a) employ the language model itself to augment and curate high-quality training examples to improve its own performance. Cao et al. (2023) propose InstructionMining, utilizing natural language indicators to predict inference loss as a proxy for data quality without human intervention and select the best subset based on this. Chen et al. (2023b) introduce AlpaGasus, which uses a strong LLM to select high-quality subsets. Lu et al. (2023) and Madaan et al. (2024) also leverage the power of fine-tuned LLM itself to evaluate the quality of instructions. Attendu and Corbeil (2023) employs dynamic data subset selection by filtering out unimportant samples during finetuning, based on an extended EL2N metric (Paul et al., 2021; Fayyaz et al., 2022). Taori et al. (2023) propose #InsTag to assess instruction diversity in SFT datasets using ChatGPT. Additionally, Wan et al. (2023) propose Explore-Instruct, utilizing LLMs to
actively explore domain-specific spaces and gather diverse instruction-tuning data. Wu et al. (2023) select new data points that are distinct from existing ones in the model embedding space, augmenting the training dataset iteratively to enhance diversity within subsets.
Task Composition Many previous works show the benefit of scaling the number of tasks (Wei et al., 2021; Chung et al., 2022; Wang et al., 2022; Sanh et al., 2021). However, works like Iyer et al. (2022); Longpre et al. (2023) have also acknowledged that task balancing is also very important for effective instruction tuning. Dong et al. (2023) explore data composition across GSM8k, Code Alpaca, and ShareGPT datasets, finding differential impacts of data scaling on performance across abilities and propose a Dual-stage mixed fine-tuning strategy as a promising solution to activate multiple abilities efficiently. Ivison et al. (2022) identifies relevant multitask subsets based on the similarity between the pre-trained model’s representations, using a small amount of target task data. Yin et al. (2023) uses instruction representations for task selection, acting as a replay strategy to mitigate catastrophic forgetting and improve generalization in continual learning. Yue et al. (2023) construct math generalist models via instruction tuning on hybrid of chain-of-thought and program-of-thought rationales in math. Lou et al. (2023); Zhang et al. (2023) provide a survey of instruction tuning in general and Wang et al. (2023b) provide a detailed survey of data management for instruction tuning.

heading: 5.2 Submodularity for Subset Selection
text: Submodularity (Fujishige, 2005) has a long history in combinatorial optimization, game theory, economics etc (Edmonds, 1970; Lovász, 1983; Carter, 2001; Topkis, 1998). It has recently gained traction in machine learning where it has been used for data subset selection for machine translation (Kirchhoff and Bilmes, 2014), speech recognition (Wei et al., 2014; Mittal et al., 2022; Kothawade et al., 2023), efficient pre-training of language models (Renduchintala et al., 2023), active learning (Wei et al., 2015; Kothawade et al., 2021), hyperparameter tuning (Killamsetty et al., 2022), domain adaptation (Karanam et al., 2022), computer vision (Kaushal et al., 2019), continual learning (Tiwari et al., 2022) etc. For a more detailed review of submodularity and its applications, please refer to the survey by Bilmes (2022).

heading: 6 Conclusion & Future Work
text: In this paper, we introduced SMART — a novel data mixture strategy for instruction tuning that utilizes a submodular function to assign importance scores to tasks, determine the mixture weights, and also select non-redundant samples from each task. Further, we also reveal that in a low-budget setting, splitting the budget among a small subset of representative tasks yields superior performance when compared to dividing it among all tasks, which suggests that task scaling should be done more judiciously. Future work could explore making this method more model-specific (e.g., modify task embedding computation) and also possibly modify the approach for targeted instruction-data selection for creating expert LLMs that specialize in specific skills such as math, code etc.

heading: 7 Limitations
text: While the approach has its own advantages of computing the data mixture only once for a given dataset and using it for as many LLMs as one may wish, the approach might benefit from taking inputs from the language model as well and perform model specific instruction tuning. Secondly, SMART shows that there is an optimal number of tasks at which there is peak in performance observed. However, it doesn’t say anything on how to find the optimal point as it may depend on language model, the total budget and most importantly the underlying dataset.

heading: 8 Ethical Considerations
text: While instruction tuning leverages pretrained language models and encompasses similar considerations, we anticipate that this approach will predominantly yield positive outcomes. It offers an enhanced methodology for task balancing, potentially allowing for more cost-effective fine-tuning of large language models compared to conventional data mixture strategies.

heading: Acknowledgements
text: The authors acknowledge the use of ChatGPT3 for solely paraphrasing and summarizing some parts of the paper and declare that none of the generated content is presented in the paper without rigorous manual checking.
3https://chat.openai.com/

heading: A Code and Data
text: Our code we used for instruction tuning and for creating data mixtures is opensourced at https://github.com/kowndinyarenduchintala/SMART. The code development utilized open-source tools, primarily relying on the HuggingFace library for model training with PyTorch as the underlying framework. Both PyTorch and HuggingFace are licensed under permissive licenses, with PyTorch under the BSD license and HuggingFace under the Apache 2.0 license. Additionally, submodular optimization was performed using SUBMODLIB, which is an openly accessible library on GitHub at https: //github.com/decile-team/submodlib under the MIT license.

heading: B Task Scaling Curves: Varying f1 and f2
text: Figure 3 consists of task scaling curves for the 9 possible combinations (Section 4.8) of f1 and f2.

heading: C Task Scaling Curves: Mistral, Falcon
text: Figure 4 and Figure 5 consist of task scaling curves for Mistral-7B and Falcon-7B respectively (Section 4.9).

heading: 8 16 32 64 128 256 512 1024 1840

heading: 8 16 32 64 128 256 512 1024 1840

heading: 8 16 32 64 128 256 512 1024 1840

heading: 8 16 32 64 128 256 512 1024 1840
text: D Visualization of Task Subsets
As pointed out in Section 2, the three submodular functions in Table 1 are different. Facility Location predominantly models representation; Graph Cut models a trade-off between representation and diversity; Log Determinant predominantly models diversity. To better visualize this, we present
t-SNE plots of 1840 tasks present in the FLAN 2022 collection (Longpre et al., 2023) and highlight the tasks selected by these functions for different values of M ′. Figure 6, Figure 7 and Figure 8 respectively contains the visualizations for Facility Location, Graph Cut and Log Determinant.

heading: Future Work
text: Future work could explore making this method more model-specific (e. g. , modify task embedding computation) and also possibly modify the approach for targeted instruction-data selection for creating expert LLMs that specialize in specific skills such as math, code etc