Title: Skill-it! A data-driven skills framework for understanding and training language models
Venue: NeurIPS 2023
Accepted: yes
Keywords: language models

data selection
Abstract: The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop a new framework based on a simple hypothesis: just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data. If such an order exists, it can be utilized for improved understanding of LMs and for data-efficient training. Using this intuition, our framework formalizes the notion of a skill and of an ordered set of skills in terms of the associated data. First, using both synthetic and real data, we demonstrate that these ordered skill sets exist, and that their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills. Second, using our proposed framework, we introduce an online data sampling algorithm, Skill-It, over mixtures of skills for both continual pre-training and fine-tuning regimes, where the objective is to efficiently learn multiple skills in the former and an individual skill in the latter. On the LEGO synthetic in the continual pre-training setting, Skill-It obtains 37.5 points higher accuracy than random sampling. On the Natural Instructions dataset in the fine-tuning setting, Skill-It reduces the validation loss on the target skill by 13.6% versus training on data associated with the target skill itself. 
We apply our skills framework on the RedPajama dataset to continually pre-train a 3B-parameter LM, achieving higher accuracy on the LM Evaluation Harness with 1B tokens than the baseline approach of sampling uniformly over data sources with 3B tokens.

Skill-it! A data-driven skills framework for understanding and training language models

Abstract

The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop a new framework based on a simple hypothesis: just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data. If such an order exists, it can be utilized for improved understanding of LMs and for data-efficient training. Using this intuition, our framework formalizes the notion of a skill and of an ordered set of skills in terms of the associated data. First, using both synthetic and real data, we demonstrate that these ordered skill sets exist, and that their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills. Second, using our proposed framework, we introduce an online data sampling algorithm, SKILL-IT, over mixtures of skills for both continual pre-training and fine-tuning regimes, where the objective is to efficiently learn multiple skills in the former and an individual skill in the latter. On the LEGO synthetic in the continual pre-training setting, SKILL-IT obtains 37 . 5 points higher accuracy than random sampling. On the Natural Instructions dataset in the fine-tuning setting, SKILL-IT reduces the validation loss on the target skill by 13 . 6 %versus training on data associated with the target skill itself. We apply our skills framework on the RedPajama dataset to continually pre-train a 3B-parameter LM, achieving higher accuracy on the LM Evaluation Harness with 1B tokens than the baseline approach of sampling uniformly over data sources with 3B tokens.

1 Introduction

Large language models (LMs) exhibit remarkable capabilities, including producing creative content [55], writing source code [9], and chatting with users [8]. A key ingredient in enabling models to perform such tasks is the data on which the models are trained [18, 20, 59]. A natural way to unlock particular capabilities is to improve this training data. However, it is unclear how to select data from a large corpus for these capabilities given a fixed budget of training tokens, as data selection methods for current state-of-the-art LMs mostly rely on heuristics for filtering and mixing together different datasets [33, 59]. We lack a formal framework for capturing how data influences the model's capabilities and how to utilize this data effectively for improving LM performance.

English QA

1

Spanish QG

ValidationLoss

D

P4

1

2

Random

P3

SpanishQA

EnglishQG

sampling

P4

2

3

P2

Skill-It

P1

4

SpanishQG

Training

Data

Ordered skill set

1234

1234

To develop such a framework, we take inspiration from how humans acquire knowledge. A classic idea in education literature is the concept of skills that form a learning hierarchy [65]. For example, one study found that students learned mathematical and scientific skills most quickly when these skills were presented in a particular order [12]. We seek to understand the extent that similar skill-based orderings characterize LM training. Such orderings, if they exist, may provide a better understanding of LMs as well as a mechanism for data-efficient training. For instance, to train an LM for Spanish question generation, we wish to know if training first on related but simpler tasks, such as Spanish grammar and English question generation, helps.

Westudy if the idea of skill orderings can help us build a framework that relates data to LM training and behavior. This requires addressing two challenges revolving around the connection between skills and data. First, in order to show that there exist sets of skills that the LM learns most efficiently in some particular order, an operational definition of LM skill and skill ordering must be developed and validated on data . In initial experiments, we investigated if semantic groupings of data, such as metadata attributes or embedding clusters, were sufficient to represent a skill and characterize how models learn. For instance, we partitioned the Alpaca dataset [56] by instruction type-a technique used to capture dataset diversity [62]-but we found that sampling based on instruction types and random sampling resulted in similar model performance, suggesting that not just any existing notion of data groups can characterize skills.

Second, these definitions of skills must be used to construct sampling distributions to actually improve model training. To develop criteria for a data selection algorithm that learns skills efficiently, we identify challenges that naive selection approaches face. The standard approach of random uniform sampling over data fails to learn skills optimally due to not accounting for skill imbalance and ordering. Skills can be distributed unevenly in the data, with more complex skills being rare-for instance, Spanish and question generation (QG) are 5% and 4% of the Natural Instructions dataset [63], respectively, but Spanish QG is only 0 . 2% . Random sampling also provides no mechanism for taking into account a particular training order and dependency structure on skills. More sophisticated techniques like curriculum learning account for sample-level ordering, but not skills or their dependencies. Our goal framework must account for these issues of imbalance and ordering.

Skill-based framework We define a skill as a unit of behavior that a model can learn using an associated slice of data (Definition 1). An ordered skill set is a collection of skills with a directed skills graph that is neither complete nor empty, where an edge from a prerequisite skill to a skill exists if the amount of training it takes to learn the skill can be reduced if the prerequisite skill is also learned (Definition 2, Figure 1 left, center). We show that ordered skill sets exist in synthetic and real datasets using this operational definition. Interestingly, the existence of these ordered skill sets unveils that one can learn a skill quickly not by training solely on that skill, but on a mixture of that skill and prerequisite skills. For instance, in Figure 3 we observe that Spanish QG can be learned more efficiently when the model also learns English QG and Spanish-we can achieve 4% lower validation loss than training on only Spanish QG over a fixed budget of overall training steps.

Next, given an ordered skill set to train on, we use our framework to propose methods for how to select data so that the LM learn skills faster: skill-stratified sampling and an online generalization, SKILL-IT. Weaddress the issue of unevenly distributed skills in datasets by proposing skill-stratified sampling, a

simple approach that allows us to explicitly optimize for learning skills by uniformly sampling relevant skills (such as a target skill and its prerequisite skills in fine-tuning). Skill-stratified sampling uses the construction of the ordered skill set but is static, which does not incorporate the ordering as training proceeds and results in oversampling skills that may be already learned early on in training. We address this issue by proposing an online data selection algorithm, SKILL-IT, for selecting mixtures of training skills that allocates more weight towards learning skills that are not yet learned or towards prerequisite influential skills (Figure 1 right). SKILL-IT is derived from an online optimization problem over the training skills for minimizing loss on a set of evaluation skills given a fixed budget of data and the skills graph. SKILL-IT is inspired by online mirror descent and can be adapted for continual pre-training, fine-tuning, or out-of-domain evaluation depending on the relationship between the evaluation skill set and the training skill set.

Weevaluate SKILL-IT on synthetic and real datasets at two model scales, 125M and 1.3B parameters. For the continual pre-training setting, we show on the LEGO synthetic [72] that we obtain a 37 . 2 point improvement in accuracy over randomly selecting training data and a 9 . 7 point improvement over curriculum learning [4]. For the fine-tuning setting, we show that on the widely-used Natural Instructions dataset [41, 64], our algorithm over a mixture of skills is able to achieve up to 13.6% lower loss on that skill than solely training on that skill, given the same overall training budget. For the out-ofdomainsetting when our training skills do not align perfectly with evaluation skills, our algorithm is able to achieve the lowest loss on 11 out of 12 evaluation skills corresponding to task categories in the Natural Instructions test tasks dataset over random and skill-stratified sampling on the training data. We finally apply our framework to a case study on the recent RedPajama 1.2 trillion token dataset [57]. We use the data mixture produced by SKILL-IT to continually pre-train a 3B parameter model. We find that SKILLIT achieves higher accuracy with 1B tokens than uniform sampling over data sources with 3B tokens.

2 Related work

Anextended related work can be found in Appendix B. Existing work on data selection for LMs has generally ranged from more computationally expensive methods for dataset condensation on smaller datasets [47, 58, 48] to broader deduplication and filtering techniques for web-scale datasets [1, 33, 69]. Another way of improving model performance through choice of data is via curriculum learning [4], which also draws inspiration from how humans learn and arranges data in order from easiest to hardest over samples or groups [60]. In contrast to existing works in both curriculum learning and data selection, our work focuses on selecting data for learning an ordered set of skills more efficiently. How LMs learn is also a topic of growing interest; one framework posits that models learn over quanta, discrete units of computation [38], and another proposes that scaling laws for LMs can be understood in terms of learning combinations of skills [2]. Lastly, the notion of skill has been studied in education, ranging from classical research on learning hierarchies [66] to methods for decision-making over lesson sequences [49].

3 Skills framework

First, we propose definitions of skills and ordered skill sets in order to formalize our intuition around how models learn skills, and we demonstrate that not just any existing notion of data groups can characterize an ordered skill set in the dataset. Then, we demonstrate the existence of ordered skill sets on synthetic and real data, which show how viewing data through a skills-based framework can help with training and understanding model performance. Finally, we explore unsupervised skill recovery from data, finding that embedding-based approaches do not adequately recover synthetic skills.

3.1 Definitions

Wefirst present a definition of an individual skill. Let the input space of all possible text data be X , where x ∈X is an individual text sample that a next-token-prediction LM f ∈F : X →X is trained on. We quantify learning via a metric L : F×X→ R , which maps from a model and evaluation data to a scalar quantity. In our setup, we use the cross-entropy validation loss applied over next-token predictions as our metric L .

Definition 1 (Skill) A skill s is a unit of behavior with associated data X s ⊆X such that if f is trained onandataset D s ⊂X s , f has improved metric L afterwards on samples belonging to X s \D s onaverage.

This definition of a skill is flexible-it simply means that given a training dataset associated with the skill, a model f has an improved metric (e.g., decreasing validation loss) when evaluated on validation

Alpaca

0.0

0.5

1.0

1.5

2.0

Pile of Law

0.0

0.5

1.0

1.5

Natural Instructions

0.0

0.2

0.4

0.6

0.8

data associated with this skill. Under this definition, a skill could be a granular task, such as Spanish question generation for a subset of Wikipedia articles, or can be defined over a data source, such as nexttoken prediction of legal data from tax court rulings. However, our next definition, the ordered skill set, has a more specific construction and provides a framework for how models learn across dependent skills.

Definition 2 (Ordered skill set, skills graph) An ordered skill set for f is a collection of skills S = { s 1 ,...,s k } over which there is a directed skills graph G =( S ,E ) on the skill set that is neither complete or empty, where ( s i ,s j ) ∈ E if the amount of data needed to learn s j when uniformly sampling from D s i ∪D s j is no more than the amount of data needed when sampling only from D s j . We equate learning a skill s j to f attaining a certain value of L or lower on average over X s j \D s j .

This definition isolates complete and empty graphs as extrema that do not capture meaningful sets of skills. We discuss the three types of skill graphs-complete, empty, intermediate-and their implications for data selection. In particular, we discuss how several initial attempts of defining skills over datasets via semantic groupings resulted in the extrema cases (see Appendix D.2 for full results):

Thecompletegraphdemonstratesthat all skills influence each other. A random partition is an example of a skill set that yields a complete graph. This graph suggests that the best approach for learning any skill or set of skills is random sampling on the dataset. This is not a setting where we can gain much with skill-based sampling. For example, using instruction types as skills on the Alpaca dataset results in a nearly complete estimated skills graph ( 97 . 4% dense), and we find that stratified sampling on these skills only improves validation loss per skill by 0 . 007 points over random sampling on average (Figure 2 left), suggesting that utilizing skills does not improve model performance in this case.

The empty graph demonstrates that each skill is independent. This can occur if skills are too granular; for instance, learning Spanish math problems is unlikely to help with English poem generation. This graph suggests that the best approach for learning an individual skill is to train on the skill itself. Wesee that empty graphs exist in real data; in Figure 2 (center), using data sources as skills on the Pile of Law [22] results in a nearly empty skills graph ( 3 . 9% dense).

Graphs that are neither empty nor complete thus suggest a nontrivial order of how skill influence each other. This is the setting in which we expect that identifying skills and exploiting their ordering will help the most . In Figure 2 right, we use task categories, which capture broader reasoning patterns, as skills on Natural Instructions and find that the estimated graph has intermediate density ( 42 . 7% dense). We show concrete examples of how skills can be learned more efficiently on Natural Instructions in Section 3.2.

While these intuitive groupings result in ordered skill sets on some datasets (e.g., task categories on NI), this is not always the case (e.g., instruction types on Alpaca and sources on Pile of Law). Even though these groupings capture some notion of diversity in the dataset, our findings suggest that not just any semantic grouping induces an ordered skill set. We now empirically demonstrate that our definition of ordered skill sets aligns with how models learn and can be exploited for more data-efficient training.

3.2 Examples of skills and ordered skill sets

Weprovide examples of ordered skill sets on the LEGO synthetic dataset, an addition synthetic dataset, and subsets of the Natural Instructions dataset. On these datasets, we find that certain skills are better learned when trained along with their prerequisite skills rather than in isolation.

0

1000

2000

3000

4000

5000

6000

Steps

0.0

0.2

0.4

0.6

0.8

Validation Loss

Model performance on LEGO skill 3

Trained on skill 3

Trained on skills 1, 2, 3

0

1000

2000

3000

4000

5000

6000

Steps

0.0

0.5

1.0

1.5

2.0

Validation Loss

Model performance on Addition skill 1

Trained on skill 1

Trained on skills 1, 2

2.4

2.6

2.8

Validation Loss

Model performance on Spanish QG

Trained on Spanish QG

Trained on [Spanish, English] x [QA, QG]

0

100

200

300

400

500

600

Steps

1.5

2.0

2.5

Validation Loss

Model performance on stance detection

Trained on stance detection

Trained on stance detection, text matching

LEGOskills The LEGO synthetic, first introduced in [72], can evaluate a model's ability to follow a chain of reasoning. In this synthetic, the letters of the alphabet, A , are variables each with some binary label in { 0 , 1 } . Anindividual sample consists of k clauses for some fixed k across the dataset, each of the form a = gx where a,x ∈A and g is either a negation ('not') or assertion ('val'), e.g. we assign a to the value of x , or we assign a to the opposite label. At the end of the sentence, we prompt the model for what the value of one of these variables is. Two samples x ∈X are given below for k =5 :

Input: b = not y, r = val 1, m = val b, q = val m, y = not r. Output: b = 1.

Input: c = val x, p = val f, x = val k, f = not c, k = val 0. Output: k = 0.

These samples each correspond to a chain of reasoning; for instance the first sample has the chain r,y,b,m,q , where knowing q 's label requires the most reasoning steps. We define the i th skill s i as the model's ability to know the i th variable of the chain. From our example above, the first sample belongs to X s 3 and the second sample belongs to X s 1 . To demonstrate the existence of ordered skill sets, we continually pre-train the 125M parameter GPT-Neo model [14, 6] over various mixtures of LEGO skills with k =5 . In Figure 3 (left), we find that in 35.9% fewer training steps, training on a balanced mixture of X s 1 , X s 2 , and X s 3 resulted in the same validation loss of 0 . 01 as training solely on X s 3 . This suggests that s 1 ,s 2 helped unlock performance on s 3 and that there exist edges from s 1 or s 2 to s 3 in the skill graph. Additional observations are available in Appendix E.1, where we examine other edges as well as more complex reasoning chains, and the full skills graph corresponding to the ordered skill set for LEGO with k =5 is in Figure 11.

Addition skills Weconsider a variant of a synthetic 5-digit addition dataset analyzed in [45]. We show the existence of ordered skill sets for a simplified 3-digit addition dataset where we treat each digit prediction as a skill-the outputs, in this case, are the integers { 0 , 1 ,..., 9 } . Examples are of the following form:

where 'A 0' refers to the ones digit of the output ( s 1 ) and 'A 2' refers to the hundreds digit ( s 3 ). In Figure 3 (center), we find that in 32% fewer training steps, training on a balanced mixture of X s 1 , and X s 2 resulted in the same validation loss of 0 . 01 as training solely on X s 1 . That is, the ones digit addition skill can be improved by simultaneously learning the tens digit addition skill, even though the former should not require information from the latter-this is in line with observations from prior work that models do not always learn the ones digit addition first [45]. The full skills graph corresponding to the ordered skill set over 3-digit addition is in Figure 12.

Natural Instructions (NI) skills Weshow that ordered skill sets exist in NI [63] when we treat task categories as skills.

In Figure 3 (top right), we show that ordered skill sets exist over crosslingual task categories. Training on Spanish question generation (QG) along with equal parts of English QG, Spanish question answering (QA), and English QA results in 4 . 1% lower validation loss than training only on Spanish QG. Remarkably, the former only uses 25% of the latter's Spanish QG data. This suggests that there are edges from Spanish QA, English QA, and English QG to Spanish QG.

In Figure 3 (bottom right), we see that training on the task category Text Matching along with Stance Detection helps decrease the loss on Stance Detection by 11% . This suggests that these categories, which both involve understanding the relationship between two input texts, share an edge.

The full skills graphs corresponding to the ordered skill sets over these task categories are in Figure 14. While equating task categories to skills may be noisy, these examples suggest that there is signal within real data that suggests that ordered skill sets can improve data efficiency.

3.3 Skill recovery

Afinal component of characterizing skills is unsupervised recovery of ordered skill sets. We consider embedding-based clustering approaches and a loss-based clustering approach for recovering LEGO skills. When clustering data using various trained and pre-trained embeddings, we find that they were unable to achieve above 39% accuracy on LEGO. Instead, we find that taking 10 random training runs and clustering data by their loss per timestep per run recovers the skills with 61% accuracy (Table 3). The intuition behind this method is that the validation losses on points from the same skill have similar trajectories as models learn. We discuss this approach more in Appendix E.2.

4 Skills-based data selection

Nowthat we have established the existence of ordered skill sets, we discuss how to use them for data selection. We state the data selection problem for learning across skills in Section 4.1. We discuss how to learn the skills graph that will be exploited in our data selection methods in Section 4.2. We then introduce two sampling methods that utilize the graph, a simple skill-stratified sampling method and the online sampling method SKILL-IT, in Section 4.3.

4.1 Problem statement

We are given an ordered training skill set S train = { s train , 1 ,...,s train ,k } on the training data, each with associated support set X s train , 1 ,... X s train ,k , and an ordered evaluation skill set S eval = { s eval , 1 ,...,s eval ,m } of m evaluation skills on a separate evaluation dataset. We aim to select n samples from S train via a mixture of training skills, p ∈ ∆ k -1 , to achieve three goals depending on how S eval is constructed:

Continual pre-training : when S eval = S train, our goal is select a mixture of training skills to learn all of them.

Fine-tuning : when S eval ⊂S train, our goal is to select a mixture of training skills to learn an individual target skill or subset of these skills.

Out-of-domain : when S eval ∩S train = ∅ , our goal is to select a mixture of training skills to learn a disjoint set of evaluation skills we cannot train on. This can arise when we have a separate downstream validation dataset or the skills identified in the training dataset are noisy.

Furthermore, we have a skills graph G =( S train ∪S eval ,E ) , where E ⊆S train ×S eval and A ∈ R k × m is a weighted adjacency submatrix, where A ij describes the strength of the edge from s train ,i to s eval ,j . In Table 1, we summarize how the three different settings are constructed and how A varies across them. Next, we discuss how A can be estimated from the data.

4.2 Skills graph learning

The skills graph is important for determining how to sample from the ordered skill set for training efficiently. We present two approaches for learning the skills graph-brute-force and linear approximation. Algorithms are provided in Appendix C.2. By definition 2, the brute-force way of identifying edges involves fixing an overall training budget of H steps and 1) training and evaluating the model on each s train ,i and 2) training the model on each pair of ( s train ,i ,s eval ,j ) and evaluating on s train ,i and s eval ,j . If the loss on s eval ,j when trained on both s train ,i and s eval ,j is lower, there exists an edge from s train ,i to s eval ,j with edge weight proportional to the difference in loss. This approach has runtime O ( Hkm ) and is only feasible when k is small and when we have access to S eval at training

Algorithm 1 SKILL-IT Online Data Selection Algorithm

1: Input: Ordered training skill set S train, ordered evaluation skill set S eval. Learning rate η , T rounds, n samples, H training steps per run for graph learning, model f 1 , window parameter w .

2: A ← LEARNGRAPH ( S train , S eval ,H,f 1 ) (Alg. 2, 3).

3: Initialize p i 1 =exp( η ∑ m j =1 A ij ) for all i ∈ [ k ] , the softmax over A .

4: for t =1 ,...,T -1 do

5: Observe losses L eval ,j ( f t ) for all s eval ,j ∈S eval .

6: Train model f t with n/T samples from mixture p t over S train. Update model f t +1 =Φ( f t ,p t ) .

7: Set p i t +1 =exp( η ∑ t τ = t -w +1 ∑ m j =1 A ij L eval ,j ( f τ )) .

8: end for

time. Otherwise, we can approximate this approach in linear time by training on each s i for h<H steps and setting A ij > 0 if the loss on s j decreases over h steps for a runtime of O ( hk ) . This linear approach is necessary in the out-of-domain setting, since it does not require training on S eval . In addition, both graph learning approaches can be performed on a smaller model, and the learned graph can be used for data selection for training a larger model (Appendix E.4).

4.3 Skills graph-aware sampling

Wepresent two approaches for sampling over the mixture of training skills according to the skills graph: skill-stratified sampling, which samples uniformly over relevant training skills according to A , and SKILL-IT, an online generalization that incorporates feedback of how skills are being learned so far.

4.3.1 Skill-stratified sampling

Astraightforward sampling approach is to discard training skills that do not benefit the evaluation skills and sample uniformly over the set of relevant training skills, which we call skill-stratified sampling . For continual pre-training, the relevant skills are the entire training skill set; for each s train ,i ∈S train , Pr( s train ,i ) = 1 k . This enables each skill to have sufficient training data. For fine-tuning, the relevant skills are the target skills and prerequisite skills, which can be identified via positive entries of the i th column of A with S prereq = { s train ,i : ∃ s eval ,j s.t. A ij > 0 } . We then set Pr( s ) = 1 |S prereq ∪S eval | for s ∈S prereq ∪S eval. For the out-of-domain setting, skill-stratified sampling is over the set of prerequisite skills. For each s ∈S prereq, we set Pr( s )= 1 |S prereq | . Next, we propose our online algorithm that exploits the graph dynamically for more efficient training.

4.3.2 SKILL-IT online data selection algorithm

Despite accounting for prerequisite skills, one shortcoming of skill-stratified sampling is that even if a skill has already obtained sufficiently low validation loss early during training, we will continue to allocate the same weight to that skill throughout training. Therefore, we formulate our data selection problem as an online learning problem and propose SKILL-IT, which both prioritizes prerequisite skills and skills that are not yet learned.

Weare given a budget of T rounds and n total samples to train on. At round t , we select a mixture p t ∈ ∆ k -1 from the k -dimensional unit simplex, and for each training skill s train ,i ∈S train, we sample from X s train ,i with proportion p i t for a total of n T samples per round. Let f t be the model at at the start of round t . Wecan define f t recursively as a function of the previous round's model f t -1 and mixture p t -1 via a dynamics function Φ: F× ∆ k -1 →F ; that is, f t =Φ( f t -1 ,p t -1 ) . Let L eval ,j ( f t ) be the validation loss of f t on s eval ,j . Our goal is to select p 1 ,...,p T to minimize loss per evaluation skill at the end of training:

This optimization problem is challenging to solve without additional assumptions. In order to make the problemtractable, we impose an explicit dynamics rule for the each evaluation skill's loss L eval ,j in terms of the current loss and data mixture. Assuming for simplicity that S eval ⊆S train, a simple rule would be L eval ,j ( f t ) = L eval ,j (Φ( f t -1 ,p t -1 )) := L eval ,j ( f t -1 )(1 -αp j t -1 ) for α ∈ [0 , 1] . That is, we expect that allocating more data to skill j should result in the validation loss on skill j decreasing. However, such

an expression assumes that only training on the j th skill will help learn the j th skill. Instead, Section 3.2 suggests that there are other skills that may help with the j th skill. We propose the following dynamics:

where A : ,j is the column with weights of all skills that influence s eval ,j , and we absorb the scalar α into A . The optimization problem in (1) can thus be simplified as follows:

In Appendix C, we derive the following update rule via online mirror descent [46] for learning rate η> 0 :

In addition, when equation 4 is expanded, we have that p i t +1 = p i 1 exp ( η ∑ t τ =1 ∑ m j =1 A ij L eval ,j ( f τ ) ) . Since this summation over τ results in diminishing strength of updates, we change it to a moving window of size w . Our full method is in Algorithm 1.

Intuitively, at each step we adjust the weight on skill i based on the losses of skills that i influences, with the assumption that more training data helps decrease loss. Note that when we use our algorithm with a complete graph or empty graph, we achieve expected behavior discussed in Section 3.1. For the complete graph, our algorithm reduces to stratified sampling. When we have a skill set with an empty graph, the update rule reduces to sampling proportional to each skill's validation loss.

5 Experimental results

Given an ordered skill set, we aim to validate SKILL-IT's ability to select data for efficiently learning skills in the continual pre-training, fine-tuning, and out-of-domain settings. We provide full tables of results in Appendix E.3.1 and results where we learn the skills graph on the 125M model and use it for the 1.3B parameter model in Appendix E.4. Skills graphs are in Appendix D.2, weight trajectories for SKILL-IT are in Appendix E.3.2, and ablations on the graph and online components of SKILL-IT are in Appendix E.5.

5.1 Continual pre-training

Setup Weevaluate the ability of SKILL-IT to select data for efficiently learning over all skills. We measure average validation loss per skill after a fixed number of training steps. We construct the LEGO synthetic and addition synthetic with k =5 and 3 , respectively, and an imbalanced dataset over the skills. On the Natural Instructions dataset, we use 23 of the task categories as skills.

Baselines We compare SKILL-IT against three baselines that do not account for skills: random sampling, curriculum learning, and anticurriculum learning. Random sampling is a standard procedure for selecting samples given no additional information. Curriculum learning [4] and anticurriculum learning [67] score the samples from easiest to hardest and vice versa, respectively, and sample over an expanding set of the lowest scored samples at every epoch; we use the pre-trained model's loss to rank points. We evaluate skill-stratified sampling, which uses knowledge of the skills but is not online, and include an additional skills curriculum and anticurriculum baseline from [60], which samples from skills in order of average loss per skill.

Analysis Across our experiments we find that SKILL-IT outperforms baselines that do not use skills as well as skill-stratified sampling and skill curriculum learning. Our results on the LEGO dataset are shown in Figure 4. SKILL-IT and skill-stratified sampling attain lower loss than other approaches on skills 2, 3, 4 and on average, and while curriculum and anticurriculum learning attain lower loss on skill 5, they fail to learn other skills, resulting in at most 68 points accuracy on skill 3. SKILL-IT reaches

0

2000

4000

6000

10

3

10

1

10 1

Validation Loss (Log)

LEGO Skill 1

0

2000

4000

6000

10

2

10

1

10 0

0

2000

4000

6000

10

1

10 0

LEGO Skill 3

0

2000

4000

6000

Steps

10

1

10 0

Validation Loss (Log)

LEGO Skill 4

0

2000

4000

6000

Steps

10

3

10

2

10

1

10 0

LEGO Skill 5

0

2000

4000

6000

Steps

10

1

10 0

Average per skill

Random

Curriculum

Anticurriculum

Skill-stratified

Skill-curriculum

Skill-anticurriculum

Skill-It

0

2000

4000

6000

Steps

10

2

10

1

10 0

10 1

Validation Loss (Log)

0

2000

4000

6000

Steps

10

2

10

1

10 0

10 1

0

2000

4000

6000

Steps

10

2

10

1

10 0

10 1

0

2000

4000

6000

Steps

10

2

10

1

10 0

10 1

Average per skill

Random

Curriculum

Anticurriculum

Skill-stratified

Skill-curriculum

Skill-anticurriculum

Skill-It

a high average accuracy earlier in training than other approaches; halfway through training, SKILL-IT obtains between 8 . 3 and 33 . 5 points higher average accuracy than other approaches, reaching a final accuracy of 99 . 3 (Figure 20). SKILL-IT initially allocates more weight to prerequisite skills such as skill 2 as suggested by Figure 11 and later on allocates more weights to the skills that are learned more slowly, such as skills 4 and 5 (Figure 22). On the addition synthetic with k =3 , SKILL-IT obtains lower validation loss than the baselines on skills 1 and 2 in Figure 5. While most approaches aside from skill curriculum learning eventually obtain 100% accuracy on all skills, SKILL-IT requires less training to reach sufficiently high accuracy; halfway through training, SKILL-IT has accuracy between 8 . 7 and 73 . 1 points higher than other approaches (Figure 21). Finally on Natural Instructions, the average validation loss from SKILL-IT is 3 . 2% lower than from random sampling (Table 7). Our results suggest that exploiting the construction and ordering of skills is critical to learning skills quickly.

5.2 Fine-tuning

0

2000

4000

6000

Steps

0.00

0.25

0.50

0.75

Validation Loss

Performance on Skill 3

Skill 3 only

Skill-stratified

Skill-It

0

2000

4000

6000

Steps

0

1

2

Validation Loss

Performance on Skill 1

Skill 1 only

Skill-It

200

400

600

Steps

2.3

2.4

2.5

2.6

Validation Loss

Performance on Spanish QG

Spanish QG only

Skill-stratified

Skill-It

200

400

600

Steps

1.2

1.4

1.6

1.8

Validation Loss

Performance on stance detection

Stance detection only

Skill-stratified

Skill-It

Figure 6: Performance of SKILL-IT in the fine-tuning setting on LEGO, addition, and NI.

Setup Weevaluate the ability of SKILL-IT to select data from an ordered training skill set for learning a target skill. Mirroring Figure 3, we evaluate on LEGO target skill 3 (third in reasoning chain), on the addition synthetic's skill 1 (ones place digit addition), and on NI's Spanish QG and Stance Detection.

Baselines Wecompare SKILL-IT against training on the target skill only and skill-stratified sampling over prerequisite skills and the target skill. The skill-stratified sampling approach uses the ordered skill set to identify prerequisite skills, but does not exploit them dynamically.

Analysis Our results are shown in Figure 6. On LEGO, SKILL-IT results in the same validation loss of 0 . 01 as training only on the target skill in 38 . 1% fewersteps. We observe a similar trend on addition, with SKILL-IT converging to a validation loss of 0 . 01 in 59% fewer steps required to do so when training only on the target skill. Finally, on NI, SKILL-IT improves validation loss on Spanish question generation by 5 . 3% and Stance Detection by 13 . 6% over just training on the respective target skill only. In this setting,

3.06

3.08

3.10

Validation Loss

Answerability Classification

2.08

2.10

2.12

2.14

Cause Effect Classification

3.10

3.12

3.14

3.16

Coreference Resolution

2.36

2.37

2.38

Data To Text

Random

Skill-stratified

Skill-It

2.32

2.34

2.36

2.38

Validation Loss

Dialogue Act Recognition

2.40

2.42

2.44

Grammar Error Correction

2.74

2.76

2.78

2.74

2.76

2.78

2.80

2.82

Overlap Extraction

1000

2000

3000

4000

5000

Steps

2.60

2.62

Validation Loss

Question Rewriting

1000

2000

3000

4000

5000

Steps

2.48

2.50

2.52

2.54

Textual Entailment

1000

2000

3000

4000

5000

Steps

3.02

3.03

3.04

3.05

Title Generation

1000

2000

3000

4000

5000

Steps

1.67

1.68

1.69

1.70

Word Analogy

0

1

2

3

Number of tokens (billion)

0.638

0.640

0.642

0.644

Uniform

Skill-It

RedPajama source

SKILL-IT mixture

ArXiv

0.1370

Books

0.0437

C4

0.4195

CommonCrawl

0.0732

GitHub

0.1890

StackExchange

0.0892

Wikipedia

0.0484

Figure 8: Left: Accuracy on LM Evaluation Harness for continual pre-training of a 3B parameter model using SKILL-IT on the RedPajama dataset. We achieve higher accuracy at 1B additional tokens than uniform at 3B tokens. Right: SKILL-IT mixture over RedPajama sources.

a significant portion of the improvement over training only on the target skill comes from identification of prerequisite skills through the learned graph in the skill-stratified sampling method. SKILL-IT is further able to improve performance with finer-grained dynamic weighting on prerequisite skills.

5.3 Out-of-domain setting

Natural Instructions Weevaluate the ability of SKILL-IT to select data from a set of training skills for learning a disjoint set of evaluation skills that we cannot train on. We use all 59 task categories in the NI train tasks split as the training skills and the 12 task categories in the test tasks split as our evaluation skills. We compare SKILL-IT against random and skill-stratified sampling, both of which do not exploit the relationships between training skills and evaluation skills. SKILL-IT achieves the lowest loss on 11 out of 12 task categories over random and skill-stratified sampling (Figure 7, tables in Appendix).

RedPajama We use SKILL-IT to produce a data mixture on the RedPajama dataset. The training skills are the data sources comprising the dataset, and the evaluation skills are several tasks from the Language Model Evaluation Harness [15]. SKILL-IT with T =1 yields the mixture in Figure 8 (right). Wecontinually pre-train a 3B parameter model trained on 1T tokens for 3B additional tokens using this mixture, and see that it outperforms uniform sampling over the data sources (Figure 8 left). In particular, SKILL-IT achieves higher accuracy with 1B additional tokens than uniform with 3B additional tokens.

6 Conclusion

Given a fixed budget of data, knowing what data to train on to induce various capabilities in an LM is challenging. As LMs continue to improve, it will become increasingly important to extract as much signal as possible from the data and to direct that signal towards acquiring a broad variety of capabilities. In this paper, we introduce a skills-based framework for understanding how LMs learn and for selecting training data. We hope our study invites others to build on such a notion of skill and further explore how to align skills with data.

References

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S. Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023.

Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models, 2023.

Yuntao Bai, Andy Jones, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48, 2009.

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.

Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. If you use this software, please cite it using these metadata.

Rishi Bommasani, Percy Liang, et al. On the opportunities and risks of foundation models, 2021.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Mark Chen, Jerry Tworek, et al. Evaluating large language models trained on code, 2021.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: Anopen-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, 2019.

Robert M Gagne. The acquisition of knowledge. Psychological review , 69(4):355, 1962.

Robert M Gagne and Noel E Paradise. Abilities and learning sets in knowledge acquisition. Psychological Monographs: General and Applied , 75(14):1, 1961.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021.

Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023.

Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning , pages 2242-2251. PMLR, 2019.

Google. Palm2 technical report. Technical report, 2023.

Anupam Gupta. Advanced algorithms: Notes for cmu 15-850 (fall 2020), 2020.

Suchin Gururangan, Ana Marasovi´ c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020.

Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Simfluence: Modeling the influence of individual training examples by simulating training runs, 2023.

Peter Henderson*, Mark S. Krass*, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset, 2022.

Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability of learning from repeated data, 2022.

Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data, 2022.

Joongwon Kim, Akari Asai, Gabriel Ilharco, and Hannaneh Hajishirzi. Taskweb: Selecting better source tasks for multi-task nlp, 2023.

Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 2611-2624. Curran Associates, Inc., 2021.

Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3499-3505, Florence, Italy, July 2019. Association for Computational Linguistics.

Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2676-2686, Melbourne, Australia, July 2018. Association for Computational Linguistics.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, 2017.

Po-Nien Kung, Sheng-Siang Yin, Yi-Cheng Chen, Tse-Hsuan Yang, and Yun-Nung Chen. Efficient multi-task auxiliary learning: Selecting auxiliary data by feature similarity. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 416-428, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Michael Langberg and Leonard J. Schulman. Universal approximators for integrals , pages 598-607.

Hugo Laurençon, Lucile Saulnier, et al. The bigscience roots corpus: A 1.6tb composite multilingual dataset, 2023.

Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2022.

David D Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum , volume 29, pages 13-19. ACM New York, NY, USA, 1995.

Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards a mechanistic understanding, 2023.

Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6565-6576. PMLR, 18-24 Jul 2021.

AmanMadaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners, 2022.

Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, 2023.

Sören Mindermann, Muhammed Razzak, Winnie Xu, Andreas Kirsch, Mrinank Sharma, Adrien Morisot, Aidan N. Gomez, Sebastian Farquhar, Jan Brauner, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learned (workshop version), 2021.

Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models, 2019.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACL , 2022.

Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers , pages 220-224, Uppsala, Sweden, July 2010. Association for Computational Linguistics.

MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023.

Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021.

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations , 2023.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training, 2021.

Jeff M. Phillips. Coresets and sketches, 2016.

Siddharth Reddy, Igor Labutov, and Thorsten Joachims. Latent skill embedding for personalized lesson sequence recommendation, 2016.

Sebastian Ruder, Parsa Ghaffari, and John G. Breslin. Data selection strategies for multi-domain sentiment analysis, 2017.

Sebastian Ruder and Barbara Plank. Learning to select data for transfer learning with bayesian optimization. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , 2017.

Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning, 2022.

Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International Journal of Computer Vision , 130(6):1526-1565, Apr 2022.

Claire Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. Putting gpt-3's creativity to the (alternative uses) test, 2022.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Together. Redpajama-data: An open source recipe to reproduce llama training dataset, 2023.

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning, 2018.

HugoTouvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.

Neeraj Varshney, Swaroop Mishra, and Chitta Baral. Let the model decide its curriculum for multitask learning, 2022.

Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, Songyang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yunwen Chen, and Tao Gui. Farewell to aimless large-scale pretraining: Influential subset selection for language model, 2023.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022.

Yizhong Wang, Swaroop Mishra, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, 2022.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2021.

Richard T White. Research into learning hierarchies. Review of Educational Research , 43(3):361-375, 1973.

Richard T. White and Robert M. Gagné. Past and future research on learning hierarchies. Educational Psychologist , 11(1):19-28, 1974.

Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work?, 2020.

Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining, 2023.

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling, 2023.

Amir R. Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , Jun 2018.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.

Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: a synthetic reasoning task, 2022.

A Broader Impacts and Limitations

Broader Impacts As more LMs are developed, a key criteria for their adoption and utility is if they exhibit a wide array of useful capabilities, such as generating harmless content, summarizing essays, and being conversational with the user. While improvements in other parts of the LM development pipeline such as training and architecture are important, many recent advances in building LMs with a wide array of useful capabilities have come from the data itself [18, 56, 10, 16, 43]. Our work is fundamental in investigating how LMs learn and how to select data to learn skills more efficiently. However, we recognize that data selection methods can always be utilized to optimize for particular skills that may be considered malicious or negatively target or exclude specific groups [3]. Furthermore, pre-trained LMs have been found to have various biases [26, 44, 36, 7].

Limitations The skills graph can either be provided (e.g., using a knowledge graph) or learned. Our work learns the skills graph using Algorithm 2 or Algorithm 3, which requires initial training runs on pairs of skills or each skill, respectively. This can be made more efficient by performing these training runs on a smaller model and for fewer number of steps, but tradeoffs here have yet to be thoroughly investigated. SKILL-IT also assumes that the ordered skill set is provided; as discussed in sections 3.1 and 3.3, it is challenging to recover ordered skill sets simply via metadata attributes or embedding clustering. Otherwise, the best way to sample over collections of skills that form a complete or empty graph is random or stratified sampling with no ordering to exploit. Our loss-based clustering approach presented in section 3.3 demonstrates that grouping by losses can provide an explanation for how skills are defined over data. An important direction for future work is to use such a clustering approach or other unsupervised algorithms in an end-to-end pipeline for skill discovery, skill graph learning, and data selection based on such skills.

Code release The code for SKILL-IT is available at https://github.com/HazyResearch/ skill-it .

B Extended related work

Data selection for LMs There have been several studies of large-scale data selection for LMs. Data deduplication [33, 1, 23], in which identical or nearly identical samples are removed, is a method that enables LMs to be trained on smaller, cleaned datasets and has been increasingly used as a pre-processing step for training data [59, 5, 71]. Other methods applied at scale involve ensuring high quality of data by explicitly filtering out samples or comparing the training dataset with a cleaned reference dataset [8, 59, 32]. Importance reweighting approaches have also been proposed for identifying training data from a large corpus that best approximates a smaller target distribution [69], and influence functions have been used to select a subset of training data to improve performance on downstream tasks [61]. These approaches can identify data pertaining to a particular target distribution or filter out low quality data according to some heuristic, while our work aims to understand how the choice of data is related to the numerous skills that LMs learn.

RecentdevelopmentofLMshasshiftedfocusfromemphasizingthescaleofthemodeltoprioritizingthe training data utilized. For example, models like Alpaca [56], Vicuna [10], and Koala [16] are all based on the LLaMA model combined with instruction data generated by an existing LM. Palm 2's technical report states that the data mixture was a critical component of the final model [18], and Mosaic ML's recent MPT model was trained on a hand-engineered mixture of the RedPajama dataset [43]. However, these works lack rigorous explanation for why their training datasets were constructed in this way.

Finally, perhaps most related to our approach is the contemporary work DoReMi [68], which uses group distributionally robust optimization on a smaller LM to select data source mixtures for training a larger LM. Their approach focuses on selecting data at the data source level for optimizing worst-case performance across the training data sources, rather than at the more general skills level for a variety of target skill sets. Furthermore, we focus on understanding how skills are related to each other and induce some order in how LMs learn by explicitly modeling skill graph structure, which we find to be important for data-efficient LM training (see ablations in Appendix E.5).

Data selection methods Many data selection methods have been proposed for supervised, taskspecific settings. In this setting, the most typical objective is dataset condensation, which aims to identify a small subset of data that captures the larger dataset's properties with respect to the model.

Some approaches include constructing coresets [31, 48], identifying samples that the model forgets during training [58]; identifying samples with the largest gradients [47] or gradients that approximate the overall gradient [40]; clustering in embedding space and selecting points farthest from cluster centers [53]; and selecting samples with the highest uncertainty or entropy [34]. These approaches have also been shown to transfer from smaller models to larger models [11]. Unlike these methods, we study how to select data for learning one or many skills at the mixture level for LMs instead of the instance level.

Another area of interest is data selection for domain adaptation and multitask learning. For domain adaptation, there are a wide range of methods that select data to best match the target distribution. For example, the Moore-Lewis method matches data based on the difference in cross-entropy using a model trained on the target versus a model trained on the source data [42]. Several other approaches suggest training a model to distinguish between source and target and selecting points with high uncertainty [50], or selecting points based on some divergence in an embedding space [51]. In comparison to these approaches, our work focuses on learning one or many skills and also finds that embedding-based heuristics do not fully identify skills.

Data attribution Another perspective on understanding training data is data attribution, which seeks to identify what data is responsible for particular model behaviors. Influence functions [29] and shapley values [17] are two ways to quantify the role of individual samples. Datamodels [24] fit a model to predict behavior given a subset of training data, providing a framework for understanding individual samples as well as dataset counterfactuals. Simfluence [21] fits a Markov process to a set of training trajectories for finer-grained understanding of how data impacts training. We focus on understanding how groups of data associated with skills elicit broader model capabilities, and utilize this understanding to select data for more efficient training.

Curriculum learning Curriculum learning [4] proposes to show the model data in order from easy samples to hard ones. Various criteria have been used to determine hardness, and anticurriculum as well as various pacing functions and mixing rates have been explored [54]. Curriculum learning can also be performed at the group level [60]. More sophisticated approaches include parametrizing each sample with a dynamic importance [52], and also accounting for irrelevant and noisy data [39]. Our approach similarly utilizes a curriculum, but it is defined over a skills graph and does not necessarily align with training on easiest to hardest skills.

HowLMslearn Many different explanations for how LMs learn from data have been proposed. One hypothesis is that there exist discrete, universal building blocks of LM knowledge called quanta, and power law scaling emerges from a learning over a particular distribution of quanta in the right order [38]. Another work proposes that scaling laws for LMs can be understood in terms of learning combinations of skills [2]. Others have provided theoretical analysis of how transformers learn topics by studying co-occurrences of words in the training data [35]. Empirically, how models learn is still a mystery-for instance, models trained on code are found to perform fairly well at commensense reasoning [37]. Our work initiates a study on how LMs learn various skills and how to exploit this for better data selection.

Task selection In multitask auxiliary learning, the goal is to train a model to perform well on a target task(s) by selecting the most beneficial source tasks to train on. One can use feature similarity to select tasks [30], but we find in our synthetics that feature similarity does not always recover skills. In Taskonomy [70], a hypergraph over a set of tasks is learned and used to select tasks. The methods used to develop the taxonomy can be applied to further expand our graph learning (e.g., studying transitive and higher-order properties). However, their focus is on task selection in computer vision rather than data selection for LMs to learn skills. Lastly, the contemporary work of TaskWeb [25] builds a graph among 22 common NLP tasks in order to determine what the best source tasks are for a target task. Their definition of an edge in the task graph is less strict than ours (their comparison is on if training on additional data from s i helps with s j , while we fix the overall amount of training data over both s i and s j ). Overall, our approach is similar in use of the skills graph, but we incorporate it into a dynamic sampling algorithm. Furthermore, we look more broadly at skills, rather than tasks, and characterize when we expect using the skills graph to improve model performance.

Education The notion of skill has been studied in education. Classical research on learning hierarchies [66] identify sets of skills that make up subordinate capabilities for students. For instance, [13] identified that in order for students to solve linear equations, there were many prerequisite

skills, ranging from the simplest being symbol recognition to the most complex being the ability to add, subtract, multiple, and divide from both sides of the equation. More recently, decision-making over lesson sequences based on skills, e.g., what the student already knows versus what the lesson teaches, has become an area of interest in personalized learning [49].

C Additional Algorithmic Details

C.1 Derivation of SKILL-IT Update Rule

First, we provide the derivation of our update rule from online mirror descent using the proximal point view [19]. We restate our optimization problem from (3):

Let ¯ L t ( p )= 1 m ∑ m i = j L eval ,j ( f t +1 )= 1 m ∑ m i = j L eval ,j (Φ( f t ,p )) ; that is, p is the mixture we must choose at time t and ¯ L t is the average loss per skill of the model after it is trained on p at round t . Agreedy approximation of (5) is minimize p ∈ ∆ k -1 ¯ L t ( p ) , given the model and mixtures at previous rounds. A linear approximation of ¯ L t ( p ) is

Then, the problem of minimizing ¯ L t ( p ) becomes

after we drop terms from (6) that do not depend on p . Note that the η is a constant and does not impact the solution. The optimal solution to this problem is selecting the p that has the most weight on the slice with the largest gradient. To improve stability and prevent overfitting, we introduce regularization via a Bregman divergence D h ( p || p t -1 ) = h ( p ) -h ( p t -1 ) -⟨ ▽ h ( p t -1 ) ,p -p t -1 ⟩ . After dropping terms that do not contain p , our problem is now

Taking the gradient and setting it equal to 0 gives us

Similar to in standard multiplicative weights, we set h ( p )= ∑ i p i ln p i and ▽ h ( p )=[ln p i +1] i . Then,

where ▽ i is the i th element of the gradient. Now we wish to compute ▽ i ¯ L t ( p t ) = 1 m ∑ m j =1 ▽ i [ L eval ,j ( f t +1 )] = 1 m ∑ m j =1 ▽ i [ L eval ,j (Φ( f t ,p t ))] . Recall the dynamics model for L eval :

The gradient of this model with respect to each training skill s i is

where we can absorb the 1 m into η .

Plugging this back into (10),

Algorithm 2 Brute-Force Graph Learning

1: Input: Ordered skill set S = { s 1 ,...,s k } . Number of training steps K , base model f 0 . 2: for j ∈ [ k ] do 3: Train f 0 on samples from X s j for K steps and denote f K,j to be the model after training. 4: Observe change in loss, δ j j = L eval ,j ( f 0 ) -L eval ,j ( f K,j ) . 5: end for 6: for i,j ∈ [ k ] do 7: Train f 0 on samples from X s i ∪X s j for K steps and denote f K,i,j to be the model after training. 8: Observe change in loss, δ i,j j = L eval ,j ( f 0 ) -L eval ,j ( f K,i,j ) . 9: if δ ij j >δ j j then 10: Draw edge s i → s j and set A ij > 0 . 11: end if 12: end for 13: return Adjacency Matrix A ∈ R k × k

Algorithm 3 Approximate Graph Learning

1: Input: Ordered skill sets S train and S eval. Number of training steps K , base model f 0 . 2: for i ∈ [ k ] do 3: Train f 0 on samples from X s train ,i for K steps and denote f K,i to be the model after training. 4: for j ∈ [ m ] do 5: Observe change in loss, δ i j = L eval ,j ( f 0 ) -L eval ,j ( f K,i ) . 6: If δ i j > 0 , draw edge s train ,i → s train ,j and set A ij > 0 . 7: end for 8: end for 9: return Bipartite Adjacency Matrix A ∈ R k × m

C.2 Graph Learning Method

Weprovide algorithms for learning the graph over an ordered skill set. In Algorithm 2, we discuss the brute-force approach for learning the adjacency matrix. This approach only works when S eval ⊆S train (e.g. pre-training and fine-tuning cases), so we denote S = S train in the algorithm box. In Algorithm 3, we discuss the linear approach for learning the adjacency matrix. This approach works even in the out-of-domain case when S eval and S train are disjoint.

In both approaches, the exact value of A ij can vary, but we can typically set it proportional to δ i,j j -δ j j in the brute-force case or δ i j in the approximate case. The exact constructions and methods for learning each A in our experiments are in Appendix D.2.

D Additional Experimental Details

D.1 Datasets

We present details about each dataset used, including information on the skills and the validation dataset. A summary is presented in Table 2.

Alpaca dataset [56]: the Alpaca dataset consists of 52 Kinstruction examples that were generated from text-davinci-003. We applied the Berkeley Neural Parser [27, 28] to each instruction, keeping 40777 samples it was able to parse successfully. If the sample began with a question, we annotated it with the skill 'question', and otherwise we annotated it with the verb identified from the parser. We grouped the data into a total of 38 skills, such as "list", "edit", "calculate", "describe" and "identify".

Pile of Law [22]: the Pile of Law dataset consists of various sources of legal and administrative data, ranging from tax rulings to the world's constitutions. We evaluate on a subset of the Pile of Law validation dataset consisting of 13883 samples, where we selected max( 645 , source size) samples per source. We truncated each sample to be no more than 100 Kcharacters.

LEGO[72]: for the LEGO synthetic, we set k =5 and sample 192000 points across the skills. Our validation dataset consisted of 100 samples per skill.

Addition: for the 3 -digit addition synthetic, we set k =3 and sample 192000 points across the skills. Weuse a validation dataset of 100 samples per skill.

Natural Instructions [63, 41]: the Natural Instructions dataset is a large collection of tasks and their definitions in natural language. For the pre-training setting, we used a set of 23 task categories that had the largest degree (in-degree + out-degree) in the learned skills graph, for a total of 1 , 232 , 437 samples and 425 tasks to select from. We evaluated on 50 samples per task.

Forthefine-tuning setting with Spanish question generation, we select data over 4 skills (Spanish question generation, Spanish question answering, English question generation, English question answering) for a total of 513210 samples and 212 tasks to select from. We evaluated on 100 samples per task. For the fine-tuning setting with stance detection, we select data over 2 skills (stance detection, text matching) for a total of 50990 samples and 19 tasks to select from. We evaluated on 50 samples per task.

For the out-of-domain setting, we select data over all 59 task categories for a total of 2 , 417 , 867 samples and 753 tasks to select from. The test split consisted of 12 task categories and 119 tasks, and we evaluated on min( 400 , task size) samples per task.

RedPajama [57]: the RedPajama dataset is a 1 -trillion token dataset that aims to reproduce the LLaMA[59] training dataset. We select over the 7 data sources and evaluate using the LM evaluation harness [15].

D.2 Graph Learning Details

Wedescribe how the skills graph was learned on each dataset.

Alpaca (Figure 9): we use Algorithm 3 and train for K =150 steps per skill. Each edge i → j has a weight of δ i j , the difference in loss on skill j before and after training on i . Next, we compare the average validation loss of skill-stratified sampling versus random sampling when we train for K =1000 steps. We find that skill-stratified sampling only does 0.007 better than random sampling, confirming that Alpaca's dense skills graph suggests that random sampling is the best we can do.

Pile of Law (Figure 10): we use Algorithm 3 and train for K =150 steps. Each edge i → j has a weight of δ i j , the difference in loss on skill j before and after training on i .

LEGO(Figure 11): we use both Algorithm 2 and Algorithm 3 and train for K =6000 steps each. Each edge i → j has a weight of 0 . 5 if the amount of data associated with skill j that is needed to reach 0 . 01 validation loss is less when training on ( i,j ) than on j (edges are set to 0 if 0 . 01 validation loss is not reached, even if loss is decreasing). Each edge i → j is also set to 0 . 5 if training on i decreases loss directly on j . Weset each diagonal entry of A to be 1 .

Addition (Figure 12): we use Algorithm 2 and train for K =6000 steps. Each edge i → j has a weight of 0 . 5 if the amount of data associated with skill j that is needed to reach 0 . 01 validation loss is less when training on ( i,j ) than on j (edges are set to 0 if 0 . 01 validation loss is not reached, even if loss is decreasing). We set each diagonal entry of A to be 1 .

Natural Instructions (Figure 13, 14, 15): we use Algorithm 3. For the pre-training setting, we train for K =600 steps and assign each edge i → j a weight δ i j equal to the change in loss on j in the first 100 steps for all i,j ∈ [ k ] , including diagonal entries. For the fine-tuning setting, we train for K =600 steps and assign each edge i → j a weight δ i j equal to the change in loss before and after training. For the out-of-domain setting, we train for K =600 steps and assign each edge i → j a weight δ i j equal to the change in loss before and after training in the first 100 steps.

add

analyze

arrange

calculate

categorize

change

choose

classify

compare

compose

construct

convert

create

describe

design

determine

develop

edit

evaluate

explain

find

generate

give

identify

list

make

name

outline

provide

question

rewrite

select

suggest

summarize

take

tell

translate

write

add

analyze

arrange

calculate

categorize

change

choose

classify

compare

compose

construct

convert

create

describe

design

determine

develop

edit

evaluate

explain

find

generate

give

identify

list

make

name

outline

provide

question

rewrite

select

suggest

summarize

take

tell

translate

write

Alpaca

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

RedPajama (Figure 16): we use Algorithm 3 and train for 1 billion tokens per data source. We assign each edge i → j a weight δ i j equal to the change in perplexity on the validation datalsoa before and after training.

D.3 Training Details

Wedescribe the parameters used for SKILL-IT.

SKILL-IT pre-training

LEGO: η =0 . 5 ,T =6 ,w =3 . Wetrain for 6000 steps.

Addition: η =0 . 1 ,T =5 ,w =3 . Wetrain for 6000 steps.

Natural Instructions (pre-training): η =0 . 2 ,T =1 . Wetrain for 5000 steps.

For the LEGO random baseline, when we selected points at random, we used an imbalanced training dataset with proportions 1:1:1:3:5. For the addition random baseline, we used an imbalanced dataset with randomly selected proportions: 13:14:18. For the curriculum learning baselines, the pacing function, g ( i ) , denotes the size of the subset of the highest scoring samples that we uniformly select from in the i th epoch. We define our pacing function as g ( i ) = iH M , where H is the number of steps and M is 5 epochs for LEGO and NI, and 3 for addition.

SKILL-IT fine-tuning

LEGO: η =0 . 5 ,T =10 ,w =3 . Wetrain for 6000 steps.

Addition: η =0 . 1 ,T =5 ,w =3 . Wetrain for 6000 steps.

Natural Instructions (Spanish QG): η =0 . 8 ,T =6 ,w =3 . Wetrain for 600 steps.

Natural Instructions (stance detection): η =0 . 2 ,T =6 ,w =3 . Wetrain for 600 steps.

SKILL-IT out-of-domain

Natural Instructions: η =0 . 2 ,T =10 ,w =3 . Wetrain for 5000 steps.

RedPajama: η =100 ,T =1 . Wetrain for 3 billion tokens.

All results are computed over 5 random seeds.

atticus_contracts

bva_opinions

canadian_decisions

cfpb_creditcard_contracts

congressional_hearings

courtlistener_docket_entry_documents

courtlistener_opinions

dol_ecab

echr

edgar

eoir

eurlex

euro_parl

federal_register

hhs_alj_opinions

icj-pcij

medicaid_policy_guidance

nlrb_decisions

oig

olc_memos

r_legaladvice

resource_contracts

scotus_filings

scotus_oral_arguments

sec_administrative_proceedings

tax_rulings

un_debates

us_bills

uspto_office_actions

doj_guidance_documents

ed_policy_guidance

ftc_advisory_opinions

atticus_contracts

bva_opinions

canadian_decisions

cfpb_creditcard_contracts

congressional_hearings

courtlistener_docket_entry_documents

courtlistener_opinions

dol_ecab

echr

edgar

eoir

eurlex

euro_parl

federal_register

hhs_alj_opinions

icj-pcij

medicaid_policy_guidance

nlrb_decisions

oig

olc_memos

r_legaladvice

resource_contracts

scotus_filings

scotus_oral_arguments

sec_administrative_proceedings

tax_rulings

un_debates

us_bills

uspto_office_actions

Pile of Law

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1

2

3

4

5

1

2

3

4

5

LEGO

0.0

0.2

0.4

0.6

0.8

1.0

1

2

3

1

2

3

Addition

0.0

0.2

0.4

0.6

0.8

1.0

Natural Instructions

answer_verification

code_to_text

discourse_connective_identification

entity_generation

entity_relation_classification

information_extraction

irony_detection

preposition_prediction

punctuation_error_detection

question_answering

question_generation

question_understanding

sentence_expansion

sentiment_analysis

stance_detection

summarization

text_categorization

text_matching

text_simplification

text_to_code

toxic_language_detection

word_semantics

wrong_candidate_generation

answer_verification

code_to_text

discourse_connective_identification

entity_generation

entity_relation_classification

information_extraction

irony_detection

preposition_prediction

punctuation_error_detection

question_answering

question_generation

question_understanding

sentence_expansion

sentiment_analysis

stance_detection

summarization

text_categorization

text_matching

text_simplification

text_to_code

toxic_language_detection

word_semantics

wrong_candidate_generation

0.0

0.2

0.4

0.6

0.8

English QA

Spanish QA

English QG

Spanish QG

English QA

Spanish QA

English QG

Spanish QG

Spanish question generation

0.0

0.2

0.4

0.6

Stance Detection

Text Matching

Stance Detection

Text Matching

Stance detection

0.0

0.5

1.0

answerability_classification

cause_effect_classification

coreference_resolution

data_to_text

dialogue_act_recognition

grammar_error_correction

keyword_tagging

overlap_extraction

question_rewriting

textual_entailment

title_generation

word_analogy

answer_verification

code_to_text

coherence_classification

commonsense_classification

dialogue_generation

dialogue_state_tracking

discourse_connective_identification

discourse_relation_classification

entity_generation

entity_relation_classification

explanation

fact_verification

fill_in_the_blank

gender_classification

grammar_error_detection

information_extraction

intent_identification

irony_detection

linguistic_probing

mathematics

named_entity_recognition

negotiation_strategy_detection

number_conversion

paraphrasing

poem_generation

pos_tagging

preposition_prediction

program_execution

punctuation_error_detection

question_answering

question_decomposition

question_generation

question_understanding

sentence_composition

sentence_compression

sentence_expansion

sentence_ordering

sentence_perturbation

sentiment_analysis

spam_classification

speaker_identification

spelling_error_detection

stance_detection

stereotype_detection

story_composition

style_transfer

summarization

text_categorization

text_completion

text_matching

text_quality_evaluation

text_simplification

text_to_code

toxic_language_detection

translation

word_relation_classification

word_semantics

wrong_candidate_generation

Natural Instructions

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

arc_challenge

arc_easy

copa

lambada_openai

piqa

winogrande

arxiv

books

c4

common_crawl

github

stackexchange

wikipedia

RedPajama

0.0

0.1

0.2

Batch sizes of 32 and 64 were used for the LEGO and addition synthetic on the 125M and 1.3B parameter model, respectively. Batch sizes of 4 and 16 were used for the Natural Instructions experiments on the 125M and 1.3B parameter model.

For the out-of-domain Natural Instructions experiment and Alpaca graph learning experiments, a learning rate of 5e-6 with linear scheduler and 50 warmup steps was used. All other experiments used a learning rate of 5e-5. All experiments used AdamW with betas = 0.9, 0.999, eps = 1e-8, and weight decay = 0 . 01 . A context window of 512 was used for all experiments except LEGO and addition, which used a window of 128 .

Experiments with the Addition dataset were run using an Nvidia RTX A6000. Other experiments using the GPT-Neo 125M parameter model were run on an Nvidia Tesla P100. Experiments using the GPT-Neo 1.3B parameter model were run on an Nvidia Tesla A100.

E Additional Experimental Results

E.1 Additional examples of LEGO ordered skill sets

For the LEGO synthetic, it may appear obvious that the skills graph is equivalent to the reasoning chain over the variables. However, in Figure 17 we see that this is not the case. Training on skills 2 and 4 together results in lower loss on skill 4 than when trained on skill 4 alone. However, training on skills 3 and 4 together results in roughly the same loss on skill 4 as when training on skill 4 alone, even though skill 3 and skill 4 share an edge in the LEGO synthetic's underlying reasoning chain. This suggests that our intuition for how skills influence each other does not always match how the model learns skills.

Next, we consider a slightly more complex reasoning pattern on the LEGO synthetic. Instead of a chain, we construct a tree, where two variables in the LEGO synthetic are both defined in terms of the same parent variable. For example,

In this example, k =4 and both v and w are written in terms of c , and the reasoning graph has edges 1 → 2 , 1 → 3 , 2 → 4 . In this case, we see that training on skill 2 or skill 3 both improve losses on skills 2 and 3 (Figure 18). However, unlike the previous figures, training on skills 2 and 4 or skills 3 and 4 do not significantly help reduce loss on skill 4 (Figure 19). Again, these measurements demonstrate that the reasoning graph does not necessarily equal the skills graph.

E.2 Unsupervised skill recovery

We explore several clustering techniques for recovering the skills in the LEGO synthetic on the validation dataset. Our results are shown in Table 3.

Wefirst cluster based on the pre-trained model embeddings of the last token and the average token. We also report accuracies of clustering based on the trained model embedding's last token, where we train

0

2000

4000

6000

Steps

0.4

0.5

0.6

0.7

0.8

0.9

Validation Loss

Model performance on LEGO skill 4

Trained on skill 4

Trained on skills 2, 4

0

2000

4000

6000

Steps

0.68

0.70

0.72

Validation Loss

Model performance on LEGO skill 4

Trained on skill 4

Trained on skills 3, 4

0

1000

2000

3000

Steps

0.0

0.2

0.4

0.6

Validation Loss

Model performance on LEGO skill 3 (tree)

Trained on skill 3

Trained on skill 2

0

1000

2000

3000

Steps

0.0

0.2

0.4

0.6

Validation Loss

Model performance on LEGO skill 2 (tree)

Trained on skill 3

Trained on skill 2

0

1000

2000

3000

Steps

0.0

0.2

0.4

0.6

0.8

Validation Loss

Model performance on LEGO skill 4 (tree)

Trained on skill 4

Trained on skills 2, 4

0

1000

2000

3000

Steps

0.0

0.2

0.4

0.6

0.8

Validation Loss

Model performance on LEGO skill 4 (tree)

Trained on skill 4

Trained on skills 3, 4

the model using random sampling for 6000 steps, and clustering based on Sentence-BERT embeddings. Amongthese four methods, using the trained model embeddings has the highest accuracy of 38 . 4 points.

Next, we cluster points based on losses. In particular, we do 10 runs, each for 6000 steps and with a randomly sampled mixture of skills. For each run, we evaluate the model on the validation dataset at 120 checkpoints. Then, each sample in the validation dataset has 1200 losses associated with it, comprising a feature vector for that sample. We perform k-means clustering on these features, which has an accuracy of 61 . 0 points, significantly higher than the second best accuracy of 38 . 4 .

E.3 Full results for Section 5

E.3.1 Per-skill performance

In this section, we provide tables containing the per skill break-down of our results from Section 5.

Continual Pre-training In the continual pre-training setting, we report two additional baselines that combine curriculum learning with skills. Curriculum learning has been proposed for multitask learning [60], in which groups of data are ranked by their average score and then trained in order of this ranking (with mixing of previously seen groups to avoid forgetting). We construct two baselines, Skill-curriculum and Skill-anticurriculum, using Algorithm 1 from [60]. In contrast to the random baseline which has imbalanced skills, this approach has knowledge of skills and thus uses a skill-stratified training dataset to sample from. We set the fraction of the previous group to be frac =0 . 4 , as we found that setting frac =0 . 0 resulted in forgetting.

Wereport loss per skill for the LEGO synthetic in Table 4, which corresponds to the results in Figure 4. We report accuracy per skill in Table 5 and Figure 20. We report the loss per skill for the Addition synthetic in Table 6, which also correspond to to the results in Figure 4, and we provide the accuracy per skill in Figure 21 (we omit a table of accuracies since most methods attained 100% accuracy by the end of training). Finally, we report validation loss per task category for the Natural Instructions continual pre-training experiment in Table 7, where we find that SKILL-IT outperforms random sampling by 3 . 2% on average across skills.

0

2000

4000

6000

0.0

0.2

0.4

0.6

0.8

1.0

LEGO Skill 1

0

2000

4000

6000

0.4

0.6

0.8

1.0

0

2000

4000

6000

0.4

0.6

0.8

1.0

LEGO Skill 3

0

2000

4000

6000

Steps

0.6

0.8

1.0

LEGO Skill 4

0

2000

4000

6000

Steps

0.4

0.6

0.8

1.0

LEGO Skill 5

0

2000

4000

6000

Steps

0.2

0.4

0.6

0.8

1.0

Average per skill

Random

Curriculum

Anticurriculum

Skill-stratified

Skill-curriculum

Skill-anticurriculum

Skill-It

0

2000

4000

6000

Steps

0.2

0.4

0.6

0.8

1.0

0

2000

4000

6000

Steps

0.2

0.4

0.6

0.8

1.0

0

2000

4000

6000

Steps

0.2

0.4

0.6

0.8

1.0

0

2000

4000

6000

Steps

0.0

0.2

0.4

0.6

0.8

1.0

Average per skill

Random

Curriculum

Anticurriculum

Skill-stratified

Skill-curriculum

Skill-anticurriculum

Skill-It

Out-of-domain In Table 8, we provide a breakdown of validation loss per evaluation skill under random sampling on the training data, skill-stratified sampling over prerequisite skills (e.g., the nonzero rows in Figure 15), and SKILL-IT.

In Table 9 we provide a breakdown of the RedPajama experiment's accuracy per evaluation skill, corresponding to the results in Figure 8.

E.3.2 Weight trajectories

Weprovide SKILL-IT's weight trajectories for each result. The weight per skill across training steps for the LEGO pre-training experiment corresponding to Figure 4 (left) is shown in Figure 22. We see that SKILL-IT initially allocates more weight to skill 2 and less to 1 , 3 , 4 , 5 . Since skill 1 is learned quickly, the weight on skill 1 immediately drops to below 0 . 1 at 1000 steps. The weight on skills 3 , 4 , and 5 increase from around 0 to 3000 steps, during which their respective validation losses are higher than those of skills 1 and 2 . Near the end of training, all losses are converging to 0 , and so the weight per skill is roughly uniform.

The weight per skill across training steps for the addition pre-training experiment corresponding to Figure 4 (right) is shown in Figure 23. SKILL-IT allocates more weight to skill 2 , which has an edge to skill 1 as shown in Figure 12. It also allocates very little weight to skill 3 , which is learned faster than the other two skills. Eventually, it puts more weight on skill 1 , the hardest skill, and then converges to uniform sampling as all validation losses approach 0 .

The weight per skill across training steps for the LEGO fine-tuning experiment and the Spanish question generation and stance detection experiments corresponding to Figure 6 is shown in Figure 24. Since there is only one target skill in these experiments, the mixture of weights approaches uniform

0

1000

2000

3000

4000

5000

6000

Steps

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Weight per skill

Skill-It LEGO weights

Skill 1

Skill 2

Skill 3

Skill 4

Skill 5

0

1000

2000

3000

4000

5000

6000

Steps

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

Weight per skill

Skill-It addition weights

Skill 1

Skill 2

Skill 3

as the loss on the target skill approaches 0 . It is interesting to explore how to reduce edge weights and regularization so that the mixture approaches the target skill instead, although preliminary experiments where we decayed the edge weight and the strength of the Bregman divergence term did not appear better. We hypothesize that since training on a uniform mixture (as in Figure 3) did strictly better than training on the target skill and their loss curves did not intersect during the training run, it is better to allocate non-negligible weight on all skills throughout the training run.

The weight per skill across training steps for the Natural Instructions out-of-domain experiment corresponding to Figure 7 is shown in Figure 25, where the legend is provided for the top 10 task categories with the largest weights. While the initial weights based on the skills graph roughly establishes the order of weight magnitude, the differences among the losses on the evaluation skills increases the range of weights as training continues. As validation losses saturate, the weights also converge to fixed values.

0

1000

2000

3000

4000

5000

6000

Steps

0.24

0.26

0.28

0.30

0.32

0.34

0.36

0.38

Weight per skill

Skill-It LEGO weights for learning skill 3

Skill 1

Skill 2

Skill 3

0

100

200

300

400

500

600

Steps

0.15

0.20

0.25

0.30

0.35

Weight per skill

Skill-It weights for Spanish QG

English QA

Spanish QA

English QG

Spanish QG

0

100

200

300

400

500

600

Steps

0.425

0.450

0.475

0.500

0.525

0.550

0.575

Weight per skill

Skill-It weights for Stance Detection

Stance Detection

Text Matching

0

1000

2000

3000

4000

5000

Steps

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Weight per skill

Skill-It weights for Natural Instructions

question_generation

question_answering

text_categorization

sentiment_analysis

wrong_candidate_generation

text_matching

summarization

information_extraction

question_understanding

toxic_language_detection

E.4 Experiments on 1.3B parameter model

Wedemonstrate that the skills graph learned on the 125M parameter model, as described in section D.2 can be used for data selection with the GPT-Neo-1.3B model across all three evaluation settings. All results are reported over 3 random seeds.

Continual pre-training Wereport results for the LEGO continual pre-training experiment and the Natural Instructions continual pre-training experiment.

For the LEGO experiment, we use the skills graph learned on the 125M parameter model and train the 1.3B parameter model with SKILL-IT for 1500 steps with η =0 . 5 ,T =30 ,w =3 . In Figure 26, SKILL-IT still outperforms random and skill-stratified sampling on average. In particular, while performance across sampling methods is similar for early skills, the discrepancy is larger for skill 5 , for which SKILLIT allocates more weight to dynamically. In Figure 27, we provide the weight trajectories of SKILL-IT. Weobserve that the weight trajectories are similar to that on the 125M parameter model, where initial weight is allocated towards skill 2 . Later on, more weight is allocated towards skills 4 and 5 , whose losses are higher, and eventually the weight mixture converges to uniform as all losses converge to near 0 .

For the Natural Instructions experiment, we use the skills graph learned on the 125M parameter model and train the 1.3B parameter model with SKILL-IT for 5000 steps with η =0 . 2 , T =1 , and learning rate 1 e -6 . In Table 10, we report performance of SKILL-IT on the Natural Instructions skills and find that the trends from the smaller model hold-SKILL-IT outperforms random and skill-stratified sampling on average.

Fine-tuning Weuse the skills graph from the 125M parameter model to train the 1.3B parameter model with SKILL-IT for Spanish Question Generation fine-tuning. For 300 steps with learning rate 5 e -6 , η =0 . 8 , and T =12 , we find in Figure 28 that SKILL-IT outperforms both skill-stratified and random sampling.

Out-of-domain Weevaluate if the skills graph learned using the 125M parameter model can be used in SKILL-IT on a 1.3B parameter model for the out-of-domain setting on Natural Instructions. Our results are shown in Figure 29, where we trained for 1000 steps, learning rate 1 e -6 , η =0 . 5 , and T =2 .

0

500

1000

1500

10

5

10

4

10

3

10

2

10

1

Validation Loss (Log)

LEGO Skill 1

0

500

1000

1500

10

4

10

3

10

2

10

1

10 0

0

500

1000

1500

10

3

10

2

10

1

10 0

LEGO Skill 3

0

500

1000

1500

Steps

10

3

10

2

10

1

10 0

Validation Loss (Log)

LEGO Skill 4

0

500

1000

1500

Steps

10

3

10

2

10

1

10 0

LEGO Skill 5

0

500

1000

1500

Steps

10

3

10

2

10

1

10 0

Average per skill

Random

Skill-stratified

Skill-It

0

250

500

750

1000

1250

1500

Steps

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Weight per skill

Skill-It LEGO weights for 1.3B param model

Skill 1

Skill 2

Skill 3

Skill 4

Skill 5

Out of 12 evaluation skills, SKILL-IT outperforms random and skill-stratified sampling on 10 skills (compared to 11 skills when we used the same size model for both skills graph learning and training).

E.5 Ablations

Wereport ablations on components of the skills graph learning algorithm and on other components of SKILL-IT. First, we study how the performance of SKILL-IT is robust to changes in the skills graph-learning the graph with fewer steps, using the approximate algorithm, and adding noise to

50

100

150

200

250

300

Steps

1.8

1.9

2.0

2.1

2.2

Validation Loss

Spanish QG only

Skill-stratified

Skill-It

the adjacency matrix. Second, we study the effect of removing the skills graph, using a static skills mixture, and varying the η in SKILL-IT.

E.5.1 Skills Graph Ablations

We explore the effect of changing how the skills graph is learned and adding noise to the graph on SKILL-IT.

First, in Appendix C.2 we propose a brute-force graph learning algorithm and an approximate graph learning algorithm. We find that these algorithms exhibit little difference in performance. For the LEGOcontinual pre-training experiment, we find that replacing the brute-force skills graph with an

2.55

2.60

2.65

2.70

Validation Loss

Answerability Classification

1.80

1.85

1.90

Cause Effect Classification

2.65

2.70

2.75

Coreference Resolution

1.875

1.900

1.925

1.950

Data To Text

Random

Skill-stratified

Skill-It

2.00

2.05

2.10

Validation Loss

Dialogue Act Recognition

2.15

2.20

Grammar Error Correction

2.30

2.35

2.30

2.35

2.40

Overlap Extraction

0

500

1000

Steps

2.20

2.25

Validation Loss

Question Rewriting

0

500

1000

Steps

2.10

2.15

2.20

2.25

Textual Entailment

0

500

1000

Steps

2.50

2.55

Title Generation

0

500

1000

Steps

1.35

1.40

1.45

Word Analogy

approximate skills graph yields an average validation loss of 0 . 017 ± 0 . 005 compared to 0 . 021 ± 0 . 009 . For the Spanish question generation fine-tuning experiment, we find that using the brute-force algorithm yields a validation loss of 2 . 339 ± 0 . 028 compared to 2 . 328 ± 0 . 029 . In both experiments, there was no more than a 0 . 011 gap in validation loss from using one skills graph learning algorithm over the other, and both algorithms resulted in SKILL-IT outperforming baselines.

We next show that we can reduce H , the number of steps for learning each edge in the skills graph, without significantly compromising SKILL-IT's performance. In Table 11, we are able to reduce H by at least 50% without changing the downstream average validation loss of SKILL-IT by more than 0 . 01 . Results are shown for three experiments: LEGO continual pre-training, Spanish question generation fine-tuning, and Natural Instructions out-of-domain.

We also show that perturbations in the values of A do not significantly impact the performance of SKILL-IT. We learn each individual skills graph using one out of 5 random seeds. We then use each skills graph as input to SKILL-IT and measure the per-element variance in the A adjacency matrix as well as the variance in the average validation loss from using SKILL-IT. In Table 12, we see that the skills graph has O (1 e -3) variance while SKILL-IT's loss has O (1 e -4) variance. We further study the effect of perturbations by applying Gaussian noise to each element of A and measuring the performance of SKILL-IT (Figure 30). We find that SKILL-IT becomes worse than baselines only after perturbing A by a standard deviation of roughly 0 . 13 per element, which we never attain in Table 12.

Lastly, one may wonder if it is better to allocate the O ( Hk ) training steps needed for skills graph learning towards just training a model longer using a naive baseline that does not utilize skills. Let h be the re-

0.00

0.05

0.10

0.15

0.20

Standard deviation of Gaussian noise added to A

0.0

0.1

0.2

0.3

0.4

0.5

Average Validation Loss per Skill

LEGO with noisy skills graph

Random sampling

Skill-stratified

Skill-It (noisy graph)

0.00

0.05

0.10

0.15

0.20

Standard deviation of Gaussian noise added to A

2.34

2.36

2.38

2.40

2.42

2.44

2.46

Validation Loss

Performance on Spanish QG with noisy skills graph

Spanish QG only

Skill-stratified

Skill-It (noisy graph)

0.00

0.05

0.10

0.15

0.20

Standard deviation of Gaussian noise added to A

2.540

2.545

2.550

2.555

2.560

2.565

2.570

Average Validation Loss per Skill

NI OOD with noisy skills graph

Random sampling

Skill-stratified

Skill-It (noisy graph)

ducedoverall number of steps for skills graph learning from Table 11, and let n bethe number of steps for training the LM. We consider a random sampling baseline on n + h steps and compare it to SKILL-IT on n steps, with the skills graph learned over h steps. In Table 13, the random baseline does not outperform SKILL-IT, suggesting that the improvement in model performance outweighs the graph learning cost.

E.5.2 SKILL-IT ablations

We report ablations on removing the skills graph, using a static skills mixture, and varying the η in SKILL-IT.

First, instead of using A in Algorithm 1, we study the performance when the identity matrix is used instead; intuitively, this corresponds to a misspecified skills graph where no skill influences another skill. We refer this approach as 'No graph'. Note that the opposite case of a complete graph recovers skill-stratified sampling, which we already have as a baseline.

Second, instead of sampling over multiple rounds and weighting according to the loss of each skill, we study the effect of setting T =1 , which only uses a softmax on A to yield static weights on the skills. We refer to this approach as 'Static'. We omit results on Natural Instructions continual pre-training, since SKILL-IT uses T =1 and using no graph with T =1 recovers skill-stratified sampling. Intuitively, we expect the static version of SKILL-IT to perform somewhat well unless there is significant discrepancy among the losses (e.g. in synthetics where the loss on one skill can be close to 0 while the other is not, versus in Natural Instructions where all losses decrease consistently). For both ablations, we sweep over values of η =[0 . 1 , 0 . 2 , 0 . 5 , 0 . 8] .

Figure 31 shows the comparison between SKILL-IT and no graph on the continual pre-training LEGO experiment, and Figure 32 shows the comparison between SKILL-IT and a static approach. We see that both the graph and the online dynamics of SKILL-IT are important for its performance. In particular, using no graph results in allocating significant weight to harder skills early on, even though many of them have easier prerequisite skills (such as skill 3 having edges to skills 1 and 2). Using a static graph results in consistent allocation of significant weight to prerequisite skills even after their validation losses converge to near 0 , and thus the harder skills that have higher loss are not learned quickly afterwards.

Weperform the same ablation on the Addition dataset-the results for this are shown in Figures 33 and Figure 34. We find that these simple baselines, including using a static graph and no graph perform similarly to SKILL-IT on average across all skills-while SKILL-IT performs the best on skill 2 compared to vanilla multiplicative weights, and SKILL-IT performs the best on skill 1 compared to a static graph. This suggests that Addition is somewhat easier than the other datasets that we consider, as SKILL-IT still outperforms other baselines, as shown in Figure 4.

Figure 35 compares SKILL-IT, no graph, and static data selection for the LEGO fine-tuning experiment. Nograph can be interpreted as allocating equal weight to all training skills not equal to the target skill,

0

2000

4000

6000

10

4

10

3

10

2

10

1

10 0

Validation Loss (Log)

LEGO Skill 1

0

2000

4000

6000

10

2

10

1

10 0

0

2000

4000

6000

10

1

LEGO Skill 3

0

2000

4000

6000

Steps

10

2

10

1

Validation Loss (Log)

LEGO Skill 4

0

2000

4000

6000

Steps

10

1

LEGO Skill 5

0

2000

4000

6000

Steps

10

2

10

1

Average per skill

No graph (0.1)

No graph (0.2)

No graph (0.5)

No graph (0.8)

Skill-It

0

2000

4000

6000

10

4

10

3

10

2

10

1

10 0

Validation Loss (Log)

LEGO Skill 1

0

2000

4000

6000

10

2

10

1

0

2000

4000

6000

10

1

10 0

LEGO Skill 3

0

2000

4000

6000

Steps

10

2

10

1

10 0

Validation Loss (Log)

LEGO Skill 4

0

2000

4000

6000

Steps

10

1

LEGO Skill 5

0

2000

4000

6000

Steps

10

2

10

1

Average per skill

Static (0.1)

Static (0.2)

Static (0.5)

Static (0.8)

Skill-It

and varying this weight versus the weight on the target skill. While SKILL-IT and setting T =1 behave similarly, we see that SKILL-IT is slightly better than using no graph. For instance, SKILL-IT obtains a validation loss of 0 . 05 in 2000 steps, compared to 2050 -2200 steps when using no graph.

Figure 36 and 37 compare SKILL-IT, no graph, and static data selection for the Natural Instructions fine-tuning experiments. For both Spanish QG and stance detection, SKILL-IT attains lower loss than using no graph or using T =1 round.

0

2000

4000

6000

10

2

10

1

10 0

Validation Loss (Log)

0

2000

4000

6000

10

2

10

1

10 0

0

2000

4000

6000

Steps

10

2

10

1

10 0

Validation Loss (Log)

0

2000

4000

6000

Steps

10

2

10

1

10 0

Average per skill

No graph (0.1)

No graph (0.2)

No graph (0.5)

No graph (0.8)

Skill-It

0

2000

4000

6000

10

2

10

1

10 0

Validation Loss (Log)

0

2000

4000

6000

10

2

10

1

10 0

0

2000

4000

6000

Steps

10

2

10

1

10 0

Validation Loss (Log)

0

2000

4000

6000

Steps

10

2

10

1

10 0

Average per skill

Static (0.1)

Static (0.2)

Static (0.5)

Static (0.8)

Skill-It

Figure 38 compares SKILL-IT and static data selection for the Natural Instructions out-of-domain experiment. SKILL-IT attains the lowest validation loss on 7 out of 12 evaluation skills. It has an average loss of 2 . 540 compared to a range of 2 . 541 -2 . 551 for static data selection.

Lastly, we study the effect of varying η in SKILL-IT. We use η ∈ { 0 . 1 , 0 . 2 , 0 . 5 , 0 . 8 } . For LEGO continual pre-training and Spanish question generation fine-tuning (Figure 39), we find that SKILL-IT with varying η outperforms skill-stratified and random sampling (with the exception of η =0 . 1 and skill-stratified being roughly the same for Spanish question generation). On the Natural Instructions out-of-domain experiment, we find that for η =0 . 1 and 0 . 2 , we outperform baselines on 9 and 11 out of 12 evaluation skills, respectively, while increasing η =0 . 5 and 0 . 8 resulted in outperforming on 5 and 3 skills. These findings show that SKILL-IT is generally not hypersensitive to the choice of η , and we hope they provide some guidelines for a reasonable range of η without requiring significant tuning.

0

1000

2000

3000

4000

5000

6000

Steps

0.0

0.2

0.4

0.6

0.8

Validation Loss

Performance on Skill 3

No graph (0.1)

No graph (0.2)

No graph (0.5)

No graph (0.8)

Skill-It

0

1000

2000

3000

4000

5000

6000

Steps

0.0

0.2

0.4

0.6

0.8

Validation Loss

Performance on Skill 3

Static (0.1)

Static (0.2)

Static (0.5)

Static (0.8)

Skill-It

200

300

400

500

600

Steps

2.30

2.35

2.40

2.45

2.50

2.55

2.60

Validation Loss

Performance on Spanish QG

No graph (0.1)

No graph (0.2)

No graph (0.5)

No graph (0.8)

Skill-It

200

300

400

500

600

Steps

2.30

2.35

2.40

2.45

2.50

2.55

2.60

Validation Loss

Performance on Spanish QG

Static (0.1)

Static (0.2)

Static (0.5)

Static (0.8)

Skill-It

200

300

400

500

600

Steps

1.2

1.3

1.4

1.5

1.6

1.7

1.8

Validation Loss

Performance on stance detection

No graph (0.1)

No graph (0.2)

No graph (0.5)

No graph (0.8)

Skill-It

200

300

400

500

600

Steps

1.2

1.3

1.4

1.5

1.6

1.7

1.8

Validation Loss

Performance on stance detection

Static (0.1)

Static (0.2)

Static (0.5)

Static (0.8)

Skill-It

3.06

3.08

3.10

Validation Loss

Answerability Classification

Static (0.1)

Static (0.2)

Static (0.5)

Static (0.8)

Skill-It

2.08

2.10

2.12

Cause Effect Classification

3.10

3.12

3.14

3.16

Coreference Resolution

2.35

2.36

2.37

Data To Text

2.32

2.34

2.36

Validation Loss

Dialogue Act Recognition

2.38

2.40

2.42

Grammar Error Correction

2.74

2.76

2.78

2.74

2.76

2.78

2.80

Overlap Extraction

1000

2000

3000

4000

5000

Steps

2.59

2.60

2.61

2.62

Validation Loss

Question Rewriting

1000

2000

3000

4000

5000

Steps

2.48

2.50

2.52

Textual Entailment

1000

2000

3000

4000

5000

Steps

3.02

3.03

3.04

3.05

Title Generation

1000

2000

3000

4000

5000

Steps

1.67

1.68

1.69

1.70

Word Analogy

0

1000

2000

3000

4000

5000

6000

Steps

10

2

10

1

Validation Loss

LEGO average validation loss per skill

Random

Skill-stratified

=0.1

=0.2

=0.5 (Skill-It)

=0.8

200

300

400

500

600

Steps

2.30

2.35

2.40

2.45

2.50

2.55

2.60

Validation Loss

Performance on Spanish QG

Random

Skill-stratified

=0.1

=0.2

=0.5

=0.8 (Skill-It)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a way to introduce a curriculum of tokens to train LLMs. To do that, the paper introduces the concept of skills, i.e. behaviors that can be acquired by training the LLM in a subset of data, and ordered skill sets: partial orderings of skills that can be learned faster when using the skill preconditions. The paper shows that both in real and synthetic datasets, following the skill order allows for more efficient learning. Next, the paper presents techniques to sample data points from skills, given an ordered skill set. The apporach is evaluated on different datasets and model sizes, obtaining significant reductions in validation loss. Finally, the paper presents an approach to learn the skill ordering.\n\nSTRENGTHS:\n- Relevance and novelty: The paper addresses an important problem; given the computational cost of training LLMs, optimizing performance given a fixed compute budge is an important problem. It also presents a new framing to do so, that departs from curriculum learning approaches and focuses in selecting data to learn a set of skills. The approach to sample skills is also novel and well thought, dealing with both sampling data that both improve the weaker skills and selects the amount of data to sample from each skill.\n\n- Method: Starting from a reasonable optimization objective and constraint (reduce the validation loss on every skill at every round), the method derives a closed from sampling rule, which is fast and simple to compute. The proposed rule can be applied to multiple cases: 1) those where all the training skills have to be evaluated, 2) cases where only a subset are considered for eval and 3) cases where no data of the eval skill is provided during training. \n\n- Generally (see questions) clear paper writing, the definitions and discussion on how different kinds of graphs affect good sampling strategies provides great intuition for the rest of the paper. The walkthrough examples in section 3.2 also clarifies the intuition of skill curriculum.\n\n- While the graph learning approach is computationally expensive, it can be applied to subsets of the dataset, allowing to transfer the task dependency matrix to the full data for sampling. \n\n- Experimental results: The method is tested across different datasets and model sizes. The proposed approach improves upon different sampling baselines, including methods that sample uniformly across skills, by obtaining lower validation losses or comparable losses than the baselines in a shorter amount of time. The method also improves over baseliens on out of domain settings, where the test skills are not available during training.\n\nWEAKNESSES:\n- The method relies very strongly on having a task dependency graph. While the paper proposes two approaches to obtain these graphs, I would like further discussion into that, particularly on how to obtain the edge weights, and the robustness in the appraoch to errors or noise in this depnedncy graph.\n\nClarity on writing:\n- Definitions: In the definition of skill it would be good to clarify what the improved metric is with respect to. Is it compared to training from a random subset of X?\n- Similarly, it only becomes clear in Section 4.1 that the goal is to learn a subset of skills (which can include only 1 skill or the full skill training set). It would be good to clarify this better in the introduction.\n\nQUESTIONS:\n- On the definition skill, it would be good to clarify what the improved metric is with respect to. Is it compared to training from a random subset of X?\n- I understand from 4.4 how to learn the presence or absence of edges in the graph, but how do we find from there the weight of that edge? How sensible is the present algorithm to the edge weight.\n\nLIMITATIONS:\nNo limitation is mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

comments: SUMMARY:
This paper addresses the problem of how to order the training data for a LM.  The authors define the notion of a skill, show that skills can be (noisily) identified in training data, show that skills can be ordered, and show that sampling from the data defining a skill in an intelligent manner can improve pretraining and finetuning.  They provide both batch and online training algorithms.  They present moderately comprehensive experimental results demonstrating the effectiveness of the approach.

STRENGTHS:
I really liked this paper. I think it's a great piece of work, and deserves to be published.

+ The paper addresses an important and timely problem of great interest to a large community
+ The paper is well-written and polished
+ The paper strikes an excellent balance of notation, high-level ideas, and solid empirical work
+ The resulting algorithm seems easy to implement and empirically effective
+ I enjoyed the derivation of the Skill-IT algorithm - a tidy piece of work.  Well done.

WEAKNESSES:
- It's not clear exactly how skills can be effectively identified. Right now, the paper uses various forms of meta data, but this doesn't seem like quite the best (or most general) approach.  Maybe something based on the entropy of the text?

QUESTIONS:
None

LIMITATIONS:
There is no dedicated section to limitations.  I do not foresee any negative societal impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a new way of curriculum learning that is thematically similar to active learning: subsets of the data are broken up into \"skills\" and in cases where there is a dependency structure of skills, learning precursor skills can help improve performance. When the dependency structure is either complete or empty there is not huge motivation for such approaches (as they layout), but cases where there is a clear dependency structure, they find gains in both final model performance and learning speed. They test GPT-NEO on synthetic tasks that require variable tracking and addition, as well as Natural Instructions. They demonstrates these gains in both pre-training and fine-tuning settings.\n\nSTRENGTHS:\n0. This paper brings together some ideas on how to better train and fine-tune models on related tasks. They introduce a simple but clean (and flexible-enough) formalism of a skill that can be applied in an NLP setting.\n1. They demonstrate their method over 3 datasets.\n2. If one knows the graph of skills then it seems like this method will tend to be quite helpful in practice. (If not: see my concerns below!).\n3. I like Figure 1, making it a tiny bit bigger with bigger fonts would be awesome!\n\nI like this paper and I think it could be of practical use to people.\n\nWEAKNESSES:\n0. I'd prefer if more space were dedicated to talking about the adjacent matrix. It seems learning it is extremely expensive. How effective is the approximate approach? If we include the learning costs of the adjacency matrix (i.e., give all that extra training to uniform sampling), do the other approaches equal out? I think there are scenarios where we have a clear idea regarding the dependency structure and order.\n1. If the adjacency matrix is slightly wrong or noised, how does that impact the results of skill it? How consistent is the matrix over different random seeds / approximations?\n2. Continuing on, if there is a large quadratic operation to find the adjacency structure, where each cell basically asks, does learning task i help task j, is there any sense of which this paper is begging the question? I think not, because you're finding a practical and specific way of when and which tasks ia, ib, ic to sample to improve j.\n\nNit:\n1. I think the presentation of the LEGO example could possibly be made better, but I don't have great suggestions.\n2. The charts are not consistent in sizing. It is tough but doing so would improve presentation.\n\nQUESTIONS:\n0. I think this paper could be greatly improved with some deeper analysis in how the algorithm ends up actually working. Where and how does it differ from a straightforward curriculum? Even for a simple case like addition, are we mostly learning add1 and then add2 and then add3? Is it mixed? Do the proportions shift? Showing diagrams with the proportions + in comparison to random sampling and anti-curriculum could be illuminating. (I now see you have a motivation version of this in Figure 1. I'd really like to see the actual versions.\n1. Do you have a sense why Skill-stratified did worse across the board in Figure 6?\n2. How can Skill-it be better for Add-1? How is it different than random or curriculum? (To confirm: Those other cases are sampling across all the various Add-#s).\n3. Perhaps I missed this, but how does active-learning compare to anti-curriculum?\n\nLIMITATIONS:\nSeems good.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nReview Summary:\n\nThe paper explores the concept of clustering data into skills to establish a curriculum, aiming to increase the efficiency of language model (LM) training. The concept of a 'skill' is introduced as a unit of behavior with a corresponding data distribution. A skill, in this context, signifies that a model's performance improves on other data that assess a similar skill when it is trained on a dataset associated with that skill. \n\nAttempts to use embedding-based clustering for the unsupervised recovery of the curriculum did not yield desired results. However, loss-based clustering proved to be more successful, establishing that validation loss points from the same skill tend to follow similar trajectories. These trajectories were used to cluster by loss per timesteps per run. \n\nEquipped with this understanding, the authors suggest it is possible to generate an ordered skill set and extract a curriculum from it. The existence of such an ordered skill set in real data is demonstrated in the paper. \n\nThe paper also presents 'Skill-It', an approach to dynamically create training datasets, adjusting mixture weights based on expected improvements in generalization on the target task. However, the efficacy of this method is highly contingent on the accuracy of the transfer matrix $A$ approximation. \n\nThe method employed in the paper consists of two steps: \n1. Learning $A$ via brute force or linear approximation, and by approximating $A$ with a smaller model to reduce computational cost.\n2. Implementing 'Skill-It' to dynamically sample training data based on the evolution of losses.\n\nUsing synthetic and real data, the authors show that such ordered skill sets exist and can enable more data-efficient training. However, the paper also acknowledges that these ordered skill sets do not align with intuitive data groupings based on metadata and embedding clustering. Lastly, a novel online data sampling algorithm, SKILL-IT, is introduced for more efficient learning of skills. The authors demonstrate the effectiveness of their approach with improved accuracy in various training scenarios.\n\nSTRENGTHS:\n1. Theoretical Soundness: The paper presents a robust framework based on the innovative concept of 'skills'. The methodology, which encompasses learning an approximation of the transfer matrix $A$, skill-stratified sampling, and the implementation of 'Skill-It', appears to be well-constructed and theoretically sound.\n\n2. Clarity of Presentation: The writing in the paper is clear, making the complex concepts and methodology accessible and comprehensible. The authors effectively communicate their ideas.\n\n3. Potential Impact: If the proposed method is effectively evaluated and shown to significantly surpass existing baselines, this work could potentially make a substantial contribution to the field of language model training. The concepts of ordered skill sets and their application for more efficient, skill-based training could lead to improved performance of language models and open up new avenues for research in this area.\n\nWEAKNESSES:\nMethodological Issues:\n\n1. Overlooking Computational Requirements: As the primary goal of this paper revolves around reducing computational expense, it's essential to consider the additional compute requirements of the proposed methods (e.g., graph learning part) in the evaluation of the results. \n\n2. Emphasis on a Singular Experiment: The additional computational requirements could be offset by significant improvements in larger scale models, making the 125M to 1B experiment particularly important. However, this leaves only the results from Figure 7 / Table 8 and Table 9 as practically relevant for real-data experiments.\n\n3. Computational Cost and Hyperparameters: The methodology is not without its complexities, including a fixed computational cost to learn $A$ and the introduction of a new hyperparameter, $\\nabla$. These increase the complexity of the method, making it necessary to demonstrate a strong outperformance over the random baseline to justify their inclusion.\n\nExperimental Shortcomings:\n\n1. Unspecified Deviations: The authors should clarify whether the deviations reported in the tables are standard errors or standard deviations. Without this clarification, the statistical significance of the results remains uncertain.\n\n2. Missing Data in Section 5.1: The lack of standard deviations or standard errors in the main text necessitates referring to the appendix for assessing the significance of results. For the Addition task, the paper does not sufficiently address the observed equivalence between Skill-It, Curriculum, and Anti-Curriculum. In the case of Natural Instructions (real-data), it's uncertain whether Skill-It significantly outperforms random.\n\n3. Incomplete Reporting in Sections 5.2 and 5.3: The omission of standard errors or deviations in Section 5.2, along with the absence of a random baseline, makes it challenging to evaluate the results. Similarly, for Section 5.3 (Figure 6 / Table 7 and Figure 7 / Table 8), it's unclear whether the results are statistically significant due to the absence of reported standard errors or deviations.\n\n4. Need for Statistical Tests in E.4: As the E.4 experiment is likely the one where the initial cost of computing $A$ can be minimized, it's critical to determine if Skill-It is significantly better than skill-stratified sampling. This requires performing appropriate statistical tests with a sufficient number of trials.\n\n5. Absence of Hyperparameter Ablation for Skill-It: While there's an ablation study for $\\nabla$ in E.5, one for Skill-It is notably absent. The authors need to demonstrate that Skill-It isn't hypersensitive to its hyperparameters, or otherwise recognize the associated costs of hyperparameter tuning.\n\n6. Limited Comparison to Stronger Baselines: Even without being experts in curriculum learning, one may wonder why the proposed method isn't benchmarked against stronger baselines from this field. This comparison would be valuable in further establishing the strengths and weaknesses of the proposed method.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nThe limitations aren't discussed. \nI suggest some in the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}