Skill-it! A data-driven skills framework for understanding and training language models
Authors: Mayee F Chen
Nicholas Roberts
Kush Bhatia
Jue WANG
Ce Zhang
Frederic Sala
Christopher Re
Venue: NeurIPS 2023 spotlight (2023)
Keywords: language models
data selection
TL;DR: We examine if there exist ordered collections of skills in training data that are associated with how language models learn, and if this can be used to inform more a effective training data selection procedure.

Abstract: The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop a new framework based on a simple hypothesis: just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data. If such an order exists, it can be utilized for improved understanding of LMs and for data-efficient training. Using this intuition, our framework formalizes the notion of a skill and of an ordered set of skills in terms of the associated data. First, using both synthetic and real data, we demonstrate that these ordered skill sets exist, and that their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills. Second, using our proposed framework, we introduce an online data sampling algorithm, Skill-It, over mixtures of skills for both continual pre-training and fine-tuning regimes, where the objective is to efficiently learn multiple skills in the former and an individual skill in the latter. On the LEGO synthetic in the continual pre-training setting, Skill-It obtains 37.5 points higher accuracy than random sampling. On the Natural Instructions dataset in the fine-tuning setting, Skill-It reduces the validation loss on the target skill by 13.6% versus training on data associated with the target skill itself. 
We apply our skills framework on the RedPajama dataset to continually pre-train a 3B-parameter LM, achieving higher accuracy on the LM Evaluation Harness with 1B tokens than the baseline approach of sampling uniformly over data sources with 3B tokens.

Skill-it! A data-driven skills framework for understanding and training language models
Mayee F. Chen' Stanford University
Nicholas Roberts
Kush Bhatia Stanford University
Jue Wang
University of Wisconsin- -adiison
Together AI
Ce Zhang Together AI, University of Chicago
Frederic Sala
Christopher Ré Stanford University
University of Misconsin--Mddisee
Abstract
The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop new framework based on a simple hypothesis: just as humans acquire interdependent skills in deliberate order, language models also follow natural order when learning set of skills from their training data. If such an order exists, it can be utilized for improved understanding of LMs and for data-efficient training. Using this intuition, our framework formalizes the notion of skill and of an ordered set of skills terms of the associated data. First, using both synthetic and real data, we demonstrate that these ordered skill sets exist, and that their existence enables more advanced skills learned with less data when we train on their prerequisite skills. Second, using proposed framework, algorithm, SKILL-IT, over mixtures skills for both continual pre-training and fine-tuning regimes, where the objective is to efficiently learn multiple skills in the former and an individual skill in the latter. On the LEGO synthetic the continual pre- pre-training setting, SKILL-IF IT obtains .5 points higher accuracy than random sampling. On the Natural Instructions dataset in the fine-tuning setting, SKIIII.IIT reduces the validation loss the target skill by 13.6% versus training on data associated with the target skill itself. We apply our skills framework on the RedPajama dataset continually pre-train 3B- 3B-parameter LM, achieving higher accuracy on the LM Evaluation Harness with IB tokens than the baseline approach of sampling uniformly over data sources with 3B tokens.
introduce online data sampling
Introduction
Large language models (LMs) exhibit remarkable capabilities, including producing creative content [55], writing source code [9], and chatting with users [8]. A key ingredient enabling models to perform such tasks is the data on which the models are trained [18. 20, 59]. A natural way to unlock particular capabilities to improve this training data. However, it is unclear how to select data from a large corpus for these capabilities given fixed budget of training tokens, as data selection methods for current state-of- oate--of-the-arr LMs mostly rely on heuristics for filtering and mixing together different datasets [33, 59]. We lack formal framework for capturing how data influences the model's capabilities and how to utilize this data effectively for improving LM performance.
"Correspondence to mf then@cs.stanford edu.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

English QA
Spanish QG Validation Loss
Random sampling
Spanish
English QG
P
Skill-it
Spanish QG
Training
Data
Ordered skill set
Figure 1: Inspired by how humans acquire knowledge, we hypothesize that LMs best learn skills in particular order and that this can help improve our understanding and training of L.M.. We show that these ordered skill sets exist in real data, which enables skills to be learned with less data given that we train on their prerequisite skills. We then propose SKIIII-IT, online data selection algorithm that learns skills quickly by exploiting their ordering.
To develop such a framework, we take inspiration from how humans acquire knowledge. A classic idea in education literature is the concept of skills that form learning hierarchy [65]. For example, one study found that students learned mathematical and scientific skills most quickly when these skills presented in a particular order [12]. We seek to understand the extent that similar kiill-baedd orderings characterize LM training. Such orderings, if they cxist, may provide a better understanding of LMs as well as mechanism for data-efficien, training. For instance, to train LM for Spanish question generation, we wish to know if training first on related but simpler tasks, such as Spanish grammar and English question generation, helps.
We study the idea of skill orderings can help us build framework that relates data to LM training and behavior. This requires addressing two challenges revolving around the connection between skills and data. First, order to show that there exist sets of skills that the LM learns most efficiently in some par- ticular order, operational definition of LM skill and skill ordering must be developed and validated on data. In initial experiments, we investigated if semantic groupings of data, such as metadata attributes or embedding clusters, were sufficient to represent a skill and characterize how models learn. For instance, we partitioned the Alpaca datasct [56] by instruction type- a technique used to capture datasct diver- sity [62] but found that sampling based on instruction types and random sampling resulted in simi- lar model performance, suggesting that not just any existing notion of data groups can characterize skills.
Second, these definitions of skills must be used to construct sampling distributions to actually improve model training. To develop criteria for data selection algorithm that learns skills efficiently, we identify challenges that naive selection approaches face. The standard approach of random uniform sampling over data fails to learn skills optimally due to not accounting for skill imbalance and ordering. Skills can be distributed unevenly in the data, with more complex skills being rare- for instance, Spanish and question generation (QG) are 5% and 4% of the Natural Instructions dataset [63], respectively, but Spanish QG is only 0.2%. Random sampling also provides mechanism for taking into account a particular training order and dependency structure on skills. More sophisticated techniques like curriculum learning account for sample-level ordering, but not skills their dependencies. Our goal framework must account for these issues of imbalance and ordering.
Skill-based framework We define a skill unit of behavior that model can learn using an associated slice of data (Definition 1). An ordered skill sei is collection of skills with directed skills gruph that is neither complete nor empty, where an edge from prerequisite skill to skill exists if the amount of training it takes to learn the skill can be reduced the prerequisite skill is also Iearned (Definition 2, Figure I left, center). We show that ordered skill sets exist in synthetic and real datasets using this operational definition. Interestingly, the existence of these ordered skill sets unveils that one can learn a skill quickly not by training solely that skill, but on mixture of that skill and prerequisite skills. For instance, in Figure we observe that Spanish QG can be learned more efficiently when the model also learns English QG and Spanish-aa can achieve 4% lower validation loss than training on only Spanish QG over a fixed budget of overall training steps.
Next, given an ordered skill set to train on, we use our framework to propose methods for how to select data so that the LM learn skills faster: still-stratified sampling and an online generalization, SKILI.IIT. We address the issue of unevenly distributed skills in datasets by proposing skill-stratified sampling.

---

simple approach that allows us to explicitly optimize for learning skills by uniformly sampling relevant skills (such as a target skill and its prerequisite skills in fine-tuning). Skilll-tratified sampling uses the construction of the ordered skill set but is static, which does not incorporate the ordering as training proceeds and results oversampling skills that may be already learned early on in training. We address this issue by proposing an online data selection algorichm, SKILII-IT, for selecting mixtures of training skills that allocates more weight towards learning skills that are not yet learned or towards prerequisite influential skills (Figure I right). SKILL-IT is derived from online optimization problem over the training skills for minimizing loss on a set of evaluation skills given a fixed budget of data and the skills graph. SKILL-II IT is inspired by online mirror descent and can be adapted for continual pre-trae training, fine-tuning, or out-of of-domain evaluation depending on the relationship between the evaluation skill set and the training skill set.
We evaluate SKIIII-IT on synthetic and real datasets at two model scales, 125M and 1.3B parameters. For the continual pre-training setting, we show on the LEGO synthetic [72] that we obtain a 37.2 point improvement in accuracy over randomly selecting training data and a 9.7 point improvement over curriculum learning [4]. For the fine-tuning setting, we show that the widely-used Natural Instructions dataset [41, 64], our algorithm over a mixture of skills is able to achieve up to 13.6% lower loss on that skill than solely training that skill, given the same overall training budget. For the out-of- domain setting when our training skills do not align perfectly with evaluation skills, our algorithm is able to achieve the lowest loss 11 out of 12 evaluation skills corresponding to task categories in the Natural Instructions test tasks dataset over random and skill- stratista sampling on the training data. We finally apply our framework to a case study the recent RedPajama 1.2 trillion token dataset [57]. We use the data mixture produced by SKIILI-TT to continually pre-train a 3B parameter model. We find that SKIIL- IT achieves higher accuracy with 1B tokens than uniform sampling over data sources with 3B tokens.
2 Related work
An extended related work can be found in Appendix B. Existing work on data selection for LMs has generally ranged from more computationally expensive methods for dataset condensation on smaller datasets [47, 58, 48] to broader deduplication and filtering techniques for web-scale datasets [1, 33, 69]. Another way of improving model performance through choice of data via curriculum learning [4], which also draws inspiration from how humans learn and arranges data in order from easiest to hardest over samples or groups [60]. In contrast to existing works in both curriculum learning and data selection, our work focuses on selecting data for learning ordered set of skills more efficiently. How LMs learn also topic of growing interest; one framework posits that models learn over quanta, discrete units of computation [38], and another proposes that scaling laws for LMs can be understood in terms of learning combinations of skills [2]. Lastly, the notion of skill has been studied education, ranging from clas- sical research on learning hierarchies [66] to methods for decision making over lesson sequences [49].
3 Skills framework
First, we propose definitions of skills and ordered skill sets in order to formalize our intuition around how models learn skills, and we demonstrate that not just any existing notion of data groups can characterize an ordered skill set in the dataset. Then, demonstrate the existence of ordered skill sets on synthetic and real data, which show how viewing data through a basedd-based framework can help with training and understanding model performance. Finally, we explore unsupervised skill recovery from data, finding that embedding- based approaches do not adequately recover synthetic skills.
3.1 Definitions
We first present a definition of an individual skill. Let the input space of all possible text data be X, where x E X' is an individual text sample that a -eret-ookeen-eeiictionn LM f FX - X' is trained on. We quantify learning via a metric L: Fx X -R, which maps from a model and evaluation data to scalar quantity. In our setup, we use the cross-entropy validation loss applied over next-token predictions as our metric L.
Definition (Skill) A skill s is a unit of behavior with associated data X, S. X such that if f is trained on an dataset D, C X,, has improved metric I. aafterwardss on samples belonging to X, XD, on average.
This definition of a skill is flexible-it simply means that given a training dataset associated with the skill, a model f has an improved metric (c.g., decreasing validation loss) when evaluated on validation
3

---

Alpaca
Pile of Law
Natural Instructions
0.8
1.5
0.6
-1.0
1.0
0.4
0.5
0.2
0.0
0.0
0.0
Figure 2: Heatmaps of adjacency matrices we compute for skill graphs for Alpaca, Pile of Law, and Natural Instructions. Negative elements and diagonals are thresholded to 0 for clarity. See Appendix D.2 for descriptions of how they were constructed and larger versions.
data associated with this skill. Under this definition, skill could be granular task, such as Spanish question generation for subset of Wikipedia articles, or can be defined over a data source, such next- token prediction of legal data from tax court rulings. However, our next defimition, the ordered skill set, has a more specific construction and provides framework for how models learn across dependent skills.
Definition (Ordered skill set, skills graph) An ordered skill set for I is a collection of skills S =() (51 over which there a directed skills graph (8,,E)) on the skill set that is neither complete or empty, where (si,8)) E E if the amount of data needed to learn 8j when uniformly sampling from D., UDa, is no more than the amount of data needed when sampling only from Day- We equate learning skill 8j to f attaining certain value of L or lower average over Xs, \Day
This definition isolates complete and empty graphs as extrema that do not capture meaningful sets of skills. We discuss the three types of skill complet, empty, intermediate- and their implications for data selection. In particular, we discuss how several initial attempts defining skills over datasets via semantic groupings resulted in the extrema cases (see Appendix D. 9.2 for full results);
The complete graph demonstrates that all skills influence each other. A random partition is an example of skill set that yields a complete graph. This graph suggests that the best approach for learning any skill or set of skills is random sampling on the dataset. This is not setting where we can gain much with skill-based sampling. For example, using instruction types as skills the Alpaca dataset results in a nearly complete estimated skills graph (97.4% dense), and we find that stratified sampling on these skills only improves validation loss per skill by 0.007 points over random sampling on average (Figure 2 left), suggesting that utilizing skills does not improve model performance in this case.
The empty graph demonstrates that each skill is independent. This can occur if skills are too granular; for instance, learning Spanish math problems is unlikely to help with English poem generation. This graph suggests that the best approach for learning an individual skill is train on the skill itself. We see that empty graphs exist in real data; in Figure 2 (center), using data sources as skills on the Pile of Law [22] results nearly empty skills graph (3.9% dense).
Graphs that are neither empty nor complete thus suggest nontrivial order of how skill influence each other. This is the setting which we expect that identifying skills and exploiting their ordering will help the most. In Figure 2 right, use task categories, which capture broader reasoning patterns, as skills on Natural Instructions and find that the estimated graph has intermediate density (42.7% dense). We show concrete examples of how skills can be learned more efficiently Natural Instructions in Section 3.2.
While these intuitive groupings result in ordered skill sets on some datasets (e.g., task categories on NI), this is not always the case (e.g., instruction types on Alpaca and sources on Pile of Law). Even though these groupings capture some notion of diversity the dataset, our findings suggest that not just any semantic grouping induces an ordered skill set. We now empirically demonstrate that our definition of ordered skill sets aligns with how models learn and can be exploited for more fata-efficient training.
3.2 Examples of skills and ordered skill sets
We provide examples of ordered skill sets the LEGO synthetic dataset, an addition synthetic dataset, and subsets of the Natural Instructions dataset. On these datasets, we find that certain skills are better leamned when trained along with their prerequisite skills rather than in isolation.

---

Model performance on LEGO skill 3
Model performance on Addition skill
Model performance on Spanish QG
Irsited on set1
Teitsedon on difle 2,2,3
2.0
Trained on skills 1,2
Trained skill
Vaddetter
2.0
Trnited ot Spazia QG
Tinined or (Spenish Baglellx (0A 961
Wild L23
Vnluh
Wilddattmm Lo80
0.6
Model performmance on slance delection
0.4
2.0
Trinst on share desection
2.5
Trained on share devcction. led meehhnn
0.2
2.
0.0
1.5
1000
2000
3200
4200
5000
5000
1,000
20.00
3000
4000
5000
6000
100
500
420
500
Steps
Stogs
Stape
Figure 3: On the LEGO synthetic, 3-digit addition, and Natural Instructions, we identify examples of ordered skill sets which training on mixture of skills helps learn an individual skill faster than just training on that skill itself, given fixed training budget.
LEGO skills The LEGO synthetic, first introduced in [72], can evaluate a model's ability to follow a chain of reasoning. In this synthetic, the letters of the alphabet, A, are variables each with some binary label in {0,1}. An individual sample consists of k clauses for some fixed k across the dataset, each of the form a gar where a,z E A and g is either a negation ("not") or assertion ("val"), e.g. we assign a to the value of z, or we assign a to the opposite label. At the end of the sentence, we prompt the model for what the value of one of these variables is. Two samples x E X' are given below for k= 5:
Input: = not y, r= val 1, m val b, q= val m. y = not Output: b= 1. Input: c= val x, p val f, x val k, f= not k val O. Output: k= 0.
These samples each correspond to chain of reasoning; for instance the first sample has the chain r.y,b, b, m.,g. where knowing q's label requires the most reasoning steps. We define the ith skill 8; as the model's i ability to know the ith variable of the chain. From example above, the first sample belongs to Xz, and the second sample belongs to Xs, To demonstrate the existence of ordered skill sets, we continually pre-train the 125M parameter GPT-Neo model [14, 6] over various mixtures of LEGO skills with k = 5. In Figure (left), we find that in 35.9% fewer training steps, training on a balanced mixture of Xx ,2 and X., resulted in the same validation loss of 0.01 as training solely on Xeg. This suggests that $1.82 helped unlock performance on $3 and that there exist edges from 81 or $2 to 23 in the skill graph. Additional observations are available in Appendix 1. where we examine other edges as well as more complex reasoning chains, and the full skills graph corresponding to the ordered skill set for LEGO with k ==5 is in Figure 11.
We consider a variant of a synthetic 5-digit addition dataset analyzed in [45]. We show the existence of ordered skill sets for a simplified 3-digit addition dataset where we treat each digit predic- tion as a shill-the outputs, in this case, are the integers {0,1,..,99}. Examples are of the following form:
Addition skills
Input: A = 060001++1111111 A0=? Output: Input: A 879.0668999222 Output: 4
where 'A 0' refers to the ones digit of the output (81) and 'A 2' refers to the hundreds digit (sg). In Figure 3 (center), we find that in 32% fewer training steps, training on a balanced mixture of X., and Xx2 resulted in the same validation loss of 0.01 as training solely on Xx,- That is, the ones digit addition skill can be improved by simultaneously learning the tens digit addition skill, even though the former should not require information from the latter-this -his in line with observations from prior work that models do not always learn the ones digit addition first [45]. The full skills graph corresponding to the ordered skill set over 3-digit addition is in Figure 12.
Natural Instructions (NI) skills We show that ordered skill sets exist in NI [63] when we treat task categories skills.
In Figure (top right), we show that ordered skill sets exist over crosslingual task categories. Training on Spanish question generation (QG) along with equal parts of English QG, Spanish question answering (QA), and English QA results in 4.1% lower validation loss than training only on Spanish QG. Remarkably, the former only uses 25% of the latter's 's Spanish QG data. This suggests that there are edges from Spanish QA, English QA, and English QG to Spanish QG.
* In Figure 3 (bottom right), we see that training on the task category Text Matching along with Stance Detection helps decrease the loss on Stance Detection by 11.. This suggests that these categories, which both involve understanding the relationship between two input texts, share an edge.
5

---

Table 1: Summary of three settings continual pre- pre-training, fine-tuning, and out-of-domain. These settings are determined by how Seal is defined and result in different skills graphs used for our sampling methods.
Setting
Seal
Skills graph
Continual pre-training
Seal = Stain
A R & edges among all Suain
Fine-tuning
& edges from training skills to target subset Seatl SSmaia = Ac I xmm edges from training skills to evaluation skill set
Seall C Stain
E IR
Out-of-domain
The full skills graphs corresponding to the ordered skill sets over these task categories are in Figure 14. While equating task categories skills may be noisy, these examples suggest that there is signal within real data that suggests that ordered skill sets can improve data efficiency.
3.3 Skill recovery
A final component of characterizing skills is unsupervised recovery of ordered skill sets. We consider embedding-basee clustering approaches and a loss-based clustering approach for recovering LEGO skills. When clustering data using various trained and pre-trained embeddings, we find that they were unable to achieve above 39% accuracy on LEGO. Instead, we find that taking 10 random training runs and clustering data by their loss per timestep per run recovers the skills with 61% accuracy (Table 3). The intuition behind this method is that the validation losses on points from the same skill have similar trajectories as models learn. We discuss this approach more in Appendix E.2.
4 Skills-based data selection
Now that we have established the existence of ordered skill sets, we discuss how use them for data selection. We state the data selection problem for learning across skills in Section 4.1. We discuss how to learn the skills graph that will be exploited in our data selection methods in Section 4.2. We then introduce two sampling methods that utilize the graph, a simple skill-stratified sampling method and the online sampling method SKILL-ITE in Section 4.3.
4.1 Problem statement
We are given an ordered training skill set Strain = 1 J on the training data, each with associated support set 1
and ordered evaluation skill Seval 1 Seval m of m evaluation skills on a separate evaluation dataset. We aim to select n samples from Suain via mixture of training skills, P A 1-1, to achieve three goals depending on how Seval is constructed:
Continual pre-training: when Soval = Swain our goal is select mixture of training skills to learn
all of them.
Fine-tuning: when Seval C Suain. our goal select a mixture of training skills learn an individual target skill or subset of these skills.
Out-of-domaite when Seval n Strain 0, goal is to select mixture of training skills to learn a disjoint set of evaluation skills we cannot train on. This can arise when we have a separate downstream validation dataset or the skills identified in the training dataset are noisy.
Furthermore, we have skills graph G (Strin USeval, E), where E C Stain x Seval and A E kx is weighted adjacency submatrix, where A.y describes the strength of the edge from Stain, to Seval.j- In Table 1, we summarize how the three different settings are constructed and how A varies across them. Next, we discuss how A can be estimated from the data.
4.2 Skills graph learning
The skills graph is important for determining how to sample from the ordered skill set for training efficiently. We present two approaches for learning the skills graph graph-bbutt--orre brute-force and linear approximation. Algorithms are provided in Appendix C.2. By definition 2, the brute-force way of identifying edges involves fixing an overall training budget of H steps and 1) training and evaluating the model on each Si and 2) training the model on each pair of (Smmin,;; and evaluating on Strinn; and Seval.j. If the loss on Seval,j when trained on both S and Sevaljj is lower, there exists an edge from Strain, to Seval.j with edge weight proportional to the difference in loss. This approach has runtime ((kk)) and is only feasible when à is small and when we have access to Soval at training

---

Algorithm 1 SKILL-IT Online Data Selection Algorithm
1: Input: Ordered training skill set Strinn ordered evaluation skill set Seval- Learning rate 17. 7 rounds, n. samples, H training steps per run for graph learning, model f1, window parameter w.
2: A<- LEARNGGAAHHHHH (S S H, (Alg. 2, 3).
3: Initialize pj = cxp(1) Aiy) for all E [k), the softmax over A.
4: for fort=l...,
5: Observe losses I (f) for all Seval.j E Seval-
6: Train model fa with n samples from mixture Pt over Stain- Update model f++1=((JrPP).
7: Set pi+1 - -xp(pxxp = Ef=
Asg (
8:
end for
time. Otherwise, can approximate this approach in linear time by training on each for h < H steps and setting A. > if the loss on s, decreases over h steps for a runtime of ((k)). This linear approach is necessary in the out-of-domain setting. since it does not require training Seval- In addition, both graph learning approaches can be performed on a smaller model, and the learned graph can be used for data selection for training a larger model (Appendix E.4).
4.3 Skills graph-aware sampling
We present two approaches for sampling over the mixture of training skills according to the skills graph: skill-stratified sampling, which samples uniformly over relevant training skills according to A, and SKILL-IT, an online generalization that incorporates feedback of how skills are being learned so far.
4.3.1 Skill-stratified sampling
straightforward sampling approach is to discard training skills that do not benefit the evaluation skills and sample uniformly over the set of relevant training skills, which we call skill-stratified sampling. For continual pre-training, the relevant skills are the entire training skill set; for each Smmin, E Staiin Pr(Stranns)) = - This enables each skill to have sufficient training data. For fine-tuning, the relevant skills are the target skills and prerequisite skills, which can be identified via positive entries of the äth column of with Sprereq = (suain, Seval.j s.t. Aij >> 0}. We then set Pr(s) = [S for s Sprereg U For the -fomain setting, skill strat-stratified sampling is over the set of prerequisite skills. For each e Sprene. we set Pr(s)= Sppepa Next, we propose our online algorithm that exploits the graph dynamically for more efficient training.
4.3.2 SKILL-IT online data selection algorithm
Despite accounting for prerequisite skills, one shortcoming of skill-stratified sampling is that even if a skill has already obtained sufficiently low validation loss early during training, we will continue to allocate the same weight to that skill throughout training. Therefore, we formulate our data selection problem as an online learning problem and propose SKILL-IT, which both prioritizes prerequisite skills and skills that are not yet learned.
We are given a budget of T rounds and total samples train on. round we select mixture pt k-1 from the k-dimensional unit simplex, and for each training skill Smain, E Smain we sample from
X Smiss with proportion p for a total of + samples per round. Let fe be the model at at the start of round function of the previous round's model ft-1 and mixture Pt-1 via dynamics function $ F x A 6-1 - F; that is, ft = (fff 1-t--, )) Let T (f+) ) be the validation loss of fe Seval.j- Our goal is select Pr to minimize loss per evaluation skill the end of training:
t. can
can define fa recursively
minimize PL EA E
L
(1)
This optimization problem is challenging to solve without additional assumptions. In order to make the problem tractable, we impose an explicit dynamics rule for the each evaluation skill's loss Leval, in terms of the current loss and data mixture. Assuming for simplicity that Senal S Suain- a simple rule would be Trevat. (ft) = T (#(fe- :Pt- 1)) := T (eeaa,,(ee- (fe-1)(1 orp/1) for a E [0,1]. That is, we expect that allocating more data to skill; j should result in the validation loss on skill j decreasing. However, such
7

---

an expression assumes that only training on the jth skill will help learn the jth skill. Instead, Section 3.2 suggests that there are other skills that may help with the jth skill. We propose the following dynamies:
Leval (ft) ) Leval, (ft-1)(1-A)) Ptp--))
(2)
where A., is the column with weights of all skills that influence j.valj and we absorb the scalar a into A. The optimization problem in (1) can thus be simplified as follows:
minimize PT....PEEA-T m j=1
Leeval (Sr)
s.t ft = 1f -t= 1,..I L (fa) 1 (1- A.,
In Appendix C, derivee the following update rule via online mirror descent [46] for learning rate 77>
p1+1 px,ox
Leval (fa))
(4)
In addition, when equation 4 expanded, we have that pb+1 =p exp Since this summation over т results in diminishing strength of updates, we change it to a moving window of size w. Our full method is in Algorithm I.
L (f+))
Intuitively, at each step we adjust the weight on skill based on the losses of skills that influences, with the assumption that more training data helps decrease loss. Note that when we use our algorithm with a complete graph or empty graph, we achieve expected behavior discussed in Section 3.1. For the complete graph, our algorithm reduces to stratified sampling. When we have skill set with an empty graph, the update rule reduces to sampling proportional to each skill's validation loss.
Experimental results
Given ordered skill set, aim validate ability ability select data for efficiently learning skills in the continual pre-training, fine-tuning, and out-of-domanl settings. We provide full tables of results in Appendix E.3.1 and results where learn the skills graph the 125M model and use it for the 1.3B parameter model in Appendix E.4. Skills graphs are in Appendix D.2, weight trajectories for SKILL-IT are in Appendix E.3.2, and ablations on the graph and online components of SKILL-IT are in Appendix E.5.
5.1 Continual pre-training
Setup We evaluate the ability of SKILL-IT to select data for efficiently learning over all skills. We measure average validation loss per skill after fixed number of training steps. We construct the LEGO synthetic and addition synthetic with i = and 3, respectively, and an imbalanced dataset over the skills. On the Natural Instructions dataset, we use 23 of the task categories as skills.
Baselines We compare SKILL-IT against three baselines that do not account for skills: random sampling, curriculum learning, and anticurriculum learning. Random sampling is a standard procedure for selecting samples given no additional information. Curriculum learning [4] and anticurriculum learning [67] score the samples from casiest to hardest and vice versa, respectively, and sample over expanding set of the lowest scored samples at every epoch; we use the pre- pre-trained model's 's loss to rank points. We evaluate skill-stratffied sampling, which uses knowledge of the skills but not online, and include an additional skills curriculum and anticurriculum baseline from [60], which samples from skills in order of average loss per skill.
Across our experiments we find that SKILL-IT outperforms baselines that do not use skills as well as skill-stratified sampling and skill curriculum learning. Our results on the LBGO dataset are shown in Figure 4. SKILL-ITT and skill-stratified sampling attain lower loss than other approaches on skills 2, 3, and on average, and while curriculum and anticurriculum learning attain lower loss on skill 5, they fail to learn other skills, resulting in at most 68 points accuracy on skill 3. SKILL-IT reaches
Analysis

---

LEGO SKIB
LEGO Stall
LEGO Sxill
141
ValDatten L405 (L.g)
1014
2000 4030
5050
2000
4000
19700)
2000
4003
0100
LEGO SkLE
LECO Skill 5
Average per skill
Valiaaiion Lane (Log)
7.20
200
2014
2480 Baller
E10
2300
4030
70100
2000 $0.00
2000
4003
Strps
Stope
Strpe
Figure 4: Performance of IK on each skill in the continual pre-training setting (learning over
all skills in the ordered training skill set) the LEGO synthetic.
Additioo Skill
Addition
Acidition
Average per
Ueidbatt Lane (t.ag)
Ia!
701
107
100
LG-1
10
KLl Scoiii
00
1oml
Stopa
Stopp
Stepa
Figure 5: Performance of SKILL-IT IT in the continual pre-training setting on the addition synthetic.
high average accuracy earlier in training than other approaches; halfway through training, SKILL-IIT obtains between 8.3 and 33.5 points higher average accuracy than other approaches, reaching final accuracy of 99.3 (Figure 20). SKILL-IT initially allocates more weight to prerequisite skills such as skill 2 as suggested by Figure 11 and later on allocates more weights to the skills that are learned more slowly, such as skills 4 and 5 (Figure 22). On the addition synthetic with k= 3, SKILL-IT obtains lower validation loss than the baselines on skills and 2 in Figure 5. While most approaches aside from skill curriculum learning eventually obtain 100% accuracy on all skills, SKILL-IT requires less training to reach sufficiently high accuracy; halfway through training. SKILL-IT has accuracy between 8.7 and 73.1 1 points higher than other approaches (Figure 21). Finally on Natural Instructions, the average validation loss from IK is 3.2 5% lower than from random sampling (Table 7). Our results suggest that exploiting the construction and ordering of skills is critical to learning skills quickly.
5.2 Fine-tuning
Performance on Skill
Performancc on Skill
Performaacc on Spazish QG
Performancce stance detection
Waldsson Lea
075
Still only Stall stratidedd
WWirrbstom Loos
Skall onl
Valdaatio Lawn
Skill atr stiied
Spankh 0G only
VallSason Laks
0.50
Skell-it
Still-2:
Skell-tt
0.25
0.00
21000
4000
6:000
2,000
4000
6000
2.3
230
400
Stags
Stagss
Steps
Stept
Figure 6: Performance of SKILL-IT in the fine-tuning setting on LEGO, addition, and NI. Setup We evaluate the ability of IKILL-IIT select data from ordered training skill set for learning a target skill. Mirroring Figure 3, we evaluate on LEGO target skill 3 (third in reasoning chain), on the addition synthetic's skill 1 (ones place digit addition), and on NI's Spanish QG and Stance Detection.
Baselines We compare SKILL-IT against training on the target skill only and skill-stratified sampling over prerequisite skills and the target skill. The skill-stratified sampling approach uses the ordered skill set to identify prerequisite skills, but does not exploit them dynamically.
Analysis Our results are shown in Figure 6. On LEGO, SKILL-IT results in the same validation loss of 0.01 as training only on the target skill in 38. 388.1% 1% fewer steps. observe a similar trend on addition, with SKILLI-IT converging to validation loss of 0.01 in 59% fewer steps required to do so when training only on the target skill. Finally, on NI, SKILL-ITT improves validation loss on Spanish question generation by 5.3% and Stance Detection by 13.6% over training the respective target skill only. In this setting,

---

Answerabllity Clasitfication
Cause Effect Classificatio
Coreforeace Resolution
Data Text
2.54
110-
108
2:50
2.12
3,30
3.16
2.30
2.37
Randon
206
3.12
3.30
2.36
Dialogue Recomuttion
Grammar Correction
Reyword Tagçing
Oveclap Extraction
-38-
2.44
236-
2.42
1.76
1.78
234
2.70
3.40
Question Rowriting
Twstunl Fntellment
Title Generation
Word Analagy
3.70
2.42
1.59
2.52
3.04
1.00
2.50
2.48
3.02
1.67
1000
2300
3000
4000
5000
1000
2000
3000
4002
5000
1002
2000
2030
1000
5003
1000
2003
3000
4030
5000
Stape
Stape
Strps
Strie
Figure 7: Performance of SKILL-IT in the out-of-domain setting for the NI task split. SKILL-IT uses the graph between the train and evaluation skills to produce an online mixture on the training dataset.
0.604
RedPajana source
SKILL-1 mixture
Accuracy
0.642
ArXiv
Books
0.0437
0.1370
0.4195
0.640
Uniform Skill I:
CommonCCrawl
0.0732
GitHuh 0.1890
0.638
StackExchange
0.0892
Number of tokens (billion)
Wikipedia 0.0484
Figure 8: Left: Accuracy on LM Evaluation Harness for continual pre-training of a 3B parameter model using SKILL-IT on the RedPajama dataset. We achieve higher accuracy at 1B additional tokens than uniform at 3B tokens. Right: SKILL- mixture over RedPajama sources.
a significant portion of the improvement over training only on the target skill comes from identification of prerequisite skills through the learned graph in the skill-stratified sampling method. SKILL-IT is further able to improve performance with finer mner-grained dynamic weighting on prerequisite skills.
5.3 Out-of-domain setting
Natural Instructions We evaluate the ability of SKILIIIIT select data from set of training skills for learning disjoint set of evaluation skills that we cannot train on. We use all 59 task categories in the NI train tasks split as the training skills and the 12 task categories in the tasks split our evaluation skills. We compare SKILL-IT against random and skill-stratified sampling. both of which do not exploit the relationships between training skills and evaluation skills. SKILL-IT achieves the lowest loss on 11 out of 12 task categories over random and skill-stratified sampling (Figure 7, tables in Appendix).
RedPajama We SKILL-IT to produce data mixture the RedPajama dataset. The training skills are the data sources comprising the dataset, and the evaluation skills are several tasks from the Language Model Evaluation Harness [15]. SKIIII-IIT with T yields the mixture in Figure (right). We continually pre-train a 3B parameter model trained on IT tokens for 3B additional tokens using this mixture, and see that outperforms uniform sampling over the data sources (Figure 8 left), In particular, SKILL-IT achieves higher accuracy with IB additional tokens than uniform with 3B additional tokens.
Conclusion
Given a fixed budget of data, knowing what data to train on to induce various capabilities an LM is challenging. As LMs continue to improve, it will become increasingly important to extract as much signal as possible from the data and to direct that signal towards acquiring a broad variety of capabilities. In this paper, we introduce skills- framework for understanding how LMs learn and for selecting training data. We hope our study invites others build on such a notion of skill and further explore how align skills with data.
10

---

Acknowledgements
We thank Together AT compute used to train models in this paper. We thank Sabri Eyuboglu, Karan Goel, Arjun Desai, Neel Guha, Michael Zhang, Vishnu Sarrukai, Simran Arora, Ben Spector, Brandon Yang. Gautam Machiraju, and Sang Michael Xie for their helpful feedback and discussion.
ai/) and Stanford HAT for providing portions of the
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity). CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under No. I91-NF-21-2-0251 (Interactive Human-AI Teaming): ONR under No. N000141712266 (Unifying Weak Supervision); ONR NO00 1-2480: Understanding and Applying Non-Euclidean Geometry Machine Learning: N00014201 42011225557 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASE, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. FS supported by NSF CCF2106700 and the Wisconsin Alumni Research Foundation (WARF). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
References
[1] Amro Abbas, Kushal Tirumala, Dánicl Simig, Surya Ganguli, and Ari Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023.
[2] Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models, 2023.
[3] Yuntao Bai, Andy Jones, et al. Training a helpful and harmless assistant with reinforcement learning from human feedhack, 2022.
[4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international confereee machine learning, pages 41-48, 2009.
[5] Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallaban, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling, 2023.
[6] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Autoregressive Language Modeling with Mesh- Tensorfon,r,,w,, March 2021. If you use this software, please cite it using these metadata.
Large Scale
[7] Rishi Bommasani, Percy Liang, et al. On the opportunities and risks of foundation models, 2021.
[8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[9] Mark Chen, Jerry Tworek, et al. Evaluating large language models trained code, 2021.
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[11] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, 2019.
[12] Robert M Gagne. The acquisition of knowledge. Psychological review, 69(4)3355, 1962.

---

[13] Robert M Gagne and Noel E Paradise. Abilities and learning sets in knowledge acquisition. Psychological Monographs: General and Applied, 75(14):1, 1961.
[14] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv: 200.:::0001.0022.. 2020.
[15] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang. Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, September 2021.
[16]
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Picter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023.
[17] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference Machine Learning, pages 2242-2251. PMLR, 2019.
[18] Google. Palm2 technical report. Technical report, 2023.
[19] Anupam Gupta. Advanced algorithms: Notes for emu 15-850 (fall 2020), 2020.
[20] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't it stop pretraining: Adapt language models domains and tasks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
[21] Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. Sim- fluence: Modeling the influence of individual training examples by simulating training runs, 2023.
[22] Peter Henderson Mark S. Krass Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and 256gb open-source legal dataset, 2022.
[23] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Haffield-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability of learning from repeated data, 2022.
[24] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data, 2022.
[25] Joongwon Kim, Akari Asai, Gabriel Ilharco, and Hannaneh Hajishirzi. Taskweb: Selecting better source tasks for multi-task nlp, 2023.
[26] Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 2611-26244 Curran Associates, Inc., 2021.
[27] Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self-attention and pre-training In Proceedings of the 57th Anmual Meeting the Association for Computational Linguistics, pages 3499- 3505, Florence, Italy, July 2019. Association for Computational Linguistics.
[28] Nikita Kitaev and Dan Klein. Constituency parsing with attentit encoder. In Proceedings of the 56th Annual Meeting the Association for Computational Linguistics ( (Volume 1: Long Pupers), pages 2676-2686, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[29] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, 2017.
12

---

[30] Po-Nien Kung, Sheng-Siang Yin, Yi-Cheng Chen, Tse-Hsuan Yang. and Yun-Nung Chen. Efficient multi-task auxiliary learning: Selecting auxiliary data by feature similarity. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 416-428, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[31] Michael Langberg and Leonard J. Schulman. Universal approximators for integrals, pages 598-607.
[32] Hugo Laurençon, Lucile Saulnier, al. The bigscience roots corpus: A 1.6tb composite multilingual dataset, 2023.
[33] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burchh and Nicholas Carlini. Deduplicating training data makes language models better. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
[34] David D Lewis. sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pages 13-19. ACM New York, NY, USA, 1995.
[35] Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards mechanistic understanding, 2023.
[36] Paul Pu Liang, Chiyu Wu, Mouis-Philippy Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6565-6576 PMLR, 18-24 Jul 2021.
[37] Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners, 2022.
[38] Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, 2023.
[39] Sören Mindermann, Muhammed Razzak, Winnie Xu, Andreas Kirsch, Mrinank Sharma, Adrien Morisot, Aidan N. Gomez, Sebastian Farquhar, Jan Brauner, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learned (workshop version), 2021.
[40] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovee. Coresets for data-efficient training of machine learning models, 2019.
[41] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. generalization via natural language crowdsoureing instructions. In AnAC 2022.
Cross-task
[42] Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers, pages 220-224, Uppsala, Sweden, July 2010. Association for Computational Linguistics.
[43] MosaicMI.. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023.
[44] Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1t International Joint Conference on Natural Language Processing (V (Volume 1: Long Papers), 2021.
[45] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilityy. In The Eleventh International Conference on Learning Representations, 2023.
[46] Arkadij Semenović Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.
[47] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training, 2021.
13

---

[48] Jeff M. Phillips. Coresets and sketches, 2016.
[49] Siddharth Reddy, Igor Labutov, and Thorsten Joachims. Latent skill embedding for personalized lesson sequence recommendation, 2016.
[50] Sebastian Ruder, Parsa Ghaffari, and John G. Breslin, Data selection strategies for multi-domain sentiment analysis, 2017.
[51] Sebastian Ruder and Barbara Plank. Learning to select data for transfer learning with bayesian optimization. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
[52] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: A new family of parameters for learning a differentiable curriculum. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus Buc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[53] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning, 2022.
[54] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nieu Sebe. Curriculum learning: A survey. International Journal of Computer Vision, 13066))552--155, Apr 2022.
[55] Claire Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, and Han van der Maas. Putting gpt-3's creativity to the (alternative uses) test, 2022.
[56] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instructionffolloiing llama model. https://gg gi thub. com/ com/tatsu taastaatsalll labl ttabf//tafooo ordooardlllll allaca, 2023.
[57] Together. Redpajama-date An open source recipe to reproduce Ilama training dataset, 2023.
[58] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning, 2018
[59] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[60] Neeraj Varshney, Swaroop Mishra, and Chitta Baral. Let the model decide its curriculum for multitask learning, 2022.
[61] Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, Songyang Gao, Junzhe Wang. Menghan Zhang, Xiang Gao, Yunwen Chen, and Tao Gui. Farewell aimless large-scale pretraining: Influential subset selection for language model, 2023.
[62] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022.
[63] Yizhong Wang, Swaroop Mishra, et al. Suppernaturrlinsstuuctoosss Generalization via declarative instructions on 1600+ nip tasks, 2022.
[64] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2021.
[65] Richard T White. Research into learning hierarchies. Review of Educational Research, 4313);36 1973.
[66] Richard T. White and Robert M. Gagné. Past and future research on learning hierarchies. Educational Psychologist, 11(1):19-2, 1974.
[67] Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work?, 2020.
14

---

[68] Sang Michael Xie, Hieu Pham. Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Perey Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining, 2023.
[69] Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling, 2023.
[70] Amir R. Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, Jun 2018.
[71] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
[72] Yi Zhang, Arturs Backurs, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, and Tal Wagner. Unveiling transformers with lego: a synthetic reasoning task, 2022.
15

---

A Broader Impacts and Limitations
Broader Impacts As more LMs are developed, a key criteria for their adoption and utility is if they exhibit a wide array of useful capabilities, such as generating harmless content, summarizing essays, and being conversational with the user. While improvements in other parts of the LM development pipeline such as training and architecture are important, many recent advances in building LMs with a wide array of useful capabilities have come from the data itself [18. 56, 10, 16, 43 Our work is fundamental in investigating how LMs learn and how to select data to learn skills more efficiently. However, we recognize that data selection methods can always be utilized optimize for particular skills that may be considered malicious or negatively target or exclude specific groups [3]. Furthermore, pre-trained LMSs have been found to have various biases [26, 44, 36, 7].
Limitations The skills graph can either be provided (e.g., using a knowledge graph) or learned. Our work learns the skills graph using Algorithm or Algorithm 3, which requires initial training runs on pairs of skills or each skill, respectively. This can be made more efficient by performing these training runs on smaller model and for fewer number of steps, but tradeoff's here have yet be thoroughly investigated. SKILL-IT also assumes that the ordered skill set is provided; as discussed in sections 3. and 3.3, it is challenging to recover ordered skill sets simply via metadata attributes or embedding clustering. Otherwise, the best way to sample over collections of skills that form complete or empty graph is random or stratified sampling with no ordering to exploit. Our loss-based clustering approach presented in section 3.3 demonstrates that grouping by losses can provide an explanation for how skills are defined over data. An important direction for future work is to use such a clustering approach or other unsupervised algorithms in an end-to-end pipeline for skill discovery, skill graph learning, and data selection based on such skills.
Code release The code for SKILL-IT is available at https: com/Hazzyeeeeaarchh skill-it.
B Extended related work
Data selection for LMs There have been several studies of large-scale data selection for LMs. Data deduplication [33, 1. 23], in which identical or nearly identical samples are removed, is a method that enables LMs to be trained on smaller, cleaned datasets and has been increasingly used pre-processing step for training data [59, 5, 71]. Other methods applied at scale involve ensuring high quality of data by explicitly filtering out samples comparing the training dataset with cleaned reference dataset [8. 59, 32]. Importance reweighting approaches have also been proposed for identifying training data from large corpus that best approximates smaller target distribution [69], and influence functions have been used to select a subset of training data to improve performance on downstream tasks [61]. These approaches can identify data pertaining a particular target distribution or filter out low quality data according to some heuristic, while our work aims to understand how the choice of data is related to the numerous skills that LMs learn.
Recent development LMs has shifted focus from emphasizing the scale of the model to prioritizing the training data utilized. For example, models like Alpaca [56]. Vicuna [10], and Koala [16] are all based on the LLaMA model combined with instruction data generated by an existing LMM Palm 2's technical report states that the data mixture was a critical component of the final model [18], and Mosaic ML's recent MPT model was trained on a haan-enginneeeed mixture of the RedPajama dataset [43]. However, these works lack rigorous explanation for why their training datasets were constructed in this way.
Finally, perhaps most related to our approach the contemporary work DoReMi [68], which uses group distributionally robust optimization on a smaller LM to select data source mixtures for training a larger LMM Their approach focuses on selecting data at the data source level for optimizing worst-case performance across the training data sources, rather than at the more general skills level for variety of target skill sets. Furthermore, we focus on understanding how skills are related to each other and induce some order in how LMs learn by explicitly modeling skill graph structure, which we find to be important for data-efficient LM training (see ablations in Appendix E.5).
Data selection methods Many data selection methods have been proposed for supervised, task- specific settings. In this setting, the most typical objective is dataset condensation, which aims to identify a small subset of data that captures the larger dataset's 's properties with respect to the model.
16

---

Some approaches include constructing coresets [31, 48], identifying samples that the model forgets during training [58]; identifying samples with the largest gradients [47] or gradients that approximate the overall gradient [40]; clustering in embedding space and selecting points farthest from cluster cen- ters [53]; and selecting samples with the highest uncertainty or entropy [34]. These approaches have also been shown to transfer from smaller models to larger models [11]. Unlike these methods, we study how to select data for learning one or many skills at the mixture level for LMs instead of the instance level.
Another area of interest is data selection for domain adaptation and multitask learning. For domain adaptation, there are a wide range of methods that select data to best match the target distribution. For example, the Moore-Lewis method matches data based on the difference in cross- cross-entropy using a model trained on the target versus model trained on the source data [42]. Several other approaches suggest training a model to distinguish between source and target and selecting points with high uncertainty [50], or selecting points based on some divergence in an embedding space [51]. In comparison to these approaches, our work focuses on learning one or many skills and also finds that embedding- based heuristics do not fully identify skills.
Data attribution Another perspective on understanding training data is data attribution, which seeks to identify what data is responsible for particular model behaviors. Influence functions [29] and shapley values [17] are two ways to quantify the role of individual samples. Datamodels [24] fit model to predict behavior given subset of training data, providing framework for understanding individual samples as well as dataset counterfactuals. Simfluence [21] fits a Markov process to set of training trajectories for finer-graines understanding of how data impacts training. We focus on understanding how groups of data associated with skills elicit broader model capabilities, and utilize this understanding to select data for more efficient training.
Curriculum learning Curriculum learning [4] proposes to show the model data in order from easy samples to hard ones. Various criteria have been used to determine hardness, and anticurriculum as well as various pacing functions and mixing rates have been explored [54]. Curriculum learning can also be performed at the group level [60]. More sophisticated approaches include parametrizing each sample with a dynamic importance [52], and also accounting for irrelevant and noisy data [39]. Our approach similarly utilizes a curriculum, but it is defined over skills graph and does not necessarily align with training on easiest to hardest skills.
How LMs learn Many different explanations for how LMs learn from data have been proposed. One hypothesis is that there exist discrete, universal building blocks of LM knowledge called quanta, and power law scaling emerges from a learning over a particular distribution of quanta in the right order [38]. Another work proposes that scaling laws for LMs can be understood in terms of learning combinations of skills [2]. Others have provided theoretical analysis of how transformers learn topics by studying co-occurrences of words in the training data [35]. Empirically, how models learn is still mystery-for instance, models trained on code are found to perform fairly well at commensense reasoning [37]. Our work initiates a study how LMs learn various skills and how to exploit this for better data selection.
Task selection In multitask auxiliary learning, the goal is to train model to perform well on target task(s) by selecting the most beneficial source tasks to train on. One can use feature similarity to select tasks [30], but we find in our synthetics that feature similarity does not always recover skills. In Taskonomy [70], a hypergraph over a set of tasks is learned and used to select tasks. The methods used to develop the taxonomy can be applied to further expand our graph learning (e.g.. studying transitive and higher-order properties). However, their focus is on task selection in computer vision rather than data selection for LMs to learn skills. Lastly, the contemporary work of Task Web [25] builds graph among 22 common NLP tasks in order to determine what the best source tasks are for a target task. Their definition of an edge in the task graph is less strict than ours (their comparison is on if training on additional data from s; helps with sj. while we fix the overall amount of training data over both s; and 8j). Overall, our approach is similar in use of the skills graph, but we incorporate it into dynamic sampling algorithm. Furthermore, we look more broadly at skills, rather than tasks, and characterize when we expect using the skills graph to improve model performance.
Education The notion of skill has been studied in education. Classical research on learning hierarchies [66] identify sets of skills that make up subordinate capabilities for students. For instance, [13] identified that in order for students to solve linear equations, there were many prerequisite
17

---

skills, ranging from the simplest being symbol recognition to the most complex being the ability to add, subtract, multiple, and divide from both sides of the equation. More recently, Decision-making over lesson sequences based on skills, e.g., what the student already knows versus what the lesson teaches, has become an area of interest in personalized learning [49].
C
Additional Algorithmic Details
C.1 Derivation of SKILL-IT Update Rule
First, provide the derivation of our update rule from online mirror descent using the proximal point view [19]. We restate our optimization problem from (3):
minimize 1
Lovall (Jr)
(5)
Pi E 3=1
s.t Leval, (ft) -o II.... fo = ft-1 1t Vt=1....T
Let [,())= m " L (f++)) 1. S. ( (f(f=-))); that is, p is the mixture must choose at time and L, is the average loss per skill of the model after is trained on at round A greedy approximation of (5) is minimize L((p), given the model and mixtures at previous rounds. A linear pEAk-1
in
approximation of (p) is
PC-1)
(6)
Then, the problem of minimizing (p) becomes
argmin (
after we drop terms from (6) that do not depend on Note that the is constant and does not impact the solution. The optimal solution this problem is selecting the p that has the most weight on the slice with the largest gradient. To improve stability and prevent overfitting, we introduce regularization via a Bregman divergence D2 1 h(p) h(pt-1) (7h(pt-1).P - terms that do not contain our problem is now
Pt-1). After dropping
argminp==4 - (1) L, (p. .).)) +h(p) (b))) (V/(p)))))))
(8)
Taking the gradient and setting it equal 0 gives us
// L (p= (h(
(9)
Similar to in standard multiplicative weights, we set h(p) E,p, and Vh(p) [Inp, + +1]. Then,
Inp' = In -
+ P141 - =p Lv
(10)
where Vi is the ith element of the gradient. Now we wish to compute v,L.(p.) د Ve(Lenlls(fee1) Eド ( (
Recall the dynamics model for Levati
T (f++1) = Leval,5 (f() A.)))),
(11)
The gradient of this model with respect to each training skill is
V.Levat,(fe+)) - Leval 5 (ffe)
(12)
1
(p,) =
Ais Leval (ff)
j=1
Plugging this back into (10),
pi = =
Agy Leval. (fe))
(13)
where absorb the 1 into 77.
18

---

Algorithm 2 Brute-Force Graph Learning
Input: Ordered skill set S = ($1 [s1,..,k} Number of training steps K, base model fo-
2: for j [k]] do
3: Train fo on samples from Xz, for steps and denote fk.j to be the model after training.
4: Observe change in loss, 63 = L Lf
5: end for
6:
for i.j E
[k]] do
7:
Train fo on samples from Xx, UX2, for K steps and denote [KÚJj to be the model after training.
8:
Observe change in loss, 8/2² L Leval (fK,s.).
9:
if 5 > 83 then
10: Draw edge 8; 8 and set Aij 00
end if
12: end for
return Adjacency Matrix A A
Algorithm 3 Approximate Graph Learning
1: Input: Ordered skill sets Swain and Seval. Number of training steps K, base model fo-
2:
ie [k] do
3: Train fo on samples from Keran for K steps and denote fK, to be the model after training.
4:
for E [m)] do
5:
Observe change loss, 63 Leval, (fk,.) 8J > O, draw edge Strain, + Striin,j and set Arj >0. 0.
6:
7: end for
8: end for
9:
return Bipartite Adjacency Matrix Ae RR
C.2 Graph Learning Method
We provide algorithms for learning the graph over ordered skill set. In Algorithm 2, we discuss the brute-force approach for learning the adjacency matrix. This approach only works when Seval S Spain denote S in the algorithm box. In Algorithm we discuss the linear approach for learning the adjacency matrix. This approach works even in the out-of-domain case when Swall and Suain are disjoint.
(e.g. pre-training and fine-tuning cases),
In both approaches, the exact value of Ai can vary, but we can typically set it proportional to 6) in the brute-force case or 8) in the approximate case. The exact constructions and methods for learning each A in our experiments are in Appendix D.2.
D Additional Experimental Details
D.1
Datasets
We present details about each dataset used, including information on the skills and the validation dataset. A summary is presented in Table 2.
Alpaca dataset [56]: the Alpaca dataset consists of 52K instruction examples that were generated from text-davinci-003. We applied the Berkeley Neural Parser [27, 28] each instruction, keeping 40777 samples it was able to parse successfully. If the sample began with a question, we annotated it with the skill "question", and otherwise we annotated it with the verb identified from the parser. We grouped the data into total of 38 skills, such as "list", "edit", "calculate", "describe" and "identify".
Pile of Law [22]: the Pile of Law dataset consists of various sources of legal and administrative data, ranging from tax rulings to the world's constitutions. We evaluate on subset of the Pile of Law validation dataset consisting of 13883 samples, where we selected max6645, source size) samples per source. We truncated each sample to be no more than 100K characters.
LEGO [72]: for the LEGO synthetic, we set k k= and sample 192000 points across the skills. Our validation dataset consisted of 100 samples per skill.
19

---

Table 2: We list each dataset used as well as its corresponding skill. We include the number of skills in the training dataset, as well as details on how the validation dataset constructed.
Dataset
Skill
skills
Validation data
Alpaca Pile of Law L.EGO
Instruction type
38
50 samples per skill
Legal data source
31
645 samples per skill
Reasoning chain depth
5
100 samples per skill
Addition
Digit
3
100 samples per skill
NI (pre-training)
Task category
23
50 samples per task
NI (Spanish QG)
Task category x language
4
100 samples per task
NI (stance detection)
Task category
2
50 samples per task
NI (out-of-domain)
Task category
59,12
400 samples per task
RedPajama
Data source
7
LM eval harness
Addition: for the 3-digit addition synthetic, set l: 3 and sample 192000 points across the skills. We use a validation dataset of 100 samples per skill.
Natural Instructions [63, 41]: the Natural Instructions dataset is a large collection of tasks and their definitions in natural language. For the pre-training setting, we used a set of 23 task categories that had the largest degree (in-degree + out-degree) in the learned skills graph, for a total of 1,232,437 samples and 425 tasks to select from. We evaluated on 50 samples per task.
For the fine-tuning setting with Spanish question generation, we select data over 4 skills (Spanish ques- tion generation, Spanish question answering. English question generation, English question answer- ing) for a total of 513210 samples and 212 tasks to select from. We evaluated on 100 samples per task. For the fine-tuning setting with stance detection, we select data over skills (stance detection, text matching) for total of 50990 samples and 19 tasks to select from. We evaluated 50 samples per task.
For the out-of-domain setting, we select data over all 59 task categories for a total of 2,417, 7,667 samples and 753 tasks to select from. The test split consisted of 12 task categories and 119 tasks, and we evaluated on min(400, task size) samples per task.
RedPajama [57]: the RedPajama dataset is 1-trillion token dataset that aims to reproduce the LLaMA [59] training dataset. We select over the 7 data sources and evaluate using the LM evaluation harness [15].
D.2 Graph Learning Details
We describe how the skills graph was learned on each dataset.
Alpaca (Figure 9): we use Algorithm 3 and train for == 150 steps per skill. Each edge - has a weight of 63. the difference in loss on skill j before and after training on i. Next, we compare the average validation loss of skill-stratified sampling versus random sampling when we train for K= 1000 steps. We find that skill-stratifie sampling only does 0.007 better than random sampling, confirming that Alpaca's 's dense skills graph suggests that random sampling is the best we can do.
Pile of Law (Figure 10): we use Algorithm 3 and train for K = 150 steps. Each edge i + has a weight of 8). the difference in loss skill j before and after training on
LEGO (Figure 1I): we use both Algorithm 2 and Algorithm 3 and train for K 6000 steps each. Each edge i - j has a weight of 0.5 if the amount of data associated with skill j that is needed to reach 0.01 validation loss is less when training on (i.j) than on ý (edges are set to if 0.01 validation loss is not reached, even if loss is decreasing). Each edge i - j is also set to 0.5 if training on i be Addition (Figure 12): we use Algorithm and train for K 6000 steps. Each edge j has a weight of 0.5 if the amount of data associated with skill j that is needed to reach 0.01 validation loss is less when training on (i.j) than on (edges are set to if 0.01 validation loss is not reached, even if loss is decreasing). We set each diagonal entry of A to be 1.
decreases loss directly on j. We set each diagonal entry of
Natural Instructions (Figure 13, 14, 15): we use Algorithm 3. For the pre-training setting, we train for K = 600 steps and assign each edge j a weight 5y equal to the change in loss on j in the first 100 steps for all i.j E [k), including diagonal entries. For the fine-tuning setting, we train for K 600 steps and assign each edge ++ a weight de equal to the change in loss before and after training. For the -fm-oin setting, we train for = 600 steps and assign each edge i - j weight 83 equal to the change in loss before and after training in the first 100 steps.
20

---

Alpaca
2dd
and
calculere
2.00
categorize
chahge
classily
1.75
com
compose
construe ucu
1.50
create
design
determin
develop
1.22
evalua
explai
fnn
11.00
generale
idemary
dws
make
0.75
putine
quostion
select
suggest
summar
take
translate
Write
0.00
analy
cateponze
gener
make
outt
suddan
translill Write
art
Commn
comm
deleig avve
Figure 9: Alpaca heatmap where ijtth entry is max(0,6)) (the change in loss on sj after training on 8 for 150 steps). Diagonal entries are set to 0 for clearer visualization.
RedPajama (Figure 16): we Algorithm 3 and train for 1 billion tokens per data source. We assign each edge i+ weight 03 equal to the change in perplexity on the validation datalsoa before and after training.
D.3 Training Details
We describe the parameters used for SKILL-IT.
SKILL-II pre-training
L=G0:)=00.T=6..--- We train for 6000 steps.
Addition: 0,, We train for 6000 steps.
Natural Instructions (pre-training): y= 0.2,7
We train for 5000 steps.
For the LEGO random baseline, when we selected points at random, we used an imbalanced training dataset with proportions 1:1:1:3:5. For the addition random baseline, we used imbalanced dataset with randomly selected proportions: 13:14:18. For the curriculum learning baselines, the pacing function, (i), denotes the size of the subset of the highest scoring samples that we uniformly select from in the ith epoch. We define pacing function as g(i) sH. where H is the number of steps and M is epochs for LEGO and NI, and 3 for addition.
M
SKILL-IT fine-tuning
Lf 10 3. We train for 6000 steps.
Addition: 7 0-1,777 -5, We train for 6000 steps.
Natural Instructions (Spanish -., We train for 600 steps.
Natural Instructions (stance detection): y 0.2,7 -,, =3 train for 600 steps.
SKILL.-IT out-of-domain
Natural Instructions: 1 = 10, We train for 5000 steps.
RedPajama: // 100,7= 1. We train for billion tokens.
All results are computed over 5 random seeds.
21

---

Pile of Law
atticus contracts
bva opinions canadian decisions cfph creditcard contracts congressional hearings
1.6
courllistener docket enlry documents courtlisteieer opinions dol ecab echr
1.4
edgar
1.2
soir
eurlex
federal register
euro pari
1.0
[ounding docs
hhs mas_alj_ophhhoe opinions
icj-pcij medicaid policy guidance nlrb dectsions
0.8
0.6
olc memos
legaladvice
resource contracts scotus ligs
0.44
scotus pročeedings
sec, administrative
tax debates
0.2
us
uspto_affice affice actions
0.0
allic conttriiin
Sanadee byn contooo
contrae
cfpb congre crressionnl hearrnn
courtlistener; dockel pacelpppplll pairy document
ninns
echt
sägal
201
founth gorgit00
dets
hiss all aomiic
medicaid, policy guide hin qucision
si s
s o
federal guro
- Aoosi S0 soooooo asyood no pepooooo earepms
olc
mr in
ote
massi lop aanee
aae sentisanne
Figure 10: Pile of Law heatmap where i.jth entry is max(0.8)) (the change in loss on 8, after training on for 150 steps). Diagonal entries are set to 0 for clearer visualization.
LEGO
1.0
0.8
2
0.6
3
0.4
4
0.2
5
0.0
1
2
3
4
5
Figure 11: LEGO heatmap with l: where i.5th entry set to 0.5 if the number of steps needed to reach 0.01 loss on skill j when training on balanced mixture of skills i and is less than when training on skill j only.
22

---

Addition
1.0
0.8
0.6
2
0.4
0.2
0.0
Figure 12: Addition heatmap with 3= where i,tth entry is set to 0.5 if the number of steps needed to reach 0.01 loss on skill j when training on balanced mixture of skills and is less than when training skill j only.
Natural Instructions
answer verification code to text
discourse_conneccvvee identification entity generation
0.8
entity relation classification
information extraction
irony detection
preposition prediction irppoitttonpprediitiinnn
0.6
punctuation error detection
question answering
question generation
question understanding
sentence expansion sentiment, analysis detection summarization text categorization text matching text simplification text to code
0.4
0.2
toxic language detection word semantics
wrong_candidatt_geneeee
0.0
answer verification code text
discourse connective identification entity gegeeettioon antity relation classification information,
extraction irony detection prediction p error detection.
question. answering
quesstion generation
understanding
sentence expansion
sentiment analysis
stance. detection summarization
implification text categorization text matching
text code detection
toxtc, languade word, semantics wrong candidate generation
preposition_
question pneettion_uute
text,
Figure 13: Natural Instructions heatmap where i.ith entry is max(0.5;) (the change in loss on 8, after training on 8; for 100 steps). Diagonal entries are set to for clearer visualization.
23

---

Spanish question generation
Stance detection
English QA
0.6
Stance Detection
1.0
Spanish QA
0.4
0.5
Text Matching
English QC
0.2
Spanish QG
Stance Detection
Text Matching
0.0
English QA
Spanish QA
English QG
Spanish QG
Figure 14: Spanish question generation and stance detection heatmaps where i,jth entry is max(0.6))
(the change in loss on 8, after training on s; for 100 steps).
Natural Instructions
vertiiationn Orde to text
coberencs chasificttion ensmonserse classlicatoon dialogue geseration
discourse connective iden Rication
d'scouske relation chassficaatio
entity eatilyy gesuration
0.14
lantiaaaioon
fact vesticatioon
in the blank- geoder ficatioooon grammar
information extraction intent idea ticattn
Irony detaction-
linguisli probing
named eatity recognitinn
negotation, strategy detection
conversion
paraphrasing
preposition, prediction.
pos tagging
0.00
programs
punctuation detection
quniition answering
question deromposition
quactiion under standing
question gea eraton-
0.
0.06
sentence compression
senteate expansson
seateace ordering
sentence porturbation
sentimeet
0.06
fication
speaker idea fifcttioon
spelling ecror detection
stereotype detection.
story composition
style transfer
summarzzation
categorizatin
test. matching
text quality. avaluation.
text simplificatio.
toxic Janguage detection transiation
word. relation chassfication.
semantics.
wrong
co yeu ceme ai mon com coree comeeell
dads to Malliin
aartii stalimm
un
toe ondo
Figure 15: Natural Instructions heatmap for out-of-domain setting where rows are for the training skills and columns are for the evaluation skills. The i.jth entry is max(0,5) (0,8)) (the change in loss on sy after training on s; for 100 steps).
24

---

RedPajama
arxiv
0.2
04
crawl
gilhub
0.1
stackexcbange
wikipedia
0.0
easy
boolg
copa
hellaswag
lambada openai
pigaa
vinograadd
Figure 16: RedPajama heatmap for out-of-domain setting where rows are for the training skills and columns are for the evaluation skills. The i,jth entry max(0,6)) (the change in perplexity on 8y after training on s; for 1B tokens).
Batch sizes of 32 and 64 were used for the LEGO and addition synthetic on the 125M and 1.3B parameter model, respectively. Batch sizes of 4 and 16 were used for the Natural Instructions experiments on the 125M and 1.3B parameter model.
For the out-of-domain Natural Instructions experiment and Alpaca graph learning experiments, learning rate 5e-6 with linear scheduler and 50 warmup steps was used. All other experiments used learning rate of 5c-5. All experiments used AdamW with betas = 0.9, 0.999, eps ep9..eps le-8, and weight decay = 0.01. A context window of 512 was used for all experiments except LEGO and addition, which used window of 128.
Experiments with the Addition dataset were run using an Nvidia RTX A6000. Other experiments using the GPT-Nco 125M parameter model were run Nvidia Tesla P100. Experiments using the GPT-Neo 1.3B parameter model were run on an Nvidia Tesla A100.
E
Additional Experimental Results
E.1
Additional examples of LEGO ordered skill sets
For the LEGO synthetic, may appear obvious that the skills graph is equivalent the reasoning chain over the variables. However, in Figure 17 we see that this is not the case. Training on skills 2 and 4 together results lower loss on skill than when trained skill alone. However, training on skills and together results in roughly the same loss skill 4 as when training on skill 4 alone, even though skill and skill share an edge the LEGO synthetic's underlying reasoning chain. This suggests that our intuition for how skills influence each other does not always match how the model learns skills.
complex reasoning pattern the LEGO synthetic. Instead of chain, we construct a tree, where two variables in the LEGO synthetic are both defined in terms of the same parent variable. For example.
Next, consider slightly
Input: val 1, not w, val c, w not c. Output: y=
In this example, à: = and both e and w are written in terms e, and the reasoning graph has edges 1 1 3, 2 44 In this case, we see that training on skill 2 or skill 3 both improve losses on skills and 3 (Figure 18). However, unlike the previous figures, training on skills and skills 3 and
4 do not significantly help reduce loss skill 4 (Figure 19). Again, these measurements demonstrate that the reasoning graph does not necessarily equal the skills graph.
E.2 Unsupervised skill recovery
We explore several clustering techniques for recovering the skills in the LEGO synthetic on the validation dataset. Our results are shown in Table 3.
We first cluster based on the pre-trained model embeddings of the last token and the average token. We also report accuracies of clustering based the trained model embedding's 's last token, where we train
25

---

Model performance on LEGO skill 4
Model performance LEGO skill
0.9
Trained on skill 4
Trained on skill
Validation Loss
0.8
Trained on skills
Validation Loss
0.72
Trained on skills 3, 4
0.7
0.6
0.70
0.5
0.68
0.4
2000
4000
6000
2000
4000
6000
Steps
Steps
Performance on LEGO skill 4 when training on skill 4, skills and and skills and Even though skill 3 and skill 4 share an edge in the LEGO synthetic's 's underlying reasoning chain (i.e. a model predicting correct for the fourth variable one extra step beyond predicting correct for the third variable), we find that training on skills 2 and helps improve performance on skill 4 more.
Figure 17:
Model performance on LEGO skill 3 (tree) Model performance on LEGO skill 2 (tree)
Trained on skill 3
Trained on skill 3
Validation Loss
0.6
Trained skill
Validation Loss
0.6
Trained skill
0.4
0.2
0.2
0.0
0.0
1000
2000
3000
0
1000
2000
3000
Steps
Steps
Figure 18: Performance on LEGO skill and 3 when training on skills 2 and 3. The reasoning pattern tree rather than chain over l variables. Skills 2 and 3 are at the same "depth" in the graph and both depend on skill 1, so there is positive influence between the skills despite there being no edge between and 3 in the LEGO reasoning graph.
Model performance LEGO skill 4 (tree)
Model performance LEGO skill 4 (tree)
Trained on skill
0.8
Trained on skill
Validation Loss
0.6
Trained on skills 2, 4
Validation Loss
0.6
Trained on skills 3, 4
0.4
0.4
0.2
0.2
0.0
0.0
1000
2000
3000
0
1000
2000
3000
Steps
Steps
Figure 19: Performance on LEGO skill 4 when training on skills 2,4 and skills 3,4. We find that in both cases. the benefit from training on additional skills is minor. For instance, training 2 and reaches 0.01 loss in 2700 steps, while training on 4 only reaches it in 2100 steps.
26

---

Table 3: Clustering-base based skill recovery methods on the LEGO dataset. The validation dataset we cluster consists of 500 points with k= and results are reported over 10 runs of k-means.
Cluster method
Accuracy
Pretrained embedding of last token
0..8.10.5
Pretrained embedding of average token
2..2+1.1
Trained model embeddinn of last token
38.4
Sentence-BERR embedding
23.9.10.7
Losses over multiple runs
61.0=1..
the model using random sampling for 6000 steps, and clustering based on Sentence- BERT embeddings. Among these four methods, using the trained model embeddings has the highest accuracy of 38.4 points.
Next, we cluster points based on losses. In particular, we runs, each for 6000 steps and with randomly sampled mixture of skills. For each run, we evaluate the model on the validation dataset at 120 checkpoints. Then, each sample the validation dataset has 1200 losses associated with it, comprising feature vector for that sample. We perform k-means clustering these features, which has an accuracy 61.0 points, significantly higher than the second best accuracy of 38.4.
E.3
Full results for Section
E.3.1 Per-skill performance
In this section, we provide tables containing the per skill break-down of our results from Section 5,
Continual Pre-training In the continual pre-training setting, we report two additional baselines that combine curriculum learning with skills. Curriculum learning has been proposed for multitask learning [60], which groups of data are ranked by their average score and then trained in order of this ranking (with mixing of previously seen groups to avoid forgetting). We construct two baselines, Skill-curriclumm and skilllanticuricullmm using Algorithm from [60]. In contrast to the random baseline which has imbalanced skills, this approach has knowledge of skills and thus uses stratifit training dataset to sample from. We set the fraction of the previous group to be frac=0.4 0.4,
found that setting frac = 0.0 resulted in forgetting.
We report loss per skill for the LEGO synthetic in Table 4, which corresponds to the results in Figure We report accuracy per skill in Table 5 and Figure 20. We report the loss per skill for the Addition synthetic in Table 6, which also correspond to to the results in Figure 4, and we provide the accuracy per skill in Figure 21 (we omit table of accuracies since most methods attained 100% accuracy by the end of training). Finally, we report validation loss per task category for the Natural Instructions continual pre-training experiment in Table 7, where we find that SKILL-IT outperforms random sampling by 3.2% on average across skills.
4able Results on validation loss per skill for LEGO pre-training experiment, averaged over 5 random seeds.
Skill
Skill
Skill
Skill 4
Skill 5
Average
Random Curriculum
040.000
0.66920.05
0.686+0.013
0.676_0.046
0.678
0.54220.0a
0.003 002
0.431 028
0.875.10051
0.140=o.022
0.00116.000
0.29 0.29020.000 008
Anticurriculum
0.047
0.351 011
0.601
0.121
0.003
0.225
Skill-stratffied
O40.000
0.045 D3G
0.056440.o2r
0.080_a.644
0.050+4.022
0.016
Skill- cull-uuuriuulum
040.000
0.639 20. 101
0.701+o.o12
0.697 -0. 0012
0.693+6..002
0.5466
Skill anticurriculum
0.001
0.306
1.541+0.2000
0.596 201
1.611+0.172
0.41 0.411
SKILL-IT
0.00410.005
0.009
0.029110.06
0.043
0.01916.006
0.021 10.009
27

---

Table 5: Results on accuracy skill for Le experiment averaged over 5 random seeds.
Skill
Skill 2
Skill 3
Skill
Skill 5
Average
Random
100.040.0
53.8
57.4
28.2=6.1
51.2+4.3
62.1
Curriculum
100.010.0
80.2+1.3
49.6a3.0
97.0=2.3
100.0:0.0
85.410.7
Anticurriculum
98.8
84.613.0
68.0a6.9
96.8=1.3
100.0
89.611.9
Skill-stratified
100.040.0
98.211.8
98.2a1.3
97.8=1.6
98.2
98.5
Skill-currcclum
100.0
57.4415.7
50.423.6
46.8=7.4
51.823.7
61.324.8
kkill-nnteurrrculumm
100.040.0
86.8
65.4210.7
61.2=16.6
12.0
75.1
SKILL-IT
99.810.4
99.620.9
99.0ga.7
98.6_0.9
99.6go.9
99.320.4
LEGO Skill 1
LEGO Skill 2
LEGO Skill 3
1.0
1.0
1.0
Accuracy
0.8
0.8
0.6
0.6
0.6
0.2
0.4
0.4
2000
4000
6000
2000
1000
6000
2000
1000
6000
LEGO Skill 4
LEGO Skill 5
Average per skill
1.0
1.0
1.0
0.8
Accuracy
0.8
0.6
Random
Cartioalum
0.6
1/-
0.6
0.4
Anticurriaalom
cursiccuri
Skill eelorrstti
0.4
0.2
Skill-1t
2000
4000
6000
2000
4000
6000
2000
4000
6000
Steps
Steps
Steps
Figure 20: Accuracy of SKILL-IT each skill the LEGO synthetic (if the desired variable has the correct value out 0 or 1) in the continual pre- pre-training setting. SKILL-IT attains higher accuracy more quickly than baselines that both do and do not utilize the notion of skills.
Table 6: Results on validation loss per skill for Addition pre-training experiment, averaged over random seeds.
Skill
Skill 2
Skill 3
Average
Random
0.008 006
0.01610.000
0.006
0.01011..003
Curriculum
0.009 005
0.024
0.018
0.017.0.008
Anticurricuum
0.00440 001
0.011 1.011+00.002
0.016 003
0.01 1 1:0.002
Skill-stratified
0.015 012
0.017++..004
0.010=a.00x 0000
0.014+4.006
Skill-curriculum
0.027 D46
0.700 1.188
0.661 =0. 884
0.463+4.660
Skill-anticuriculleg
0.478 037
0.19444..00
0.188
0.287
SKILL-IT
0.003 001
0.008 4008++..00
0.009 0.00
0.007 +0.0000002
28

---

Addition Skill
Addition Slell
Addition Skill
Average per skill
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
Nundre
Figure 21: Accuracy of SKILL-IT on each skill on the addition synthetic (10-way accuracy the predicted digit is correct) in the continual- pretraiiing setting. On average, SKILL-IT attains higher accuracy more quickly than baselines that both do and do utilize the notion of skills.
29

---

Table 7: Validation loss per skill for data selection in continual pre- pre-training setting on a subset of the Natural Instructions Dataset.
Skill
Random
Curriculum
Anticurriculum
Skill-stratified
Skill-curriculum
Skill-anticurricllum
SKILII-IT
Answer Verification
2.297 =0.088
2.368+0.055
20.61
2. 2..00
2.249
2.325
2.158=0.059
Code to Text
0.246g0.02
0.203+0.019
1.099
0.178
0.12640.009
1.232=00070
0.223=0.017
Discourse Connective Identification
2.927 060
3.08440.067
2. 2.932
2.805
2.891
2.925
2.784_0.068
Entity Generation
2.033=0. 421
2.012
2.363 234
L.803
1.853
2.068
1.863
Entity Relation Classification
1.020
1.0144+0.140
1.533++0.13s
0.859 10.1181
0.825
0.959 1.95944..00 40.94400009
0.908
Information Extraction
2.154
2.247 00.007
2.352+4..44
2. 2.140 0.0007
2. 2.286+0.u22
2.338 40.0225
2.073
Irony Detection
3.024=0.154
3.79810.099
2.942:1.116
2.680 146
3.88940.06 066
2.099
2. 2.
Preposition Prediction
0.979
0.887
1.488
0.845
0.94110.011
1.04440 029
0.876=0.178
Punctuation Error Detection
2.950 066
3.12012.055
2.961 064
3.26-110.00 00.061
3.01910.010
3.360 013
3.216=0.055
Question Answering
2.277 20.00
2.367++0.006
2.398
2.5.22
2.689+0.0001
2.707 +0.000016
2.448=0.00s
30
Question Generation
2.617=0.003 005
2.777+0.015
2.695
2. 2.783+0.002
3. 3.062
2.876 032
2. 2.666=0.012
Question Understanding
1.965
2.199+0.058
2.060+0.00
1.958+0.051
2. 2.385+0.022
2. 2.100
1 1.895=0.043
Sentence Expansion
2.501 095
2.598
2.583
2,.25512
2.31110.00
2,408 074
2. 2.236
Sentiment Analysis
3.203
3.415
3.209
3.278 10.014
3..00100.012
3.308 015
3.213
Stance Detection
1.810=0.100
1.775.10.20
2.231 10.128
1.385 1.0070
1.361 1:0. 114
1.823
1.556
Summarization
2.961 10.015
3.149
3.041 10.4400.000
2.960 4:0011
3.32340.028
3.021 1:.000000
2.907
Text Categorization
2. 2.488
2.692+0.02
2.553+0 006
2.570+0.016
3.001+0.007
2. 2.635
2.448=0.017
Text Matching
2.177
2.232
2.316 048
2.152+0.061
2.32441.000
2.304
2.093_0.054
Text Simplification
2.155 023
2.193+0.039
2.325
1.926+0.022 20.026
2.037 +0.005
2.156=0.011
1.952=0.026
Text to Code
0.560 u 0087
0.495
1.215
0.490 +0.099
0.433+00.01
1.455
0.553=0.042
Toxic Language Detection
3. 3.106 u..600002
3.496
3.058
3. 3.199 40.024
3. 3.758+0.025
3. 8.155
3.129 =.129=u.v20
Word Semanties
2.092 027
2.334
2.15640.00 o64
1.916 +0.043
1.784400.049
2.42440.08
1.952
Wrong Candidate Generation
2.138
2.606
2.519
2.506
2.84940.020
2.57 2.57110.018
2.432
Average
2.173
2.307
2.366
2.115+0.027
2.30440.031
2.317+0.052
2.103_a.032

---

Out-of-domain In Table 8, we provide breakdown of validation loss per evaluation skill under random sampling on the training data, skill-stratifice sampling over prerequisite skills (e.g., the nonzero rows in Figure 15), and SKILIIIIT.
Table 8: Validation loss per skill for data selection in out-of-domain setting over Natural Instructions train task split and test task split.
Skill
Random
Skill-stratified
SKILL-ITT
Answerability Classification
3.04840.008
3.07640.002
3.044310.008
Cause Effect Classification
2.06840.00
2.101 2.101 +0.005
2.067
Coreference Resolution
3.101
3. 3.14244.004
3. 0.999000044
Data Text
2.363
2.388400.00
2.359 2.359+10.005
Dialogue Act Recognition
2.329 009
2.36-4
2. 2.320 009
Grammar Error Correction
2. 2.399 000
2.418+0.008
2.389
Keyword Tagging
2. 2.744 005
2. 2.760+0.00r
2.733++
Overlap Extraction
2.749++0011
2.763
2. 2.733++0.00
Question Rewriting
2.591 10.009
2.628
2.586
Textual Entailment
2.472:1.000
2.503
2.468 1.0000
Title Generation
3.027 0.002
3.037
3.015
Word Analogy
1.665
1.68240.013
1.666
Average
2.546440.00
2.57240.00%
2.540+0.000
In Table 9 we provide a breakdown the RedPajama experiment's accuracy per evaluation skill, corresponding to the results in Figure 8.
Table 9: Performance of model trained RedPajama with uniform sampling and SKIIII.IIT on LM evaluation harness. Unless otherwise noted, accuracy is reported for each task.
Billion Tokens
Billion Tokens
3 Billion Tokens
Uniform
SKILL-IT
Uniform
SKILL-IT
Uniform
SKILL-IT
ARC Challenge (ace norm)
35.4
34.6
35.3
34.9
34.6
34.8
ARC Easy (acc norm)
62.2
61.2
62.4
61.7
62.5
62.0
BoolQ
68.9
68.2
67.7
68.6
67.2
68.7
CОРA
81.0
82.0
80.0
81.0
81.0
81.0
HellaSwag (acc norm)
63.9
63.7
63.8
63.9
64.0
63.9
LAMBADA OpenAI
64.4
67.0
65.9
66.7
66.8
66.0
PIQA (acc norm)
74.8
75.0
75.5
75.2
75.0
75.7
Winogrande
62.8
63.9
63.9
63.2
63.4
63.1
Average accuracy
64.2
64.4
64.3
64.4
64.3
64.4
E.3.2 Weight trajectories
We provide SKILL-IT'S weight trajectories for each result. The weight per skill across training steps for the LEGO pre-training experiment corresponding to Figure (left) shown in Figure 22. We that SKILL-IT initially allocates more weight to skill and less to 1,3,4,5. Since skill is learned quickly, the weight on skill immediately drops to below 0.1 at 1000 steps. The weight on skills 3,4, and increase from around to 3000 steps, during which their respective validation losses higher than those of skills and 2. Near the end of training, all losses are converging to 0, and so the weight per skill roughly uniform.
The weight per skill across training steps for the addition pre-training experiment corresponding to Figure 4 (right) is shown in Figure 23. SKILL-IT allocates more weight to skill 2, which has an edge to skill as shown in Figure 12. also allocates very little weight to skill 3, which is learned faster than the other two skills. Eventually, it puts to uniform sampling as all validation losses approach
weight skill the hardest skill, and then converges
The weight skill across training steps for the LEGO fine-tuning experiment and the Spanish question generation and stance detection experiments corresponding to Figure 6 shown in Figure 24. Since there is only one target skill in these experiments, the mixture of weights approaches uniform
31

---

Skill-It LEGO weights
0.35
Skill
Skill
0.30
Skill 3
Weight per skill
Skill
0.25
Skill
0.20
0.15
0.10
0.05
0
1000
2000
3000 Steps
4000
5000
6000
Figure 22: Weight per skill for LEGO pre-training experiment. SKILL-IT initially allocates more weight to skill 2, but eventually puts more weight on harder skills (3,4,5) before converging uniform sampling when all losses converge roughly to
Skill-It addition weights
0.55
Skill
Skill 2
0.50
Skill 3
Weight per skill
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0
1000
2000
3000 Steps
4000
5000
6000
Figure 23: Weight per skill for addition pre- pre-training experiment. SKILL-IT initially allocates more weight to skill 2, which has edge skill 1, while allocating little weight to skill 3 which is learned quickly. Eventually, SKILL-ITT puts more weight on the harder skill before converging to uniform sampling when all losses roughly approach 0.
as the loss on the target skill approaches O. It is interesting to explore how to reduce edge weights and regularization so that the mixture approaches the target skill instead, although preliminary experiments where we decayed the edge weight and the strength of the Bregman divergence term did not appear better. We hypothesize that since training on a uniform mixture (as in Figure 3) did strictly better than training on the target skill and their loss curves did not intersect during the training run, it is better to allocate negligible weight on all skills throughout the training run.
The weight per skill across training steps for the Natural Instructions -ft-oi--omain experiment corre- sponding to Figure is shown in Figure 25, where the legend provided for top task categories with the largest weights. While the initial weights based on the skills graph roughly establishes the order of weight magnitude, the differences among the losses on the evaluation skills increases the range of weights training continues. As validation losses saturate, the weights also converge to fixed values.
32

---

Skill LEGD weight for leamnnng dall
Skill-tt wed gite for Spantssh DG
weights Detection
0.75
SWeghh par PEE
0.32
2207
Beglish 06
Spaciaa GA
Brgfth 00
1507
Siance
Trat Matccing
Spazza
0.25
0.28
0423
1002
6030
2020
1000
100
2003
3013
002
Figure 24: Weight per skill for fine-tuning experiments. Left: LEGO; Center: Spanish question generation; Right: stance detection.
Skill-It weights for Natural Instructions
0.14
question generation
6.12
queetion anowaring
0811
6.10
categoctzation
analysis
p0r
wrong candiduls generation
matching
0.06
0.06
information extrctionn
question, understamling
0.02
language detection anguage_dettcctio
1000
2000
2000
Figure 25: Weight per skill for Natural Instructions out-of-domain experiment. The legend shows the top 10 skills with the largest weight. While the relative order of weight magnitude does not change significantly across training, the incorporation of loss dramatically increases the range of the weights, showing the importance of online algorithm.
E.4 Experiments on 1.3B parameter model
We demonstrate that the skills graph learned the 125M parameter model, described in section D.2 can be used for data selection with the GPT-Neo-1. model across all three evaluation settings. All results reported over random seeds.
Continual pre-training We report results for the LEGO continual pre-training experiment and the Natural Instructions continual pre-training experiment.
For the LEGO experiment, we use the skills graph learned on the 125M parameter model and train the 1.3B parameter model with SKILL-IT for 1500 steps with ry 0.5,7'= = 30, w== = 3. In Figure 26, SKILL-IT still outperforms random and skill-stratified sampling on average. In particular, while performance across sampling methods is similar for early skills, the discrepancy is larger for skill 5, for which SKILL- IT allocates more weight to dynamically. In Figure 27, we provide the weight trajectories of SKILL-IT. We observe that the weight trajectories are similar to that the 125M parameter model, where initial weight is allocated towards skill 2. Later on, more weight is allocated towards skills 4 and 5, whose losses are higher, and eventually the weight mixture converges to uniform all losses converge to near O.
For the Natural Instructions experiment, we use the skills graph learned on the 125M parameter model and train the 1.3B parameter model with SKILL-IT for 5000 steps with y 0.2, 7 1, and learning rate le -
Table 10, we report performance SKILL-IT on the Natural Instructions skills and find that the trends from the smaller model hold-SKILL-IT outperforms random and skill-stratified sampling on average.
Fine-tuning We use the skills graph from the 125M parameter model to train the 1.3B parameter model with SKILL-IT for Spanish Question Generation fine-tuning. For 300 steps with learning rate 5e-6, 77 0.8, and T 12, we find in Figure 28 that SKILL-IT outperforms both skill-stratified and random sampling.
-omainn We evaluate if the skills graph learned using the 125M parameter model can be used in SKILL-IT on a 1.3B parameter model for the out-of-domain setting on Natural Instructions. Our results are shown in Figure 29, where trained for 1000 steps, learning rate 1e - 6. n 0.5, and T = 2.
33

---

LEGO Skill 1
LEGO Skill 2
LEGO Skill 3
100
109
Validation Loss (Log)
10-1
10-1
10
10
10
2
10-3
10
10-3
10
10
500
1000
1500
500
1000
1500
500
10000
1500
LEGO Skill 4
LEGO Skill 5
Average per skill
109
109
109
Validation Loss (Log)
10-1
10-1
10-1
10-2
10-2
10-2
Random
10
10-3
10-3
Skill-stratiffed
Skill-It
500
1000
1500
500
1000
1500
500
1000
1500
Steps
Steps
Steps
Figure 26: Performance of SKILL-IT for LEGO pre-training setting when skills graph is learned on a 125M parameter model and used for data selection with 1.3B model. SKILL-IT on average still outperforms random and skill-stratified sampling, suggesting that findings on ordered skill sets can transfer from small models to large models.
Skill-It LEGO weights for 1.3B param model
Skill 1
0.6
Skill 2
Skill 3
Weight per skill
Skill 4
0.3
0.2
250
500
750
1000
1250
1500
Steps
Figure 27: Weight per skill for LEGO pre-training experiment on 1.3B parameter model. The trajectories similar to those of the 125M parameter model in Figure 22. SKILL-IT initially allocates more weight to skill 2, but eventually puts more weight on skills and 5 before converging to uniform sampling when all losses converge near
Out of 12 evaluation skills, SKILL-IT outperforms random and skill- stratfitra sampling 10 skills (compared to skills when we used the same size model for both skills graph learning and training).
E.5 Ablations
We report ablations on components of the skills graph learning algorithm and on other components of -KI..I-IT. First, we study how the performance of SKII.I-IIT is robust changes the skills graph learning the graph with fewer steps, using the approximate algorithm, and adding noise to
34

---

Table 10: Results when skills graph for Natural Instructions learned on 125M parameter model is used for data selection with 1.3B model. We see that SKILL--TT on average still outperforms random and skill-stratified sampling, even though the edges used by SKILL-IT are not derived from the larger model.
Skill
Random
Skill-stratified
SKILL-IT
Answer Verification
2.005=0.059
1.903
1.890
Code to Text
0.302
0.204 022
0.269 032
Discourse Connective Identification
2.529 046
2.372 054
2.393 056
Entity Generation
2.108
1.788 429
1.885.10.46
Entity Relation Classification
1.130 0488
0. 1336=0.000
=.001
Information Extraction
2.032=0.013
1.992
1.933
Irony Detection
2.802
2.528
2.585+40.149
Preposition Prediction
1.095
0.686
0.77440.022
Punctuation Error Detection
2.633
3.188 40.05
2.726:40.05
Question Answering
1.947
2.119 =0.000
2.073+40.001
Question Generation
2.214
2.345 0.00
2.263110.110
Question Understanding
1.928
1.837
1. 1.700 042
Sentence Expansion
2.054
1.828 0.000
1.853
Sentiment Analysis
20.719
2.818 2.818=0.000
2. 2.77440.007
Stance Detection
1,814
1.500
1. 6..8
S Summarization
2.531
2. 2.472=0.011
2. 2.4404400033
Text Categorization
2.289
2.341:00 021
2.23110.00
Text Matching
1.967 20,008
1.913 00.000 0.00
1.872.
Text Simplification
1.861 10,003
1.692 023
19..0.022
Text to Code
0.61420.030
0.51820.00
0.585
Toxic Language Detection
2,853
2.911
2.862
Word Semantics
1.999=0.023
1.870 23003
1.902
Wrong Candidate Generation
2.187 20.08
2.192=0.023
2. 2.140
Average
1.985
1.907
1.883
Performance on Spanish QG (125M ~> 1.3B)
Spanish QG only Skill-stratifled Skill-It
2.2
Validation Loss
2.1
2.0
1.9
1.8
50
100
150
200
250
300
Steps
Figure 28: Performance of SKILL-IT for Spanish Question Generation fine-tuning setting when skills graph is learned on 125M parameter model and used for data selection with 1.3B model. SKILL-IT outperforms random and stral--tra sampling, suggesting that findings on ordered skill sets can transfer from small models to large models.
the adjacency matrix. Second, we study the effect of removing the skills graph, using a static skills mixture, and varying the y in SKI.II-IT.
E.5.1 Skills Graph Ablations
We explore the effect of changing how the skills graph is learned and adding noise to the graph SKILL-IT.
First, in Appendix C.2 we propose a brute-force graph learning algorithm and an approximate graph learning algorithm. We find that these algorithms exhibit little difference in performance. For the LEGO continual pre-training experiment, we find that replacing the brute-force skills graph with an
35

---

Answerabllity Classiiration Cause Effect Classification
Coraforence Resolution
Data To Text
1.90
2.75
Skil--ttatifed
valddntee
Skil-itt
2.00
2.70
12.65
Dialogue Act Recopnition
Grammar Error Correction
Keyword Taggiag
Overlap Extraction
2:10
2.20
2.40
Valdaton
2.05
2.30
2.30
Question Rewriting
Textual Entailment
Title Generation
Word Analogy
Velataten Less
2.25
1.45
2.20
2.55
1.40
2.15
2.20
2.50
11.35
0
5ón
1000
560
1000
0
560
1000
560
1000
Stops
Steps
Steps
Steps
Figure 29: Performance of SKILL-IT for Natural Instructions of setting when skills graph learned on a 125M parameter model and used for data selection with a 1.3B model. SKILL-IT outperforms random and skill-stratified sampling on 10 out of 12 skills.
approximate skills graph yields average validation loss of 0.017 .00 compared to 0.021 +0 +0.000 009. For the Spanish question generation fine-tuning experiment, we find that using the brute-force algorithm yields validation loss of 2.339 40.028 compared 2.328=0.022 In both experiments, there was no more than 0.011 gap in validation loss from using one skills graph learning algorithm over the other, and both algorithms resulted in SKILL-IT outperforming baselines.
We next show that we can reduce H, the number of steps for learning each edge in the skills graph, without significantly compromising SKIII-IIT's performance. perforance In Table 11, are able reduce H by at least 50% without changing the downstream average validation loss of SKIII--IT by more than 0.01. Results are shown for three experiments: LEGO continual pre-training, Spanish question generation fine-tuning, and Natural Instructions out-of-domain.
Table 11: Average validation loss of SKILL-IT with graph learned using original steps versus reduced steps.
Experiment
Original steps
Reduced steps
Skill-It (original)
Skill-It (reduced)
LEGO PT
6000
1500
0.021 1.0211+0000 .00009
0.01940.007
Spanish QG FT NI OOD
600
300
2.32840..02
2.33840.02
600
50
2.540
2.538
We also show that perturbations in the values of do not significantly impact the performance of SKILL-IT. We learn each individual skills graph using out of 5 random seeds. We then use each skills graph as input to SKILL-IT and measure the per-element variance in the A adjacency matrix well as the variance in the average validation loss from using SKILL-IT. In Table 12, we that the skills graph has (le - 3) variance while SKILL-IT'S loss has O 4) variance. We further study the effect of perturbations by applying Gaussian noise to each element of A and measuring the performance of SKILL-IT (Figure 30). We find that SKILL-IT becomes worse than baselines only after perturbing A by standard deviation of roughly 0.13 per element, which we never attain in Table 12.
Table 12: Variance in skill graph's adjacency matrix A over 5 random seeds compared with variance in SKILL-IT's performance from using different skills graphs.
Experiment
Variance in A
Variance in SKILL-IT'S average validation loss
LEGO PT Spanish QG FT NI OOD
2.40 10
4.93 x 10-5
1.31 x 10
3.76 x 10
9.22 10
6.35 x 10
Lastly, one may wonder if is better to allocate the O(Hk) training steps needed for skills graph learning towards just training model longer using naive baseline that does not utilize skills. Let be the re-
36

---

1830 with nony skilks graph
Perfommnnc on Spanish QC with neisy skills graph
NI DOD with nong skills gmah
2,570
2,505
1.33
sumping
Spareak QC eny
strattbed
Bandom sazrping
stratifedd
0.2
Joveyy gopht
2.3
onlly graph)
thely goopli
234
2.54
0.10
0.33
502
0.10
0.35
0.23 Standant Aeeivtiinn Ooarstinn alded
0.10
Standand dww.aion
Endlatine Cacocin vdded
Figure 30: Skill-It on LEGO continual pre-training, Spanish question generation fine-tuning, and Natural Instructions out-of oot-of-domain when Gaussian noise is added to each element of adjacency matrix A. We use standard deviation
...01,0.05.0....
duced overall number of steps for skills graph learning from Table I, let be the number of steps for training the LM. We consider a random sampling baseline on n-+ fi steps and compare it to SKILL-IT on steps, with the skills graph learned over h steps. In Table 13, the random baseline does not outperform SKILL-IT, suggesting that the improvement in model performance outweighs the graph learning cost.
Table 13: Comparison of SKILL-IT versus allocating skills graph learning budget to extend training with random sampling.
Experiment
n
n
Random (n+hh)
Skill-It (n)
LEGO PT
21000
6000
0.510+0.229
0.019 007
Spanish QG FT NI OOD
1800
600
2.430
2.338 +0.084
8950
6000
2.541
2.538
E.5.2 SKILL-IT ablations
We report ablations on removing the skills graph, using a static skills mixture, and varying the ry in SKILL-IT.
First, instead of using Algorithm 1, we study the performance when the identity matrix used instead: intuitively, this corresponds to a misspecified skills graph where no skill influences another skill. We refer this approach as "No graph". Note that the opposite case of a complete graph recovers skill-stratified sampling, which we already have as a baseline.
Second, instead of sampling over multiple rounds and weighting according to the loss of each skill, study the effect of setting 7 = 1, which only uses a softmax on A to yield static weights on the skills. We refer this approach as "Static". We omit results Natural Instructions continual pre-training, since SKILL-IT IT uses 7==1 1 and using no graph with T 7= 1 recovers skill-stratified sampling. Intuitively, we expect the static version of -KILII-ITT to perform somewhat well unless there is significant discrepancy among the losses (e.g. in synthetics where the loss on one skill can be close to 0 while the other is not, versus in Natural Instructions where all losses decrease consistently). For both ablations, we sweep over values of 7 (0.1,0.2,0.5,0.33
Figure 31 shows the comparison between SKILL-IT and no graph on the continual pre-training LEGO experiment, and Figure 32 shows the comparison between SKILL-IT and a static approach. We see that both the graph and the online dynamics of SKILL-II IT are important for its performance. In particular, using no graph results in allocating significant weight to harder skills early on, even though many of them have easier prerequisite skills (such skill 3 having edges skills and 2). Using static graph results in consistent allocation of significant weight to prerequisite skills even after their validation losses converge to near 0, and thus the harder skills that have higher loss are not learned quickly afterwards.
We perform the same ablation on the Addition dataset- the results for this are shown in Figures 33 and Figure 34. We find that these simple baselines, including using a static graph and no graph perform similarly to SKILL-IT on average across all diills-wiee SKILL-IT performs the best on skill compared to vanilla multiplicative weights, and SKILL-IT performs the best skill I compared to a static graph. This suggests that Addition is somewhat easier than the other datasets that we consider, as SKILL-IT still outperforms other baselines, as shown in Figure 4.
Figure 35 compares -KILL-IIT, no graph, and static data selection for the LEGO fine-tuning experiment. No graph can be interpreted allocating equal weight to all training skills not equal to the target skill,
37

---

LEGO Skill
LEGO Skill 2
LEGO Skill
10a
10a
10-1
10-2
Validation
10
10-2
10-4
2000
4000
6000
2000
4000
16000
2000
4000
6000
LEGO Skill 4
LEGO Skill 5
Average per skill
Validation Locs
10-1
10"2
No graph (0.1)
No graph (0 (0.2)
No graph (0.5)
graph (0.8)
Skill-It
10:
10
60D0
2000 4000
2000
6000
Steps
Steps
Steps
Figure 31: Comparison of SKILL-IT versus using the identity adjacency matrix (no skills graph) with 7- 0.1.0.,.0. 0.8. on the LEGO continual pre-training experiment. The latter does not capture the relationship between skills, and we find that SKILL-IT attains lower loss on all skills.
LEGO Skill
LEGO Skill 2
LEGO Skill
10a
10a
10
10-1
10-2
Validation
10
10-8
10
10-4
2000
1000
6000
0
2000
2000
6000
0
2000
1000
6000
LEGO Skill 4
LEGO Skill 5
Average per skill
10a
Validation Loss (Lag)
10-1
10-1
Static (0.1)
Static (0.2)
Static (0.5)
Static (0.8)
Skill-It
10 2
60D0
4000
2000
6000
Steps
Steps
Stops
Figure 32: Comparison of SKILL-IT versus using static data selection (T'=1) with 0.1 on the LEGO continual pre-training experiment. While SKILL-IT eventually allocates more weights skills 3, 4. 5, which have higher loss, the static approach is not able do this. We find that SKILL-IT attains lower loss on all skills.
and varying this weight versus the weight on the target skill. While SKILL-IT and setting T T- behave similarly, we that SKILL-IT is slightly better than using graph. For instance, SKILL-IT obtains a validation loss of 0.05 2000 steps, compared to 2050-2200 steps when using no graph.
Figure 36 and 37 compare SKIILIITT, no graph, and static data selection for the Natural Instructions fine-tuning experiments. For both Spanish QG and stance detection, SKILII-IT attains lower loss than using graph using T round.
38

---

Addition Skill
Addition Stall
Viltatton (oee
2010
6000
2000
4000
6050
Addition Skill
Average skall
1.0%
107
10-1
graph
Figure 33: Comparison of SKILL-IT versus using the identity adjacency matrix (no skills graph) with = 0.1.0.2.0.5,0.8 on the Addition continual pre- pre-trainnn training experiment. The latter does not capture the relationship between skills, and we find that SKILL-IT attains lower loss on skill 2, but attains similar performance to methods that do not use the skills graph.
1.0
10a
Velteton Look
10-1
10-4
10-2
2000
6000
6000
2000
4000
5000
Addition Skill
Average per skdll
Unitatten
zo-1
10
Static (8.1)
Static 10,21
Stalic 10.5)
Static 10.00
Stcil-12
20000
4000
6000
2000
4000
5000
Steps
Steps
Figure 34: Comparison of SKILL-IT versus using static data selection (T= with y 0.1.0.2..200..0000 on the Addition continual pre-training experiment. We find that SKILL-IT attains lower loss on skill 1, but attains similar performance to the static methods.
Figure compares SKILL-IT and static data selection for the Natural Instructions out-of-domain experiment. SKILL-IT attains the lowest validation loss on out of 12 evaluation skills. It has an average loss of 2.540 compared a range of 2.541-2.551 for static data selection.
For LEGO continual pre-training and Spanish question generation fine-tuning (Figure 39), find that SKILL-IT with varying outperforms stral--tra and random sampling (with the exception of 0.1 and skill-stratified being roughly the same for Spanish question generation). On the Natural Instructions out-of- do-mom experiment, we find that for =01 and 0.2, we outperform baselines on and 11 out of 12 evaluation skills, respectively, while increasing 7) 0.5 and 0.8 resulted in outperforming on and skills. These findings show that SKILL-IT is generally not hypersensitive to the choice of 1), and we hope they provide some guidelines for a reasonable range of r) without requiring significant tuning.
Lastly, we study the effect of varying in SKILL-IT. We use (0.1.0.2,0.5.04.
39

---

Performance on Skill
Performance on Skill 3
No graph (0.1)
Static (0.1)
0.8
graph (0.2)
0.8
Static (0.2)
graph (0.5)
Static (0.5)
Validation Loss
0.6
No graph (0.8)
Validation Loss
Static (0.8)
Skill-It
Skill-It
0.4
0.4
0.2
0.2
1000 2000
4000 5000 Steps
3000
6000
1000
2000
4000 Steps
3000
5000
6000
Figure 35: Comparison of SKILL-IT versus using graph (left) and static data selection (right) with 0,=0.1,02,00.50.33 on the LEGO fine-tuning experiment. All approaches have roughly the same loss trajectories, but SKILL-IT is slightly lower than using graph.
Performance Spanish QG
Performance Spanish QG
2.60
2.60
No graph (0.1)
Static (0.1)
2.55
No graph (0.2)
2.55
Static (0.2)
No graph (0.5)
Static (0.5)
Validation Loss
Validation Loss
2.50
No graph (0.8)
2.50
Static (0.8)
Skill-It
Skill-It
2.45
2.45
2.40
2.40
2.35
2.35
2.30
2.30
200
300
400
500
600
200
300
400
500
600
Steps
Steps
Figure 36: Comparison of SKILL-IT versus using no graph (left) and static data selection (right) with 7)= 0=0...0.2.0.5 0.8 on the Natural Instructions Spanish QG fine-tuning experiment. SKILL-IT attains lower validation loss than both no graph and static data selection.
Performance on stance detection
Performance on stance detection
1.8
No graph (0.1)
Static
1.7
No graph (0.2)
1.7
Static (0.2)
Validation Loss
No graph (0.5)
Validation Less
Static (0.5)
1.6
No graph (0.8)
1.6
Static (0.8)
Skill-It.
Skill-It
1.5
1.5
1.4
1.4
1.3
1.3
1.2
200
300
400
500
600
200
300
400
500
600
Steps
Steps
Figure 37: Comparison of IT versus using graph (left) and static data selection (right) with 7 = 0.1,0.2,0..500. 2,0. on the Natural Instructions stance detection fine-tuning experiment. SKILL-IT attains lower validation loss than both no graph and static data selection.
40

---

Answerability Classification
Classification
Resolution
(0.1)
3.16
Valdbation Lase
3.10
Static (0.5)
2.10
2:37
Static (0.8)
Skill-t
3.12
2.36
3.06
2.35
Dialogue Recogultion
Keyword Tagging
Overlap Extraction
2.80
Validation Loss
2.38
2.42
2.78
2.78
2.76
2.34
2.76
2.32
2.74
2.74
Question Rewriting
Textual Entailment
Title Generation
Word Analogy
1.70
Validation Loos
2.
2.52
3.05
2.61
2.60
1.68
2.59
3.02
4000 Staps
5000
1000
3000 Stega
4000
5000
1000
2000
4000
5000
1000
2000
3000
4000
5000
Staps
Staps
Figure 38: Comparison of SKILL-IT versus using static data selection with 1) 1 0.1.0.2220 on the Natural Instructions out-of-domain experiment. SKILL-IT attains the lowest validation loss on 7 out of 12 evaluation skills, and an average loss of 2.540 compared to range of 2.541-2.551 for static data selection.
LEGO average validation loss per skill
Performance Spanish QG
2.60
Random
Skill stratifted
2.55
Validation Loss
Validation Loss
nm0.2
2.50
D-0.5
n- (SKII--
10-1
Random Skillsstrtiffed
2.45
0=0.1
2.40
0.2.2
0.5 (Skill-It)
2.35
0.8
10-2
2.30
1000
2000
3000
4000
5000
6000
200
300
400
500
600
Steps
Steps
Figure 39: Skill-It ablation of E (0.1,0.2,0.5.008)) for LEGO continual pre-training and Spanish question generation fine-tuning.
41


Reviews:
[{"review_id": "VuT1JTPx4w", "reviewer": "Reviewer_j55A", "summary": "Review Summary:\n\nThe paper explores the concept of clustering data into skills to establish a curriculum, aiming to increase the efficiency of language model (LM) training. The concept of a 'skill' is introduced as a unit of behavior with a corresponding data distribution. A skill, in this context, signifies that a model's performance improves on other data that assess a similar skill when it is trained on a dataset associated with that skill. \n\nAttempts to use embedding-based clustering for the unsupervised recovery of the curriculum did not yield desired results. However, loss-based clustering proved to be more successful, establishing that validation loss points from the same skill tend to follow similar trajectories. These trajectories were used to cluster by loss per timesteps per run. \n\nEquipped with this understanding, the authors suggest it is possible to generate an ordered skill set and extract a curriculum from it. The existence of such an ordered skill set in real data is demonstrated in the paper. \n\nThe paper also presents 'Skill-It', an approach to dynamically create training datasets, adjusting mixture weights based on expected improvements in generalization on the target task. However, the efficacy of this method is highly contingent on the accuracy of the transfer matrix $A$ approximation. \n\nThe method employed in the paper consists of two steps: \n1. Learning $A$ via brute force or linear approximation, and by approximating $A$ with a smaller model to reduce computational cost.\n2. Implementing 'Skill-It' to dynamically sample training data based on the evolution of losses.\n\nUsing synthetic and real data, the authors show that such ordered skill sets exist and can enable more data-efficient training. However, the paper also acknowledges that these ordered skill sets do not align with intuitive data groupings based on metadata and embedding clustering. Lastly, a novel online data sampling algorithm, SKILL-IT, is introduced for more efficient learning of skills. The authors demonstrate the effectiveness of their approach with improved accuracy in various training scenarios.", "questions": "See Weaknesses ", "limitations": "The limitations aren't discussed. \nI suggest some in the Weaknesses section.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Theoretical Soundness: The paper presents a robust framework based on the innovative concept of 'skills'. The methodology, which encompasses learning an approximation of the transfer matrix $A$, skill-stratified sampling, and the implementation of 'Skill-It', appears to be well-constructed and theoretically sound.\n\n2. Clarity of Presentation: The writing in the paper is clear, making the complex concepts and methodology accessible and comprehensible. The authors effectively communicate their ideas.\n\n3. Potential Impact: If the proposed method is effectively evaluated and shown to significantly surpass existing baselines, this work could potentially make a substantial contribution to the field of language model training. The concepts of ordered skill sets and their application for more efficient, skill-based training could lead to improved performance of language models and open up new avenues for research in this area.", "weaknesses": "Methodological Issues:\n\n1. Overlooking Computational Requirements: As the primary goal of this paper revolves around reducing computational expense, it's essential to consider the additional compute requirements of the proposed methods (e.g., graph learning part) in the evaluation of the results. \n\n2. Emphasis on a Singular Experiment: The additional computational requirements could be offset by significant improvements in larger scale models, making the 125M to 1B experiment particularly important. However, this leaves only the results from Figure 7 / Table 8 and Table 9 as practically relevant for real-data experiments.\n\n3. Computational Cost and Hyperparameters: The methodology is not without its complexities, including a fixed computational cost to learn $A$ and the introduction of a new hyperparameter, $\\nabla$. These increase the complexity of the method, making it necessary to demonstrate a strong outperformance over the random baseline to justify their inclusion.\n\nExperimental Shortcomings:\n\n1. Unspecified Deviations: The authors should clarify whether the deviations reported in the tables are standard errors or standard deviations. Without this clarification, the statistical significance of the results remains uncertain.\n\n2. Missing Data in Section 5.1: The lack of standard deviations or standard errors in the main text necessitates referring to the appendix for assessing the significance of results. For the Addition task, the paper does not sufficiently address the observed equivalence between Skill-It, Curriculum, and Anti-Curriculum. In the case of Natural Instructions (real-data), it's uncertain whether Skill-It significantly outperforms random.\n\n3. Incomplete Reporting in Sections 5.2 and 5.3: The omission of standard errors or deviations in Section 5.2, along with the absence of a random baseline, makes it challenging to evaluate the results. Similarly, for Section 5.3 (Figure 6 / Table 7 and Figure 7 / Table 8), it's unclear whether the results are statistically significant due to the absence of reported standard errors or deviations.\n\n4. Need for Statistical Tests in E.4: As the E.4 experiment is likely the one where the initial cost of computing $A$ can be minimized, it's critical to determine if Skill-It is significantly better than skill-stratified sampling. This requires performing appropriate statistical tests with a sufficient number of trials.\n\n5. Absence of Hyperparameter Ablation for Skill-It: While there's an ablation study for $\\nabla$ in E.5, one for Skill-It is notably absent. The authors need to demonstrate that Skill-It isn't hypersensitive to its hyperparameters, or otherwise recognize the associated costs of hyperparameter tuning.\n\n6. Limited Comparison to Stronger Baselines: Even without being experts in curriculum learning, one may wonder why the proposed method isn't benchmarked against stronger baselines from this field. This comparison would be valuable in further establishing the strengths and weaknesses of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are glad you appreciated the theoretical soundness and clarity of our paper. We address your concerns around the paper’s weaknesses below.\n\n**Computational costs and 1.3B experiments:** In our global response, we demonstrate how computational costs of the skills graph can be reduced. We also provide additional results showing that the Skill-It when trained on the 1.3B model with the graph learned on the 125M model still outperforms baselines.\n\n**Skill-It hyperparameter ablation:** We are not sure what you are referring to with $\\nabla$; could you please clarify? Assuming that you are referring to $\\eta$ as $\\nabla$, we ablate values of $\\eta = 0.1, 0.2, 0.5, 0.8$ for LEGO pre-training, Spanish QG fine-tuning, and NI out-of-domain experiments. For the former two experiments, Skill-It with varied $\\eta$ consistently outperforms skill-stratified and random sampling (Figure 3). For the NI out-of-domain experiment, $\\eta = 0.1$ and $0.2$ outperformed baselines on 9 and 11 out of 12 evaluation skills, respectively, while increasing $\\eta = 0.5, 0.8$ resulted in outperforming on 5 and 3 out of 12 skills. These findings show that Skill-It is generally not hypersensitive to the choice of $\\eta$, and we hope they provide some guidelines for a reasonable range of $\\eta$ without requiring significant tuning.\n\n**Reporting deviations:** All our reported deviations are standard deviations. We will clarify this in the paper and add in the standard deviations throughout the main text rather than just the Appendix. We address your comments about individual experiments below:\n\n- Addition task: Skill-It outperforms all baselines for skill 1 (ones digit addition) and is no worse than any baseline for skill 2. However, for skill 3 we see that both Skill-It and skill-stratified sampling do worse than random, curriculum, and anticurriculum. We hypothesize this is because the addition dataset consists of the skills in the ratio of 13:14:18, and so random sampling and curriculum learning will sample skill 3 more (see Figure 4 in the attachment), thereby decreasing its validation loss.\n\n- For the difference between random sampling and Skill-It on the NI out-of-domain experiment, we ran 5 more experiments for a total of 10 trials. The average validation loss per skill with random sampling is $2.548_{\\pm 0.001}$ while with Skill-It is $2.541_{\\pm 0.002}$.\n\n- Figure 7/Table 8 consisted of one training run due to the scale of the experiment; nonetheless, scaling up our method is a line of future work we are working towards.\n\n- In section 5.2, the baseline is just to sample randomly from the target skill itself. We omit the random sampling baseline over the entire dataset since it is not conventional for fine-tuning—the typical approach here would be to select i.i.d. train samples corresponding to the target skill.\n\n**Need for Statistical Tests in E.4:** We compare skill-stratified sampling and Skill-It on the 1.3B model over 20 trials each. We perform an independent two-sample t-test, finding that the p-value is 0.012. This value is below the conventional threshold of 0.05, suggesting strong evidence to reject the null hypothesis that there is no difference between the means of the two methods.\n \n**Limited Comparison to Stronger Curriculum Learning Baselines:** The curriculum learning baselines in our paper (Tables 3, 4, and 6 in Appendix E.3) are commonly used to benchmark curriculum learning [4]. Moreover, our baselines span a variety of curriculum learning methods by manipulating three out of four key features that characterize these methods: the pacing function for skills-curriculum [5] (how large of a pool of top-k samples to select from per epoch), the granularity of data (sample versus skills level), and the ordering (curriculum versus anticurriculum) [3, 4]. We did not change the fourth feature, the scoring function (how to rank samples), since alternatives to model loss tend to require auxiliary information or labels [4]. We also did not incorporate a self-paced curriculum [6], which is dynamic (versus using a static model to score) but requires additional parameters to learn on each sample. We will implement more variations of curriculum learning—for instance, sweeping over hyperparameters for the pacing function as described in [4]—in our final paper.\n\n[3] Soviany et. al. Curriculum Learning: A Survey.\n\n[4] Wu et. al. When do curricula work? \n\n[5] Varshney et. al. Let the Model Decide its Curriculum for Multitask Learning.\n\n[6] Jiang et. al. Self-Paced Curriculum Learning."}, {"review_id": "f1EyT8cyCl", "reviewer": "Reviewer_paFF", "summary": "The authors introduce a new way of curriculum learning that is thematically similar to active learning: subsets of the data are broken up into \"skills\" and in cases where there is a dependency structure of skills, learning precursor skills can help improve performance. When the dependency structure is either complete or empty there is not huge motivation for such approaches (as they layout), but cases where there is a clear dependency structure, they find gains in both final model performance and learning speed. They test GPT-NEO on synthetic tasks that require variable tracking and addition, as well as Natural Instructions. They demonstrates these gains in both pre-training and fine-tuning settings.", "questions": "0. I think this paper could be greatly improved with some deeper analysis in how the algorithm ends up actually working. Where and how does it differ from a straightforward curriculum? Even for a simple case like addition, are we mostly learning add1 and then add2 and then add3? Is it mixed? Do the proportions shift? Showing diagrams with the proportions + in comparison to random sampling and anti-curriculum could be illuminating. (I now see you have a motivation version of this in Figure 1. I'd really like to see the actual versions.\n1. Do you have a sense why Skill-stratified did worse across the board in Figure 6?\n2. How can Skill-it be better for Add-1? How is it different than random or curriculum? (To confirm: Those other cases are sampling across all the various Add-#s).\n3. Perhaps I missed this, but how does active-learning compare to anti-curriculum?", "limitations": "Seems good.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "0. This paper brings together some ideas on how to better train and fine-tune models on related tasks. They introduce a simple but clean (and flexible-enough) formalism of a skill that can be applied in an NLP setting.\n1. They demonstrate their method over 3 datasets.\n2. If one knows the graph of skills then it seems like this method will tend to be quite helpful in practice. (If not: see my concerns below!).\n3. I like Figure 1, making it a tiny bit bigger with bigger fonts would be awesome!\n\nI like this paper and I think it could be of practical use to people.", "weaknesses": "0. I'd prefer if more space were dedicated to talking about the adjacent matrix. It seems learning it is extremely expensive. How effective is the approximate approach? If we include the learning costs of the adjacency matrix (i.e., give all that extra training to uniform sampling), do the other approaches equal out? I think there are scenarios where we have a clear idea regarding the dependency structure and order.\n1. If the adjacency matrix is slightly wrong or noised, how does that impact the results of skill it? How consistent is the matrix over different random seeds / approximations?\n2. Continuing on, if there is a large quadratic operation to find the adjacency structure, where each cell basically asks, does learning task i help task j, is there any sense of which this paper is begging the question? I think not, because you're finding a practical and specific way of when and which tasks ia, ib, ic to sample to improve j.\n\nNit:\n1. I think the presentation of the LEGO example could possibly be made better, but I don't have great suggestions.\n2. The charts are not consistent in sizing. It is tough but doing so would improve presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your positive review. We are glad you liked the paper and think it could be of practical use. We address your concerns below.\n\n**Learning the skills graph:** We hope your questions around the approximate approach and the random baseline with larger training budget to adjust for the cost of graph learning have been addressed by our global response. Regarding your comment that there are scenarios where we know the dependency structure, we agree this is largely true, such as in the Spanish QG case. However, there were also cases where this did not hold. For example, in LEGO, learning Skill 1 did not help learn Skill 2 faster, even though the LEGO synthetic is designed such that Skill 2 requires completion of Skill 1. In these cases, learning the skills graph is necessary until we better understand why certain dependencies exist, which is an exciting direction for future work.\n\n**Skills graph robustness to noise:** In our global response, we show that the performance of Skill-It across different random seeds of the skills graph adjacency matrix $A$ has low variance O(1e-4), and we analyze the performance of perturbed $A$, finding that element-wise perturbations with standard deviation less than 0.13 still result in Skill-It outperforming all baselines.\n\n**Learning the  adjacency matrix efficiently:** We study three different settings: pre-training, fine-tuning and out-of-domain: \n\n- For the pre-training setting, while the brute-force method requires one to learn the adjacency matrix in a pairwise manner, we show in our global response how one can reduce the learning cost to be linear in the number of skills without affecting validation loss. Additionally, we also provide two other solutions: using a smaller LM to learn the adjacency matrix and shorter training runs to reduce the complexity of learning each edge. Both these methods reduce the computational cost by an order of magnitude with little impact on performance.\n\n- For both the fine-tuning and out-of-domain setups, we only need to learn a subset of the adjacency matrix depending on the number of evaluation skills. For fine-tuning, we only need to compute the edges from each training skill to the one target skill, and thus the graph learning cost is linear in the number of training skills.\n\nFinally, even if we use the brute-force method for the pre-training setup, this is simply an intermediate step and does not obviate the need for a sampling method for training an LM. Skill-It can thus be viewed as having two steps: 1) skills graph learning and 2) graph-aware sampling.\n\n**Analysis of Skill-It’s weight mixture versus baselines:** We have provided weight trajectories of Skill-It for LEGO pre-training, addition pre-training, and Spanish QG fine-tuning in Figure 4 of the attachment (solid lines). For LEGO pre-training and addition pre-training, the dashed lines denote the weight trajectories of curriculum learning. From these figures, Skill-It samples skills very differently from curriculum and random sampling (whose proportions are listed in Appendix E.3). We will update our draft with all weight trajectories for Skill-It and baselines.\n\n**Skill-stratified sampling in Figure 6:** In Figure 6, skill-stratified sampling is uniform over the set of prerequisite training skills that are relevant to the evaluation skills, e.g. positive rows in A. However, these skills are all sampled with equal weight, even though we found that skills like Question Answering and Question Understanding are much more influential than skills like Text to Code in terms of magnitude of edge weights.\n\n**Skill-It’s performance on Add-1:** Skill-It is better for Add-1 (ones digit addition) by exploiting the edge from Add-2 to Add-1 in the skills graph. That is, it starts out by sampling a mixture of Add-2 and Add-1, which helps learn Add-1 faster than random and curriculum baselines (see Figure 4).\n\n**Active learning versus anticurriculum learning:** Active learning and anticurriculum learning are similar in that they pick the hardest or most uncertain, and thus hopefully the most informative, samples. Active learning is typically used to determine what points to label, while anticurriculum learning is used to improve model performance. Note that in our setting, active learning is not applicable since our training objective is self-supervised. \n"}, {"review_id": "JSPys1JWf1", "reviewer": "Reviewer_6a64", "summary": "This paper addresses the problem of how to order the training data for a LM.  The authors define the notion of a skill, show that skills can be (noisily) identified in training data, show that skills can be ordered, and show that sampling from the data defining a skill in an intelligent manner can improve pretraining and finetuning.  They provide both batch and online training algorithms.  They present moderately comprehensive experimental results demonstrating the effectiveness of the approach.", "questions": "None", "limitations": "There is no dedicated section to limitations.  I do not foresee any negative societal impacts.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I really liked this paper. I think it's a great piece of work, and deserves to be published.\n\n+ The paper addresses an important and timely problem of great interest to a large community\n+ The paper is well-written and polished\n+ The paper strikes an excellent balance of notation, high-level ideas, and solid empirical work\n+ The resulting algorithm seems easy to implement and empirically effective\n+ I enjoyed the derivation of the Skill-IT algorithm - a tidy piece of work.  Well done.", "weaknesses": "- It's not clear exactly how skills can be effectively identified. Right now, the paper uses various forms of meta data, but this doesn't seem like quite the best (or most general) approach.  Maybe something based on the entropy of the text?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your positive review! We are happy that you enjoyed the paper and found it to be well-written.\n\n**How to identify skills:** We agree that identifying skills from data is a challenging problem and an exciting direction we are studying next. We will add this problem to a future directions section of our final paper. Currently, given a dataset and a partition of it into candidate skills, our paper can learn the skills graph, identify if the graph is meaningful (otherwise, if the partition gives you a complete or empty graph, you are better off using random or stratified sampling) and then translate the skills graph into a sampling algorithm. Our preliminary results on unsupervised skills recovery in section 3.3 also suggest that clustering embeddings doesn’t always recover skills, at least for LEGO, but clustering based on validation losses of different model checkpoints trained on different mixtures is promising. \n"}, {"review_id": "EF9T9cPLMV", "reviewer": "Reviewer_XvRR", "summary": "This paper presents a way to introduce a curriculum of tokens to train LLMs. To do that, the paper introduces the concept of skills, i.e. behaviors that can be acquired by training the LLM in a subset of data, and ordered skill sets: partial orderings of skills that can be learned faster when using the skill preconditions. The paper shows that both in real and synthetic datasets, following the skill order allows for more efficient learning. Next, the paper presents techniques to sample data points from skills, given an ordered skill set. The apporach is evaluated on different datasets and model sizes, obtaining significant reductions in validation loss. Finally, the paper presents an approach to learn the skill ordering.\n", "questions": "- On the definition skill, it would be good to clarify what the improved metric is with respect to. Is it compared to training from a random subset of X?\n- I understand from 4.4 how to learn the presence or absence of edges in the graph, but how do we find from there the weight of that edge? How sensible is the present algorithm to the edge weight.", "limitations": "No limitation is mentioned in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Relevance and novelty: The paper addresses an important problem; given the computational cost of training LLMs, optimizing performance given a fixed compute budge is an important problem. It also presents a new framing to do so, that departs from curriculum learning approaches and focuses in selecting data to learn a set of skills. The approach to sample skills is also novel and well thought, dealing with both sampling data that both improve the weaker skills and selects the amount of data to sample from each skill.\n\n- Method: Starting from a reasonable optimization objective and constraint (reduce the validation loss on every skill at every round), the method derives a closed from sampling rule, which is fast and simple to compute. The proposed rule can be applied to multiple cases: 1) those where all the training skills have to be evaluated, 2) cases where only a subset are considered for eval and 3) cases where no data of the eval skill is provided during training. \n\n- Generally (see questions) clear paper writing, the definitions and discussion on how different kinds of graphs affect good sampling strategies provides great intuition for the rest of the paper. The walkthrough examples in section 3.2 also clarifies the intuition of skill curriculum.\n\n- While the graph learning approach is computationally expensive, it can be applied to subsets of the dataset, allowing to transfer the task dependency matrix to the full data for sampling. \n\n- Experimental results: The method is tested across different datasets and model sizes. The proposed approach improves upon different sampling baselines, including methods that sample uniformly across skills, by obtaining lower validation losses or comparable losses than the baselines in a shorter amount of time. The method also improves over baseliens on out of domain settings, where the test skills are not available during training.", "weaknesses": "- The method relies very strongly on having a task dependency graph. While the paper proposes two approaches to obtain these graphs, I would like further discussion into that, particularly on how to obtain the edge weights, and the robustness in the appraoch to errors or noise in this depnedncy graph.\n\nClarity on writing:\n- Definitions: In the definition of skill it would be good to clarify what the improved metric is with respect to. Is it compared to training from a random subset of X?\n- Similarly, it only becomes clear in Section 4.1 that the goal is to learn a subset of skills (which can include only 1 skill or the full skill training set). It would be good to clarify this better in the introduction.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review! We are glad that you found the paper to be novel and address an important problem. We respond to your questions below.\n\n**Robustness to errors or noise in dependency graph:** In our global response, we show that the performance of Skill-It across different random seeds of the skills graph adjacency matrix $A$ has low variance O(1e-4), and we analyze the performance of perturbed $A$, finding that element-wise perturbations with standard deviation less than 0.13 still result in Skill-It outperforming all baselines.\n\n**Clarity:** The improved metric is with respect to the original LM’s validation loss on the skill; that is, we characterize a skill as a set of data where validation loss on a held out subset of the skill goes down over time when we train on data associated with that skill. We will clarify this in the final paper. We will also clarify in the introduction that our goal is to learn a subset of skills, not necessarily the entire training skill set.\n\n**Edge weight:** When using the full algorithm (Algorithm 2), the edge weight is obtained by taking the difference between [validation loss on skill j when training for K steps on skill j] and [validation loss on skill j when training for K steps on skill i and j uniformly]. When using the approximate graph learning algorithm (Algorithm 3), the edge weight is obtained by taking the difference between [validation loss on skill j using the original LM] and [validation loss on skill j after training the LM for K steps on skill i]. More information about setting edge weights can be found in Appendix C.2 and D.2, and we will clarify this more in our paper.\n"}]

Author rebuttal:
We thank the reviewers for their valuable feedback, which we have incorporated into our draft. We are glad that reviewers found the paper to address an important and timely problem (6a64, XvRR) and our method to be novel (XvRR), clean (paFF, 6a64), and sound (j55A). Reviewers thought the approach could be very useful in practice for training LMs (j55A, paFF), and found the paper to have clear presentation (j55A, 6a64, XvRR) and consist of solid empirical work (6a64). 

In our global response, we address reviewers’ questions around the computational costs of learning the skills graph (j55A, paFF, XvRR). For context, in our paper we ask the question, “does there exist an ordered skill set/skills graph that can result in substantial improvements over random sampling?” Despite convention suggesting that random sampling is effective, our work uncovers the existence of such ordered skill sets and also proposes an algorithm to learn their skills graphs in a way that isn't combinatorial (e.g., trial-and-error of task/data order). Optimizing the runtime of the skills graph algorithm was a secondary objective, so we did not extensively explore this in our submitted paper. Nonetheless, we believe the existence of ordered skill sets and a tractable skills graph learning algorithm make the paper interesting grounds for future work. 

**Reducing skills graph learning algorithm’s cost (j55A, paFF):** There are several avenues for reducing the cost of the skills graph learning algorithm; our updated draft includes the following proposed refinements to the brute-force algorithm: 1) reducing the number of training steps per edge, 2) learning the graph on a smaller LM, and 3) using the approximate method that is linear in m, the number of skills. These refinements are not only practical techniques for reducing costs but also suggest that the signal needed to identify edges might lie in more accessible sources of information than the brute-force method. We empirically evaluate the effects of these improvements:

- Reducing # of steps: in Table 1 in our attachment, we are able to reduce the number of steps used to learn each edge of the skills graph by at least 50% without changing the downstream average validation loss by more than 0.01. Results are shown for three experiments: LEGO PT (pre-training), Spanish QG FT (fine-tuning), NI OOD (out-of-domain).

- Using a smaller LM (j55A): in section E.4, we learn the skills graph on GPT-Neo-125M and use it to train the GPT-Neo-1.3B model for LEGO and NI PT. In our attached update, Figure 1 includes parallel experiments on Spanish QG FT and NI OOD, where Skill-It (with original hyperparameters) results in 5.3% improvement over random for Spanish QG and outperforms baselines on 10 out of 12 evaluation skills for NI. 

- Approximate method (paFF): the approximate and brute-force algorithms exhibit little difference in performance—no more than 0.011 gap in validation loss—for LEGO PR and Spanish QG FT, with both approaches outperforming baselines. For LEGO PT, using the approximate algorithm yields an average validation loss of $0.017_{\pm 0.005}$ compared to $0.012_{\pm 0.008}$ (brute-force). For Spanish QG FT, using the brute-force algorithm yields a validation loss of $2.339_{\pm 0.028}$ compared to $2.328_{\pm 0.029}$ (approximate).

**Skill-It’s robustness to noise in skills graph (paFF, XvRR):** Our findings above illustrate minimal performance degradation for Skill-It from cost reduction techniques. We extend our investigation by studying Skill-It’s overall robustness to noise in the skills graph. In Table 2, we report the average element-wise variance in A, the adjacency matrix, over 5 random seeds to learn edges. We then report the variance in the overall performance of Skill-It using these A’s. The skills graph has O(1e-3) variance, while Skill-It's average validation loss has O(1e-4) variance. In addition, Figure 2 shows how model performance changes as Gaussian noise is added element-wise to $A$. Skill-It becomes worse than skill-stratified sampling only after perturbing A by a standard deviation of roughly 0.13, which we never attain in Table 2. 

Table 2
| Experiment | Variance in A | Variance in Skill-It average val loss |
| ----------- | ----------- | ----------- |
| LEGO PT |0.0024| 4.93e-05 |
| Spanish QG FT | 0.000131| 0.000376 | 
| NI OOD | 9.22e-05|6.35e-05 |

**LM training vs graph learning tradeoffs (j55A, paFF):** Thank you for the suggestion. We study the effect of allocating the skills graph learning budget towards extending LM training using a naive baseline that does not utilize skills, and we have updated the paper with these results. Let h be the reduced overall number of steps for graph learning from Table 1 and let n be the number of steps for LM training. We consider a random sampling baseline for n+h steps and compare it to Skill-It for n steps, with the graph learned for h steps. In Table 3, the random baseline does not outperform Skill-It in terms of average val loss, suggesting that the improvement in model performance from Skill-It outweighs the graph learning cost. Lastly, note that several existing approaches [1, 2] require training a model to determine how to select data; however, they do not consider this additional cost in their baselines. 

Table 3
| Experiment | n+h | n | Random (n+h) | Skill-It (n) | 
| ----------- | ----------- | ----------- | ----------- | ----------- | 
| LEGO PT | 21000 | 6000 | $0.510_{\pm 0.229}$ | $0.019_{\pm 0.007}$ | 
| Spanish QG FT | 1800 | 600 | $2.430_{\pm 0.019}$ | $2.338_{\pm 0.024}$ |
| NI OOD | 7950 | 6000 | $2.541_{\pm 0.001}$ | $2.538_{\pm 0.001}$ | 


As suggested by reviewers, we address the limitations of this paper, including skills graph learning, in Appendix A of the current draft.

[1] Toneva et. al. An Empirical Study of Example Forgetting during Deep Neural Network Learning. 

[2] Paul et. al. Deep Learning on a Data Diet: Finding Important Examples Early in Training.


Meta-review: This paper proposes an approach to curriculum learning in LMs. This consists of two pieces. First, a method for clustering data points into skills and computing dependencies between them, both of which can be performed automatically based on training dynamics in small models. Second, a method for automatically generating skill curricula from this dependency graph. Reviewers were on the whole quite enthusiastic about the paper; concerns about baselines and variability across runs were satisfactorily addressed by the author response.
Decision: Accept (spotlight)