SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the out-of-domain Natural Instructions experiment and Alpaca graph learning experiments, a learning rate of 5e-6 with linear scheduler and 50 warmup steps was used. All other experiments used a learning rate of 5e-5. All experiments used AdamW with betas = 0.9, 0.999, eps = 1e-8, and weight decay = 0.01. A context window of 512 was used for all experiments except LEGO and addition, which used a window of 128.
Experiments with the Addition dataset were run using an Nvidia RTX A6000. Other experiments using the GPT-Neo 125M parameter model were run on an Nvidia Tesla P100. Experiments using the GPT-Neo 1.3B parameter model were run on an Nvidia Tesla A100.
E.1 Additional examples of LEGO ordered skill sets
For the LEGO synthetic, it may appear obvious that the skills graph is equivalent to the reasoning chain over the variables. However, in Figure 17 we see that this is not the case. Training on skills 2 and 4 together results in lower loss on skill 4 than when trained on skill 4 alone. However, training on skills 3 and 4 together results in roughly the same loss on skill 4 as when training on skill 4 alone, even though skill 3 and skill 4 share an edge in the LEGO synthetic's underlying reasoning chain. This suggests that our intuition for how skills influence each other does not always match how the model learns skills.
Next, we consider a slightly more complex reasoning pattern on the LEGO synthetic. Instead of a chain, we construct a tree, where two variables in the LEGO synthetic are both defined in terms of the same parent variable. For example,
$$
\text {I n p u t :} \mathrm {c} = \text {v a l} 1, \mathrm {y} = \text {n o t} \mathrm {w}, \mathrm {v} = \text {v a l} \mathrm {c}, \mathrm {w} = \text {n o t} \mathrm {c}. \text {O u t p u t :} \mathrm {y} = 1.
$$
In this example,  $k = 4$  and both  $v$  and  $w$  are written in terms of  $c$ , and the reasoning graph has edges  $1 \rightarrow 2$ ,  $1 \rightarrow 3$ ,  $2 \rightarrow 4$ . In this case, we see that training on skill 2 or skill 3 both improve losses on skills 2 and 3 (Figure 18). However, unlike the previous figures, training on skills 2 and 4 or skills 3 and 4 do not significantly help reduce loss on skill 4 (Figure 19). Again, these measurements demonstrate that the reasoning graph does not necessarily equal the skills graph.
E.2 Unsupervised skill recovery
We explore several clustering techniques for recovering the skills in the LEGO synthetic on the validation dataset. Our results are shown in Table 3.
We first cluster based on the pre-trained model embeddings of the last token and the average token. We also report accuracies of clustering based on the trained model embedding's last token, where we train
Model performance on LEGO skill 4
Model performance on LEGO skill 4
Figure 17: Performance on LEGO skill 4 when training on skill 4, skills 2 and 4, and skills 3 and 4. Even though skill 3 and skill 4 share an edge in the LEGO synthetic's underlying reasoning chain (i.e. a model predicting correct for the fourth variable is one extra step beyond predicting correct for the third variable), we find that training on skills 2 and 4 helps improve performance on skill 4 more.
Model performance on LEGO skill 3 (tree)
Figure 18: Performance on LEGO skill 2 and 3 when training on skills 2 and 3. The reasoning pattern is a tree rather than a chain over  $k = 4$  variables. Skills 2 and 3 are at the same "depth" in the graph and both depend on skill 1, so there is positive influence between the skills despite there being no edge between 2 and 3 in the LEGO reasoning graph.
Model performance on LEGO skill 2 (tree)
Model performance on LEGO skill 4 (tree)
Figure 19: Performance on LEGO skill 4 when training on skills 2,4 and skills 3,4. We find that in both cases, the benefit from training on additional skills is minor. For instance, training on 2 and 4 reaches 0.01 loss in 2700 steps, while training on 4 only reaches it in 2100 steps.
Model performance on LEGO skill 4 (tree)
Table 3: Clustering-based skill recovery methods on the LEGO dataset. The validation dataset we cluster consists of 500 points with  $k = 5$ , and results are reported over 10 runs of k-means.
Cluster method	Accuracy
Pretrained embedding of last token	24.8±0.5
Pretrained embedding of average token	25.2±1.1
Trained model embedding of last token	38.4±0.8
Sentence-BERT embedding	23.9±0.7
Losses over multiple runs	61.0±1.6
the model using random sampling for 6000 steps, and clustering based on Sentence-BERT embeddings. Among these four methods, using the trained model embeddings has the highest accuracy of 38.4 points.
Next, we cluster points based on losses. In particular, we do 10 runs, each for 6000 steps and with a randomly sampled mixture of skills. For each run, we evaluate the model on the validation dataset at 120 checkpoints. Then, each sample in the validation dataset has 1200 losses associated with it, comprising a feature vector for that sample. We perform k-means clustering on these features, which has an accuracy of 61.0 points, significantly higher than the second best accuracy of 38.4.
E.3 Full results for Section 5
E.3.1 Per-skill performance
In this section, we provide tables containing the per skill break-down of our results from Section 5.
Continual Pre-training In the continual pre-training setting, we report two additional baselines that combine curriculum learning with skills. Curriculum learning has been proposed for multitask learning [60], in which groups of data are ranked by their average score and then trained in order of this ranking (with mixing of previously seen groups to avoid forgetting). We construct two baselines, Skill-curriculum and Skill-anticurriculum, using Algorithm 1 from [60]. In contrast to the random baseline which has imbalanced skills, this approach has knowledge of skills and thus uses a skill-stratified training dataset to sample from. We set the fraction of the previous group to be  $\mathrm{frac} = 0.4$ , as we found that setting  $\mathrm{frac} = 0.0$  resulted in forgetting.
We report loss per skill for the LEGO synthetic in Table 4, which corresponds to the results in Figure 4. We report accuracy per skill in Table 5 and Figure 20. We report the loss per skill for the Addition synthetic in Table 6, which also correspond to to the results in Figure 4, and we provide the accuracy per skill in Figure 21 (we omit a table of accuracies since most methods attained  $100\%$  accuracy by the end of training). Finally, we report validation loss per task category for the Natural Instructions continual pre-training experiment in Table 7, where we find that SKILL-IT outperforms random sampling by  $3.2\%$  on average across skills.
Table 4: Results on validation loss per skill for LEGO pre-training experiment, averaged over 5 random seeds.
Skill 1	Skill 2	Skill 3	Skill 4	Skill 5	Average
Random	0±0.000	0.669±0.051	0.686±0.013	0.676±0.046	0.678±0.032	0.542±0.028
Curriculum	0.003±0.002	0.431±0.028	0.875±0.051	0.140±0.022	0.001±0.000	0.290±0.008
Anticurriculum	0.047±0.012	0.351±0.011	0.601±0.128	0.121±0.017	0.003±0.001	0.225±0.026
Skill-stratified	0±0.000	0.045±0.036	0.056±0.029	0.080±0.044	0.050±0.025	0.046±0.022
Skill-curriculum	0±0.000	0.639±0.101	0.701±0.012	0.697±0.002	0.693±0.002	0.546±0.018
Skill-anticurriculum	0.001±0.001	0.306±0.218	0.541±0.252	0.596±0.201	0.611±0.172	0.411±0.144
SKILL-IT	0.004±0.005	0.009±0.011	0.029±0.016	0.043±0.020	0.019±0.006	0.021±0.009
Table 5: Results on accuracy per skill for LEGO pre-training experiment, averaged over 5 random seeds.
Skill 1	Skill 2	Skill 3	Skill 4	Skill 5	Average
Random	100.0±0.0	53.8±6.8	57.4±5.5	48.2±6.1	51.2±4.3	62.1±3.5
Curriculum	100.0±0.0	80.2±1.3	49.6±3.0	97.0±2.3	100.0±0.0	85.4±0.7
Anticurriculum	98.8±1.3	84.6±3.0	68.0±6.9	96.8±1.3	100.0±0.0	89.6±1.9
Skill-stratified	100.0±0.0	98.2±1.8	98.2±1.3	97.8±1.6	98.2±1.3	98.5±0.9
Skill-curriculum	100.0±0.0	57.4±15.7	50.4±3.6	46.8±7.4	51.8±3.7	61.3±4.8
Skill-anticurriculum	100.0±0.0	86.8±19.8	65.4±19.7	61.2±16.6	62.0±13.4	75.1±11.5
SKILL-IT	99.8±0.4	99.6±0.9	99.0±0.7	98.6±0.9	99.6±0.9	99.3±0.4
Figure 20: Accuracy of SKILL-IT on each skill on the LEGO synthetic (if the desired variable has the correct value out of 0 or 1) in the continual pre-training setting. SKILL-IT attains higher accuracy more quickly than baselines that both do and do not utilize the notion of skills.
Table 6: Results on validation loss per skill for Addition pre-training experiment, averaged over 5 random seeds.
Skill 1	Skill 2	Skill 3	Average
Random	0.008±0.006	0.016±0.004	0.006±0.001	0.010±0.003
Curriculum	0.009±0.005	0.024±0.006	0.018±0.007	0.017±0.006
Anticurriculum	0.004±0.001	0.011±0.002	0.016±0.003	0.011±0.002
Skill-stratified	0.015±0.012	0.017±0.004	0.010±0.004	0.014±0.006
Skill-curriculum	0.027±0.046	0.700±1.018	0.661±0.884	0.463±0.630
Skill-anticurriculum	0.478±1.037	0.194±0.269	0.188±0.256	0.287±0.451
SKILL-IT	0.003±0.001	0.008±0.003	0.009±0.005	0.007±0.002
Figure 21: Accuracy of SKILL-IT on each skill on the addition synthetic (10-way accuracy if the predicted digit is correct) in the continual-pretraining setting. On average, SKILL-IT attains higher accuracy more quickly than baselines that both do and do not utilize the notion of skills.
Table 7: Validation loss per skill for data selection in continual pre-training setting on a subset of the Natural Instructions Dataset.
Skill	Random	Curriculum	Anticurriculum	Skill-stratified	Skill-curriculum	Skill-anticurriculum	SKILL-IT
Answer Verification	2.297±0.058	2.368±0.055	2.391±0.061	2.180±0.059	2.249±0.116	2.325±0.085	2.158±0.059
Code to Text	0.246±0.021	0.203±0.019	1.099±0.115	0.178±0.016	0.126±0.009	1.232±0.070	0.223±0.017
Discourse Connective Identification	2.927±0.069	3.084±0.067	2.932±0.058	2.805±0.071	2.891±0.001	2.925±0.011	2.784±0.068
Entity Generation	2.033±0.421	2.012±0.437	2.363±0.234	1.803±0.384	1.853±0.483	2.068±0.719	1.863±0.418
Entity Relation Classification	1.020±0.147	1.014±0.140	1.533±0.138	0.859±0.131	0.825±0.022	0.959±0.009	0.908±0.146
Information Extraction	2.154±0.040	2.247±0.037	2.352±0.042	2.140±0.037	2.286±0.022	2.338±0.025	2.073±0.042
Irony Detection	3.024±0.154	3.798±0.095	2.942±0.158	2.680±0.146	3.889±0.066	2.099±0.152	2.797±0.155
Preposition Prediction	0.979±0.124	0.887±0.147	1.488±0.213	0.845±0.152	0.941±0.019	1.044±0.029	0.876±0.173
Punctuation Error Detection	2.950±0.065	3.120±0.052	2.961±0.064	3.264±0.061	3.019±0.010	3.360±0.013	3.216±0.055
Question Answering	2.277±0.005	2.367±0.006	2.398±0.006	2.542±0.004	2.689±0.001	2.707±0.016	2.448±0.008
Question Generation	2.617±0.005	2.777±0.015	2.695±0.008	2.783±0.021	3.062±0.006	2.876±0.032	2.666±0.012
Question Understanding	1.965±0.051	2.199±0.059	2.060±0.033	1.958±0.051	2.385±0.022	2.100±0.054	1.895±0.043
Sentence Expansion	2.501±0.095	2.598±0.097	2.583±0.074	2.225±0.095	2.311±0.076	2.408±0.074	2.236±0.083
Sentiment Analysis	3.203±0.012	3.415±0.016	3.209±0.010	3.278±0.014	3.607±0.012	3.308±0.015	3.213±0.012
Stance Detection	1.810±0.100	1.775±0.120	2.231±0.128	1.385±0.070	1.361±0.114	1.823±0.189	1.556±0.125
Summarization	2.961±0.015	3.149±0.023	3.041±0.014	2.960±0.019	3.323±0.028	3.021±0.013	2.907±0.012
Text Categorization	2.488±0.023	2.692±0.029	2.553±0.006	2.570±0.015	3.001±0.007	2.635±0.014	2.448±0.017
Text Matching	2.177±0.059	2.232±0.055	2.316±0.048	2.152±0.061	2.324±0.004	2.304±0.035	2.093±0.054
Text Simplification	2.155±0.023	2.193±0.039	2.325±0.033	1.926±0.026	2.037±0.005	2.156±0.011	1.952±0.026
Text to Code	0.560±0.037	0.495±0.036	1.215±0.052	0.490±0.029	0.433±0.014	1.455±0.086	0.553±0.042
Toxic Language Detection	3.106±0.027	3.496±0.017	3.058±0.029	3.199±0.024	3.758±0.025	3.155±0.050	3.129±0.020
Word Semantics	2.092±0.027	2.334±0.034	2.156±0.064	1.916±0.043	1.784±0.048	2.424±0.038	1.952±0.019
Wrong Candidate Generation	2.438±0.021	2.606±0.039	2.519±0.027	2.506±0.026	2.849±0.029	2.574±0.018	2.432±0.025
Average	2.173±0.028	2.307±0.025	2.366±0.026	2.115±0.027	2.304±0.031	2.317±0.052	2.103±0.032
Out-of-domain In Table 8, we provide a breakdown of validation loss per evaluation skill under random sampling on the training data, skill-stratified sampling over prerequisite skills (e.g., the nonzero rows in Figure 15), and SKILL-IT.
Table 8: Validation loss per skill for data selection in out-of-domain setting over Natural Instructions train task split and test task split.
Skill	Random	Skill-stratified	SKILL-IT
Answerability Classification	3.048±0.003	3.076±0.002	3.043±0.003
Cause Effect Classification	2.068±0.004	2.101±0.005	2.067±0.006
Coreference Resolution	3.101±0.003	3.142±0.004	3.099±0.004
Data to Text	2.363±0.004	2.388±0.005	2.359±0.005
Dialogue Act Recognition	2.329±0.009	2.364±0.010	2.320±0.009
Grammar Error Correction	2.399±0.008	2.418±0.009	2.389±0.007
Keyword Tagging	2.744±0.005	2.760±0.007	2.733±0.006
Overlap Extraction	2.749±0.011	2.763±0.012	2.733±0.010
Question Rewriting	2.591±0.009	2.628±0.011	2.586±0.010
Textual Entailment	2.472±0.002	2.503±0.003	2.468±0.002
Title Generation	3.027±0.002	3.037±0.002	3.015±0.002
Word Analogy	1.665±0.016	1.682±0.015	1.668±0.016
Average	2.546±0.003	2.572±0.003	2.540±0.003
In Table 9 we provide a breakdown of the RedPajama experiment's accuracy per evaluation skill, corresponding to the results in Figure 8.
Table 9: Performance of model trained on RedPajama with uniform sampling and SKILL-IT on LM evaluation harness. Unless otherwise noted, accuracy is reported for each task.
1 Billion Tokens	2 Billion Tokens	3 Billion Tokens
Uniform	SKILL-IT	Uniform	SKILL-IT	Uniform	SKILL-IT
ARC Challenge (acc norm)	35.4	34.6	35.3	34.9	34.6	34.8
ARC Easy (acc norm)	62.2	61.2	62.4	61.7	62.5	62.0
BoolQ	68.9	68.2	67.7	68.6	67.2	68.7
COPA	81.0	82.0	80.0	81.0	81.0	81.0
HellaSwag (acc norm)	63.9	63.7	63.8	63.9	64.0	63.9
LAMBADA OpenAI	64.4	67.0	65.9	66.7	66.8	66.0
PIQA (acc norm)	74.8	75.0	75.5	75.2	75.0	75.7
Winogrande	62.8	63.9	63.9	63.2	63.4	63.1
Average accuracy	64.2	64.4	64.3	64.4	64.3	64.4
E.3.2 Weight trajectories
We provide SKILL-IT's weight trajectories for each result. The weight per skill across training steps for the LEGO pre-training experiment corresponding to Figure 4 (left) is shown in Figure 22. We see that SKILL-IT initially allocates more weight to skill 2 and less to 1,3,4,5. Since skill 1 is learned quickly, the weight on skill 1 immediately drops to below 0.1 at 1000 steps. The weight on skills 3,4, and 5 increase from around 0 to 3000 steps, during which their respective validation losses are higher than those of skills 1 and 2. Near the end of training, all losses are converging to 0, and so the weight per skill is roughly uniform.
The weight per skill across training steps for the addition pre-training experiment corresponding to Figure 4 (right) is shown in Figure 23. SKILL-IT allocates more weight to skill 2, which has an edge to skill 1 as shown in Figure 12. It also allocates very little weight to skill 3, which is learned faster than the other two skills. Eventually, it puts more weight on skill 1, the hardest skill, and then converges to uniform sampling as all validation losses approach 0.
The weight per skill across training steps for the LEGO fine-tuning experiment and the Spanish question generation and stance detection experiments corresponding to Figure 6 is shown in Figure 24. Since there is only one target skill in these experiments, the mixture of weights approaches uniform
Figure 22: Weight per skill for LEGO pre-training experiment. SKILL-IT initially allocates more weight to skill 2, but eventually puts more weight on harder skills (3,4,5) before converging to uniform sampling when all losses converge roughly to 0.
Figure 23: Weight per skill for addition pre-training experiment. SKILL-IT initially allocates more weight to skill 2, which has an edge to skill 1, while allocating little weight to skill 3 which is learned quickly. Eventually, SKILL-IT puts more weight on the harder skill 1 before converging to uniform sampling when all losses roughly approach 0.
as the loss on the target skill approaches 0. It is interesting to explore how to reduce edge weights and regularization so that the mixture approaches the target skill instead, although preliminary experiments where we decayed the edge weight and the strength of the Bregman divergence term did not appear better. We hypothesize that since training on a uniform mixture (as in Figure 3) did strictly better than training on the target skill and their loss curves did not intersect during the training run, it is better to allocate non-negligible weight on all skills throughout the training run.
The weight per skill across training steps for the Natural Instructions out-of-domain experiment corresponding to Figure 7 is shown in Figure 25, where the legend is provided for the top 10 task categories with the largest weights. While the initial weights based on the skills graph roughly establishes the order of weight magnitude, the differences among the losses on the evaluation skills increases the range of weights as training continues. As validation losses saturate, the weights also converge to fixed values.
Figure 24: Weight per skill for fine-tuning experiments. Left: LEGO; Center: Spanish question generation; Right: stance detection.
Figure 25: Weight per skill for Natural Instructions out-of-domain experiment. The legend shows the top 10 skills with the largest weight. While the relative order of weight magnitude does not change significantly across training, the incorporation of loss dramatically increases the range of the weights, showing the importance of an online algorithm.
E.4 Experiments on 1.3B parameter model
We demonstrate that the skills graph learned on the 125M parameter model, as described in section D.2 can be used for data selection with the GPT-Neo-1.3B model across all three evaluation settings. All results are reported over 3 random seeds.
Continual pre-training We report results for the LEGO continual pre-training experiment and the Natural Instructions continual pre-training experiment.
For the LEGO experiment, we use the skills graph learned on the 125M parameter model and train the 1.3B parameter model with SKILL-IT for 1500 steps with  $\eta = 0.5, T = 30, w = 3$ . In Figure 26, SKILL-IT still outperforms random and skill-stratified sampling on average. In particular, while performance across sampling methods is similar for early skills, the discrepancy is larger for skill 5, for which SKILL-IT allocates more weight to dynamically. In Figure 27, we provide the weight trajectories of SKILL-IT. We observe that the weight trajectories are similar to that on the 125M parameter model, where initial weight is allocated towards skill 2. Later on, more weight is allocated towards skills 4 and 5, whose losses are higher, and eventually the weight mixture converges to uniform as all losses converge to near 0.
For the Natural Instructions experiment, we use the skills graph learned on the 125M parameter model and train the 1.3B parameter model with SKILL-IT for 5000 steps with  $\eta = 0.2$ ,  $T = 1$ , and learning rate  $1e - 6$ . In Table 10, we report performance of SKILL-IT on the Natural Instructions skills and find that the trends from the smaller model hold—SKILL-IT outperforms random and skill-stratified sampling on average.
Fine-tuning We use the skills graph from the 125M parameter model to train the 1.3B parameter model with SKILL-IT for Spanish Question Generation fine-tuning. For 300 steps with learning rate  $5e - 6$ ,  $\eta = 0.8$ , and  $T = 12$ , we find in Figure 28 that SKILL-IT outperforms both skill-stratified and random sampling.
Out-of-domain We evaluate if the skills graph learned using the 125M parameter model can be used in SKILL-IT on a 1.3B parameter model for the out-of-domain setting on Natural Instructions. Our results are shown in Figure 29, where we trained for 1000 steps, learning rate  $1e - 6$ ,  $\eta = 0.5$ , and  $T = 2$ .
Figure 26: Performance of SKILL-IT for LEGO pre-training setting when skills graph is learned on a 125M parameter model and used for data selection with a 1.3B model. SKILL-IT on average still outperforms random and skill-stratified sampling, suggesting that findings on ordered skill sets can transfer from small models to large models.
Figure 27: Weight per skill for LEGO pre-training experiment on 1.3B parameter model. The trajectories are similar to those of the 125M parameter model in Figure 22. SKILL-IT initially allocates more weight to skill 2, but eventually puts more weight on skills 4 and 5 before converging to uniform sampling when all losses converge to near 0.
Out of 12 evaluation skills, SKILL-IT outperforms random and skill-stratified sampling on 10 skills (compared to 11 skills when we used the same size model for both skills graph learning and training).
E.5 Ablations
We report ablations on components of the skills graph learning algorithm and on other components of SKILL-IT. First, we study how the performance of SKILL-IT is robust to changes in the skills graph—learning the graph with fewer steps, using the approximate algorithm, and adding noise to
Table 10: Results when skills graph for Natural Instructions learned on 125M parameter model is used for data selection with a 1.3B model. We see that SKILL-IT on average still outperforms random and skill-stratified sampling, even though the edges used by SKILL-IT are not derived from the larger model.
Skill	Random	Skill-stratified	SKILL-IT
Answer Verification	2.005±0.059	1.903±0.069	1.890±0.072
Code to Text	0.302±0.032	0.204±0.022	0.269±0.032
Discourse Connective Identification	2.529±0.046	2.372±0.054	2.393±0.056
Entity Generation	2.108±0.328	1.788±0.429	1.885±0.461
Entity Relation Classification	1.130±0.048	0.836±0.006	0.841±0.010
Information Extraction	2.032±0.013	1.992±0.006	1.933±0.013
Irony Detection	2.802±0.125	2.528±0.146	2.585±0.149
Preposition Prediction	1.095±0.040	0.686±0.041	0.774±0.029
Punctuation Error Detection	2.633±0.027	3.188±0.055	2.726±0.025
Question Answering	1.947±0.003	2.119±0.003	2.073±0.001
Question Generation	2.214±0.007	2.345±0.008	2.263±0.010
Question Understanding	1.928±0.020	1.837±0.031	1.700±0.042
Sentence Expansion	2.054±0.018	1.828±0.060	1.853±0.058
Sentiment Analysis	2.771±0.009	2.818±0.006	2.774±0.007
Stance Detection	1.814±0.151	1.500±0.117	1.628±0.149
Summarization	2.531±0.009	2.472±0.012	2.440±0.013
Text Categorization	2.289±0.016	2.341±0.021	2.231±0.022
Text Matching	1.967±0.008	1.913±0.005	1.872±0.005
Text Simplification	1.861±0.003	1.692±0.023	1.698±0.022
Text to Code	0.614±0.030	0.518±0.030	0.585±0.022
Toxic Language Detection	2.853±0.020	2.911±0.019	2.862±0.018
Word Semantics	1.999±0.023	1.870±0.039	1.902±0.024
Wrong Candidate Generation	2.187±0.028	2.192±0.023	2.140±0.020
Average	1.985±0.022	1.907±0.027	1.883±0.032
Performance on Spanish QG (125M -> 1.3B)
Figure 28: Performance of SKILL-IT for Spanish Question Generation fine-tuning setting when skills graph is learned on a 125M parameter model and used for data selection with a 1.3B model. SKILL-IT outperforms random and skill-stratified sampling, suggesting that findings on ordered skill sets can transfer from small models to large models.
the adjacency matrix. Second, we study the effect of removing the skills graph, using a static skills mixture, and varying the  $\eta$  in SKILL-IT.
E.5.1 Skills Graph Ablations
We explore the effect of changing how the skills graph is learned and adding noise to the graph on SKILL-IT.
First, in Appendix C.2 we propose a brute-force graph learning algorithm and an approximate graph learning algorithm. We find that these algorithms exhibit little difference in performance. For the LEGO continual pre-training experiment, we find that replacing the brute-force skills graph with an
Figure 29: Performance of SKILL-IT for Natural Instructions out-of-domain setting when skills graph is learned on a 125M parameter model and used for data selection with a 1.3B model. SKILL-IT outperforms random and skill-stratified sampling on 10 out of 12 skills.