SlowGuess/NeurIPS_2023_OCR huggingface dataset:
approximate skills graph yields an average validation loss of  $0.017_{\pm 0.005}$  compared to  $0.021_{\pm 0.009}$ . For the Spanish question generation fine-tuning experiment, we find that using the brute-force algorithm yields a validation loss of  $2.339_{\pm 0.028}$  compared to  $2.328_{\pm 0.029}$ . In both experiments, there was no more than a 0.011 gap in validation loss from using one skills graph learning algorithm over the other, and both algorithms resulted in SKILL-IT outperforming baselines.
We next show that we can reduce  $H$ , the number of steps for learning each edge in the skills graph, without significantly compromising SKILL-IT's performance. In Table 11, we are able to reduce  $H$  by at least  $50\%$  without changing the downstream average validation loss of SKILL-IT by more than 0.01. Results are shown for three experiments: LEGO continual pre-training, Spanish question generation fine-tuning, and Natural Instructions out-of-domain.
Table 11: Average validation loss of SKILL-IT with graph learned using original steps versus reduced steps.
Experiment	Original steps	Reduced steps	Skill-It (original)	Skill-It (reduced)
LEGO PT	6000	1500	0.021±0.009	0.019±0.007
Spanish QG FT	600	300	2.328±0.029	2.338±0.024
NI OOD	600	50	2.540±0.003	2.538±0.001
We also show that perturbations in the values of  $A$  do not significantly impact the performance of SKILL-IT. We learn each individual skills graph using one out of 5 random seeds. We then use each skills graph as input to SKILL-IT and measure the per-element variance in the  $A$  adjacency matrix as well as the variance in the average validation loss from using SKILL-IT. In Table 12, we see that the skills graph has  $\mathcal{O}(1e - 3)$  variance while SKILL-IT's loss has  $\mathcal{O}(1e - 4)$  variance. We further study the effect of perturbations by applying Gaussian noise to each element of  $A$  and measuring the performance of SKILL-IT (Figure 30). We find that SKILL-IT becomes worse than baselines only after perturbing  $A$  by a standard deviation of roughly 0.13 per element, which we never attain in Table 12.
Table 12: Variance in skill graph's adjacency matrix  $A$  over 5 random seeds compared with variance in SKILL-IT's performance from using different skills graphs.
Experiment	Variance in A	Variance in SKILL-IT's average validation loss
LEGO PT	2.40 × 10-3	4.93 × 10-5
Spanish QG FT	1.31 × 10-4	3.76 × 10-4
NI OOD	9.22 × 10-5	6.35 × 10-5
Lastly, one may wonder if it is better to allocate the  $\mathcal{O}(Hk)$  training steps needed for skills graph learning towards just training a model longer using a naive baseline that does not utilize skills. Let  $h$  be the re
Figure 30: Skill-It on LEGO continual pre-training, Spanish question generation fine-tuning, and Natural Instructions out-of-domain when Gaussian noise is added to each element of adjacency matrix  $A$ . We use standard deviation = {0.01, 0.05, 0.1, 0.2}.
duced overall number of steps for skills graph learning from Table 11, and let  $n$  be the number of steps for training the LM. We consider a random sampling baseline on  $n + h$  steps and compare it to SKILL-IT on  $n$  steps, with the skills graph learned over  $h$  steps. In Table 13, the random baseline does not outperform SKILL-IT, suggesting that the improvement in model performance outweighs the graph learning cost.
Table 13: Comparison of SKILL-IT versus allocating skills graph learning budget to extend training with random sampling.
Experiment	n+h	n	Random (n+h)	Skill-It (n)
LEGO PT	21000	6000	0.510±0.229	0.019±0.007
Spanish QG FT	1800	600	2.430±0.019	2.338±0.024
NI OOD	8950	6000	2.541±0.001	2.538±0.001
E.5.2 SKILL-IT ablations
We report ablations on removing the skills graph, using a static skills mixture, and varying the  $\eta$  in SKILL-IT.
First, instead of using  $A$  in Algorithm 1, we study the performance when the identity matrix is used instead; intuitively, this corresponds to a misspecified skills graph where no skill influences another skill. We refer this approach as "No graph". Note that the opposite case of a complete graph recovers skill-stratified sampling, which we already have as a baseline.
Second, instead of sampling over multiple rounds and weighting according to the loss of each skill, we study the effect of setting  $T = 1$ , which only uses a softmax on  $A$  to yield static weights on the skills. We refer to this approach as "Static". We omit results on Natural Instructions continual pre-training, since SKILL-IT uses  $T = 1$  and using no graph with  $T = 1$  recovers skill-stratified sampling. Intuitively, we expect the static version of SKILL-IT to perform somewhat well unless there is significant discrepancy among the losses (e.g. in synthetics where the loss on one skill can be close to 0 while the other is not, versus in Natural Instructions where all losses decrease consistently). For both ablations, we sweep over values of  $\eta = [0.1, 0.2, 0.5, 0.8]$ .
Figure 31 shows the comparison between SKILL-IT and no graph on the continual pre-training LEGO experiment, and Figure 32 shows the comparison between SKILL-IT and a static approach. We see that both the graph and the online dynamics of SKILL-IT are important for its performance. In particular, using no graph results in allocating significant weight to harder skills early on, even though many of them have easier prerequisite skills (such as skill 3 having edges to skills 1 and 2). Using a static graph results in consistent allocation of significant weight to prerequisite skills even after their validation losses converge to near 0, and thus the harder skills that have higher loss are not learned quickly afterwards.
We perform the same ablation on the Addition dataset—the results for this are shown in Figures 33 and Figure 34. We find that these simple baselines, including using a static graph and no graph perform similarly to SKILL-IT on average across all skills—while SKILL-IT performs the best on skill 2 compared to vanilla multiplicative weights, and SKILL-IT performs the best on skill 1 compared to a static graph. This suggests that Addition is somewhat easier than the other datasets that we consider, as SKILL-IT still outperforms other baselines, as shown in Figure 4.
Figure 35 compares SKILL-IT, no graph, and static data selection for the LEGO fine-tuning experiment. No graph can be interpreted as allocating equal weight to all training skills not equal to the target skill,
Figure 31: Comparison of SKILL-IT versus using the identity adjacency matrix (no skills graph) with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the LEGO continual pre-training experiment. The latter does not capture the relationship between skills, and we find that SKILL-IT attains lower loss on all skills.
Figure 32: Comparison of SKILL-IT versus using static data selection  $(T = 1)$  with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the LEGO continual pre-training experiment. While SKILL-IT eventually allocates more weights to skills 3, 4, 5, which have higher loss, the static approach is not able to do this. We find that SKILL-IT attains lower loss on all skills.
and varying this weight versus the weight on the target skill. While SKILL-IT and setting  $T = 1$  behave similarly, we see that SKILL-IT is slightly better than using no graph. For instance, SKILL-IT obtains a validation loss of 0.05 in 2000 steps, compared to 2050-2200 steps when using no graph.
Figure 36 and 37 compare SKILL-IT, no graph, and static data selection for the Natural Instructions fine-tuning experiments. For both Spanish QG and stance detection, SKILL-IT attains lower loss than using no graph or using  $T = 1$  round.
Figure 33: Comparison of SKILL-IT versus using the identity adjacency matrix (no skills graph) with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the Addition continual pre-training experiment. The latter does not capture the relationship between skills, and we find that SKILL-IT attains lower loss on skill 2, but attains similar performance to methods that do not use the skills graph.
Figure 34: Comparison of SKILL-IT versus using static data selection  $(T = 1)$  with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the Addition continual pre-training experiment. We find that SKILL-IT attains lower loss on skill 1, but attains similar performance to the static methods.
Figure 38 compares SKILL-IT and static data selection for the Natural Instructions out-of-domain experiment. SKILL-IT attains the lowest validation loss on 7 out of 12 evaluation skills. It has an average loss of 2.540 compared to a range of 2.541-2.551 for static data selection.
Lastly, we study the effect of varying  $\eta$  in SKILL-IT. We use  $\eta \in \{0.1, 0.2, 0.5, 0.8\}$ . For LEGO continual pre-training and Spanish question generation fine-tuning (Figure 39), we find that SKILL-IT with varying  $\eta$  outperforms skill-stratified and random sampling (with the exception of  $\eta = 0.1$  and skill-stratified being roughly the same for Spanish question generation). On the Natural Instructions out-of-domain experiment, we find that for  $\eta = 0.1$  and 0.2, we outperform baselines on 9 and 11 out of 12 evaluation skills, respectively, while increasing  $\eta = 0.5$  and 0.8 resulted in outperforming on 5 and 3 skills. These findings show that SKILL-IT is generally not hypersensitive to the choice of  $\eta$ , and we hope they provide some guidelines for a reasonable range of  $\eta$  without requiring significant tuning.
Figure 35: Comparison of SKILL-IT versus using no graph (left) and static data selection (right) with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the LEGO fine-tuning experiment. All approaches have roughly the same loss trajectories, but SKILL-IT is slightly lower than using no graph.
Figure 36: Comparison of SKILL-IT versus using no graph (left) and static data selection (right) with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the Natural Instructions Spanish QG fine-tuning experiment. SKILL-IT attains lower validation loss than both no graph and static data selection.
Figure 37: Comparison of SKILL-IT versus using no graph (left) and static data selection (right) with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the Natural Instructions stance detection fine-tuning experiment. SKILL-IT attains lower validation loss than both no graph and static data selection.
Figure 38: Comparison of SKILL-IT versus using static data selection with  $\eta = 0.1, 0.2, 0.5, 0.8$  on the Natural Instructions out-of-domain experiment. SKILL-IT attains the lowest validation loss on 7 out of 12 evaluation skills, and an average loss of 2.540 compared to a range of 2.541-2.551 for static data selection.
Figure 39: Skill-It ablation of  $\eta \in \{0.1,0.2,0.5,0.8\}$  for LEGO continual pre-training and Spanish question generation fine-tuning.