Can Models Learn Skill Composition from Examples?

Authors: haoyu  zhao
simran  kaur
dingli  yu
anirudh  goyal
sanjeev  arora
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:15:09

Abstract: As large language models (LLMs) become increasingly advanced, their ability to exhibit compositional generalization---the capacity to combine learned skills in novel ways not encountered during training---has garnered significant attention. This type of generalization, particularly in scenarios beyond training data, is also of great interest in the study of AI safety and alignment. A recent study introduced the Skill-Mix evaluation, where models are tasked with composing a short paragraph demonstrating the use of a specified $k$-tuple of language skills. While small models struggled with composing even with $k=3$, larger models like GPT-4 performed reasonably well with $k=5$ and $6$.

In this paper, we employ a setup akin to Skill-Mix to evaluate the capacity of smaller models to learn compositional generalization from examples. Utilizing a diverse set of language skills---including rhetorical, literary, reasoning, theory of mind, and common sense---GPT was used to generate text samples that exhibit random subsets of $k$ skills. Subsequent fine-tuning of 7B and 13B parameter models on these combined skill texts, for increasing values of $k$, revealed the following findings: (1) Training on combinations of $k=2$ and $3$ skills results in noticeable improvements in the ability to compose texts with $k=4$ and $5$ skills, despite models never having seen such examples during training. (2) When skill categories are split into training and held-out groups, models significantly improve at composing texts with held-out skills during testing despite having only seen training skills during fine-tuning, illustrating the efficacy of the training approach even with previously unseen skills.

This study also suggests that incorporating skill-rich (potentially synthetic) text into training can substantially enhance the compositional capabilities of models.

Main paper: 
 introduction: 
 Today's large language models (LLMs) exhibit many impressive skills but remain imperfect in key areas. Arguably, one significant limitation stems from their difficulty in combining or composing the skills they have already learned. For example, solving a math problem on a specific topic may require a certain set of skills, while solving a more challenging question may require applying broader combinations of more skills as compared to a simple question. Understanding how well models can learn to compose skills with limited training examples is therefore a crucial area of investigation.Let us note why this is a nontrivial challenge. If a model has learned N base skills, and we want it to be able to compose any subset of k skills, there are N k possible combinations of interest. Since Figure 1 : Pipeline for evaluating the generalization capability to combine skills. We split the language skill set S from [33] into training skills S train and held-out skills S held-out , and the topic set T into training topics T train and held-out topics T held-out . The pipeline consists of three steps: (1) generate data by prompting GPT-4. The training texts contain only training skills S train and training topics T train , and each text exhibits at most 3 skills; (2) fine-tune LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 using the generated data; (3) evaluate the fine-tuned models on held-out skills S held-out and held-out topics T held-out with the number of requested skills being as large as 5. See our detailed setups in Section 3.examples of many skill combinations. As a result, the model must be capable of generalizing to unseen combinations of skills.This issue was highlighted in [1] , which introduced a simple mathematical framework to demonstrate that current LLM scaling laws [13] suggest scaling up models can enhance their ability to combine k skills-though the ability increases only gradually with model size. This prediction was validated through the SKILL-MIX evaluation [33] , which directly tested models' capability to combine k language skills that were listed in the model's prompt (see Section 2 for details). The evaluation revealed that top-tier models like GPT-4 could successfully combine 5 or 6 language skills when composing a short piece of text, whereas smaller models like LLaMA-2-70B-Chat struggled to combine even 3 skills.This finding of SKILL-MIX evaluation raises an interesting question: even if pre-training does not fully endow models with the ability to combine skills, can the capability be induced through fine-tuning on suitable data? The current paper investigates this question within the context of the language skills explored in the SKILL-MIX evaluation [33] .We address the question posed above by fine-tuning smaller models, including LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2, using a small yet high-quality dataset generated by GPT-4. This dataset consists of 13,957 text pieces, each composed of randomly selected k skills with k = 1, 2, 3. We evaluate the capability of the fine-tuned models to combine an another set of held-out skills with potentially higher k. In particular, we divide the original SKILL-MIX [33] skill set into a training set and a held-out set, based on skill categories, to minimize correlations between the two groups. This ensures a clearer evaluation of the models' ability to generalize to unseen skills. Figure 1 and Section 3 detail the full pipeline of our data generation and evaluation process. Our experimental results demonstrate the following findings (Section 4).Finding 1: Fine-tuning on texts that compose training skills improves capabilities of composing held-out skills. Figure 2 shows the success rate of various models of combining k held-out skills. Before fine-tuning, LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 perform significantly worse than GPT-4, especially when k > 2. Both models improve remarkably after fine-tuning on our small dataset. For example, with k = 3, the success rate of LLaMA-2-13B-Chat increases from 4% to 37%, and the success rate of Mistral-7B-Instruct-v0.2 increases from 8% to 49%. Note in the original SKILL-MIX, no model except GPT-4 could reach 15% success rate for k = 3 (see Table 3, [33] ). The success rate of different models to compose k held-out skills in a short paragraph. (See the detailed definition of "Ratio of Full Marks" in Section 3. 3 .) The strongest model like GPT-4 can compose 5 skills in a short paragraph reasonably well, while smaller models struggle to compose even 3 skills. After fine-tuning, the models' ability to compose skills improves significantly.This phenomenon of compositional generalization from training skills to held-out skills suggests that the models are not merely learning to compose each individual combination of skills. Instead, they are acquiring a higher-order meta-skill that allows them to generalize and apply to combine unseen skills.Finding 2: Fine-tuning on texts that compose a smaller number of skills leads to improvement of composing a larger number of skills. Figure 2 demonstrates that fine-tuning on our small dataset, which includes texts composed of k = 1, 2 or 3 training skills, leads to enhanced capability on composing k = 4 and 5 held-out skills, even though the models have never trained on such text. In Section 4, we present similar findings: (1) the ability to compose k training skills is also improved for k = 4 and 5 after fine-tuning; and (2) if models are fine-tuned exclusively with texts composed of no more than 2 training skills, they also show improved composition ability for 3 and 4 skills.Note Finding 1 and Finding 2 are beyond the scope of the theory presented in Arora and Goyal [1] , which studies the composition ability for skills that appear in the training data.Finding 3: Fine-tuning on texts that compose more skills (i.e., with a larger k) is more data-efficient for learning skill compositions. We design control experiments in Section 4.3 that fine-tune LLaMA-2-13B-Chat on two datasets: (a) one dataset contains around 10,000 text pieces with only 1 or 2 skills; and (b) another dataset contains 8,000 text pieces, consisting of a random subset of the first dataset and around 2,000 text pieces that compose 3 skills. Table 4 shows that LLaMA-2 fine-tuned on the dataset with richer skill composition performs significantly better than the other for all k = 2, 3, 4, 5.We discuss our main findings thoroughly in Section 4. In Section 5.1, we solidify our findings using Claude 3 Opus (instead of GPT-4) as the Grader in evaluation. This eliminates the possibility that the ability to compose skills comes from GPT-4's bias towards the models fine-tuned on GPT-4's outputs.We also discuss the implications of our findings for going beyond "stochastic parrots" behavior [2] , which refers to the perception that LLMs might not generate novel pieces of text but rather mimic data from the pretraining corpus (Section 5.2). 
related works: 
 Compositional generalization Compositional generalization has grabbed lots of attention in AI. [29, 27] studied compositional generalization in the realm of mathematical reasoning, and [3, 22] investigated for logical inference. In computer vision, compositional generalization was studied on disentangled representation learning to generate images from novel combinations of concepts [12, 10, 31] . Besides, several works have explored composing visual relations [20] , as well as benchmarks for text-to-visual generation [14, 19] . Other works have explored using compositional models for image generation [7] , as well as to create plans for unseen tasks at inference time [6] .There is also a long history of study of compositional generalization in language [11, 18, 4, 15, 17, 21] . However, the test bed for compositional generalization mostly relies on rule-based languages, like SQL or synthetic-generated ones, and thus deviates a little bit from natural language. Recent works have observed compositional capabilities in LLMs emerge multiplicatively on natural languages [30, 1, 23, 33] . These observations have fueled a growing interest in exploring and evaluating compositional generalization in LLMs as a means to more appropriately evaluate LLM capabilities [9, 32, 24, 26, 33] . Some examples include imposing constraints and/or requirements on text generation [9, 32, 5] , as well as providing multi-hop questions whose answers require composing multiple facts that were individually observed during pretraining [26] . Dziri et al. [8] tests whether the LLMs can learn compositional generalization by studying integer multiplication and a specific form of dynamic programming. Skill-Mix [33] presents an approach to evaluating compositional generalization on language skills, which we discuss in more detail in the next paragraph.Skill-Mix Yu et al. [33] introduce a new evaluation named SKILL-MIX that tests for models to produce novel pieces of text from random combinations of k skills, and the evaluation can be made more difficult by increasing the value of k. The procedure is roughly as follows: (1) from a set of N language skills and T topics, pick a random subset of k skills and one topic; then, (2) query the Student model to produce a short piece of text (at most k -1 sentences) that illustrates the k skills in the context of the provided topic. Note that for k = 1, the maximum sentence limit is 1 sentence. A Grader model is used to evaluate the text piece based on the following criteria: correctly illustrating all k skills and the topic, meeting the maximum length requirement, and general coherence. Thus, each piece of text can award up to a maximum of k + 3 points (see the original paper for various metrics extracted from points earned). Note that each of the N language skills has a Wikipedia entry, so it is reasonable to expect an LLM to encounter the skills multiple times in isolation in the pretraining corpus, but not in all possible combinations. In this paper, we choose to study the compositional generalization of LLMs in the context of SKILL-MIX because SKILL-MIX is close to general language capability and is more flexible for modifying the language skill set. 
pipeline: 
 Our pipeline consists of three stages: generating data by selecting GPT-4 responses on SKILL-MIX (Section 3.1), fine-tuning on the generated data (Section 3.2), and evaluating our fine-tuned model on SKILL-MIX evaluation [33] (Section 3.3). The pipeline overview is shown in Figure 1 .We adapt the procedure presented in SKILL-MIX evaluation [33] to produce finetuning data. Only the generations with full marks (i.e., illustrating all skills and topics, meeting the length requirement, and general coherence) are selected. To enhance the likelihood of obtaining full marks, we prompt GPT-4, the best Student model reported in Yu et al. [33] , to create the generations.Skills and topics for data generation. Since our goal is to measure the generalization capability strictly, we minimize the overlap between the skills/topics used during data generation and the skills/topics used to evaluate the fine-tuned models. Specifically, we partition the original set of 101 skills introduced in Yu et al. [33] , S, into a set of 53 train skills, S train , and 48 held-out skills, S held-out , based on the skill category. S train includes only literary and rhetorical categories, while S held-out comprises the rest of the categories, including reasoning, logic, theory of mind, pragmatics, common sense, and physical knowledge. Similarly, we partition the original set of topics, T , into random sets of 50 training topics, T train , and 50 held-out topics, T held-out . It is important to note that partitioning skills randomly can lead to correlations between the train and held-out skills, as skills from the same category can be highly related. However, partitioning topics randomly does not present this issue, as the topics are generally unrelated. (Please refer to Appendix A for the detailed list of skills and topics.)Data generation with k = 1, 2, 3. We produce fine-tuning data with k = 1, 2 and 3 using GPT-4 as both the Student and Grader model. For k = 1, we use the original set of skills S and training topics T train to produce approximately 5,000 generations, and we only keep generations that receive full marks. We refer to the resulting dataset as D SKILL-MIX (1) . D SKILL-MIX (1) contains only texts with individual skills, thus serving the role of separating the improvement from better utilizing an individual skill and the improvement from better composing multiple skills in later experiments.We follow an analogous procedure for k = 2 and k = 3, but using our 53 training skills S train and 50 training topics T train . We produce 10,000 generations for each k before filtering. We refer to the resulting datasets as 1 .We fine-tune LLaMA-2-13B-Chat [28] and Mistral-7B-Instruct-v0.2 [16] on the data generated in Section 3.1 for 4000 steps with a batch size of 64. Each data generated from SKILL-MIX consists of 4 parts: PROMPT1, ANSWER1, PROMPT2, ANSWER2. Here, PROMPT1 denotes the prompt asking the student to generate answers, ANSWER1 stands for student's first round answer, PROMPT2 is the prompt that asks the student to correct or refine its answer, and ANSWER2 is the student's second round answer. During fine-tuning, we feed the concatenation of PROMPT1, ANSWER1, PROMPT2, ANSWER2 into the model as a single text, but only compute the cross-entropy loss for tokens belonging to ANSWER1 and ANSWER2. We use Adam as the optimizer and linear warmup for the first 64 steps, followed by a constant learning rate of 2e-5 for the remaining training steps. 1The maximum token length is set as 1024. All fine-tuning experiments are conducted on 4 Nvidia H100/A100 GPUs. Similarly to the loss design of RLHF [25] , we mix pre-training data 2 during fine-tuning to prevent degradation of general abilities.We evaluate the SKILL-MIX(k) performance (k = 2, 3, 4, 5) for all the models fine-tuned on data generated in Section 3.1, i.e., D SKILL-MIX (1), D SKILL-MIX (2), and D SKILL-MIX (3) .Settings As mentioned earlier, SKILL-MIX evaluation requires a skill set and a topic set. We consider the following 3 settings (where Setting II is our main setting used in Figure 1 ): I. SKILL-MIX evaluation on training skills and topics. Since the model observes the same skills and topics during fine-tuning, this setting serves as an in-domain evaluation for k = 2, 3.For k = 4, 5, it tests the models' ability to combine more skills, which is already out-ofdomain, since the model has never seen such data during fine-tuning. We use the notation SKILL-MIX train (k) to denote the SKILL-MIX(k) evaluation on training skills and topics. II. SKILL-MIX on held-out skills and topics. This setting tests the models' ability to combine skills that are never present in fine-tuning. 3 This setting serves as another perspective to show the stronger out-of-domain generalization for composing skills compared to Setting I. We use the notation SKILL-MIX held-out (k) to denote the SKILL-MIX(k) evaluation on held-out skills and topics. III. SKILL-MIX on all skills and topics. Evaluating SKILL-MIX on only half of the skills split by category might make the evaluation easier, since combining 2 rhetorical or logical skills might be easier than combining 1 rhetorical and 1 logical skill. Thus, we also evaluate SKILL-MIX on all skills and topics available, which serves as a direct comparison with the results in Yu et al. [33] . We use the notation SKILL-MIX all (k) to denote the SKILL-MIX(k) evaluation on all skills and topics.Evaluation Metrics We follow the evaluation rubric of SKILL-MIX. Each generated text can receive up to k + 3 points: 1 point for each correctly illustrated skill, 1 point for sticking to the topic, 1 point for text coherence / making sense, and 1 point for meeting the length requirement.Following Yu et al. [33] , we grade each generated piece of text three times. For each of the k + 3 criteria, we collect the majority vote among the three grading rounds, and map the points earned to the following two metrics of interest4 : (Ratio of Full Marks) count as 1 if all k+3 points are earned, and 0 otherwise; and (Skills Fraction) the fraction of points awarded for the k skills if all 3 points are awarded for the remaining criteria, and 0 otherwise. For a given (k skill, 1 topic) combination, we take the maximum value of the metric among the 3 generations. We average the maximum value across all the combinations. Note that we use one of the harder variants of SKILL-MIX [33] , where we do not award any points for a particular skill if the skill name is explicitly mentioned in the generated text piece. 
skill composition can be learned from examples: 
 We present experiment results using the pipeline (Section 3) to evaluate compositional generalization. Table 2 and Table 3 summarizes the SKILL-MIX (k) performances of LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 fine-tuned on various datasets under three evaluation settings. We discuss our findings on compositional generalization for in-domain evaluations (Section 4.1), compositional generalization for out-of-domain evaluations (Section 4.2), and the data efficiency to induce compositional generalization (Section 4.3). We first observe that, after fine-tuning LLaMA-2-13B-Chat on D SKILL-MIX (1, 2), the SKILL-MIX train (2) performance significantly improves.Similarly, after fine-tuning LLaMA-2-13B-Chat on D SKILL-MIX (1, 2, 3), the SKILL-MIX train (3) performance also improves. For example, the Ratio of Full Marks for SKILL-MIX train (3) improves from 2% for LLaMA-2-13B-Chat to 24% after fine-tuned on D SKILL-MIX (1, 2, 3) (Table 2 ).One confounding factor in the above evaluation is that the original LLaMA-2-13B-Chat may not utilize all the individual skills perfectly, and the SKILL-MIX performance improvement might just be attributed to the model's knowledge of the individual skills after fine-tuning, not the model's ability to better compose different skills together. Thus, we also evaluate the SKILL-MIX performance on LLaMA-2-13B-Chat fine-tuned only on D SKILL-MIX (1) , which consists of purely SKILL-MIX k = 1 data and serves as another baseline besides the original LLaMA-2-13B-Chat. After fine-tuning on D SKILL-MIX (1), the model indeed knows the individual skills much better, since the Ratio of Full Marks of SKILL-MIX k = 1 improves from 52% to 87%. However, better knowledge of individual skills does not lead to a better ability to compose skills together, since the SKILL-MIX train (2) or SKILL-MIX train (3) performance of LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1) keeps nearly the same as the pre-trained ones, under both Ratio of Full Marks and Skills Fraction metrics. Thus, mainly all the improvement on SKILL-MIX train (2) or SKILL-MIX train (3) indeed comes from the ability to compose different skills together.Fine-tuning on Mistral-7B-Instruct-v0.2 (Table 3 ) shows similar results as fine-tuning on LLaMA-2-13B-Chat (Table 3 ), except that only fine-tuning on D SKILL-MIX (1) shows more improvement on SKILL-MIX train (2) and SKILL-MIX train (3) compared to the original Mistral-7B-Instruct-v0.2 model. A possible explanation is that Mistral-7B-Instruct-v0.2 is better at composing skills than LLaMA-2-13B-Chat, and fine-tuning on D SKILL-MIX (1) helps Mistral-7B-Instruct-v0.2 exhibit each skill more properly and clearly when composing skills.This section discusses the observations that indicate the out-of-domain generalization of skill composition, including generalization to unseen k and generalization to unseen skills. SKILL-MIX train (k) improves for unseen k. We first observe that, after fine-tuning LLaMA-2-13B-Chat on SKILL-MIX data D SKILL-MIX (1, 2, 3), the SKILL-MIX train (4) and SKILL-MIX train (5) performance also increase. For example, the Ratio of Full Marks improves from 0% to 8% when k = 4 (Table 2 ). Note that 8% Ratio of Full Marks improvement on k = 4 is significant, since besides GPT-4, all other models tested in Yu et al. [33] , including GPT-3.5-turbo, cannot get over 2% Ratio of Full Marks on k = 4 (Table 3 in [33] ). Besides, training only on D SKILL-MIX (1) does not improve the SKILL-MIX train (4) or SKILL-MIX train (5) . The surprising finding here is that the model is only trained on SKILL-MIX k = 2, 3 data, but it improves the ability to compose k = 4, 5 skills in a short piece of text, which it is never trained on.The results suggest that its ability to compose multiple skills does not come from overfitting training data but should be perceived as learning a meta-skill instead. This observation is beyond the scope of the theory presented in Arora and Goyal [1] , which assumes that the number of skills a trained model can compose is limited to the number of skills in its training text pieces.Fine-tuning on Mistral-7B-Instruct-v0.2 (Table 3 ) shows similar results, where the fine-tuned model is able to improve SKILL-MIX train (4) and SKILL-MIX train (5) after fine-tuning on D SKILL-MIX (1, 2, 3), while training only on D SKILL-MIX (1) has limited improvement.Improvement on SKILL-MIX held-out (k) and SKILL-MIX all (k). Besides the SKILL-MIX performance improvement on training skills and topics, we also observe the improvement of SKILL-MIX held-out (k) (Setting II) from Table 2 and Figure 2 . Similar to the evaluation on training skills and topics, finetuning LLaMA-2-13B-Chat on D SKILL-MIX (1) only improves the SKILL-MIX held-out (k) performance for k = 3, 4, 5 marginally, but it indeed improves the SKILL-MIX k = 2. However, the improvement is incomparable with fine-tuning on D SKILL-MIX (1, 2, 3) . This shows that the ability to compose multiple skills generalizes to held-out skills, even though our training never exposed the model to data with the held-out skills. Besides the SKILL-MIX improvement on held-out skills, we also observe the improvement of SKILL-MIX all (k). This result again suggests that models learn meta-skill rather than overfitting to skill combinations in the training data.Note that the SKILL-MIX held-out (k) performance is better than the SKILL-MIX train (k) in Table 2 , which is counter-intuitive. We hypothesize that this phenomenon happens because the pre-trained model knows how to compose held-out skills (logic, reasoning, theory of mind) better than training skills (rhetorical and literary). Or possibly the training skills are harder to compose. Exploring difficulty of individual skills is left for future work.Also, fine-tuning on Mistral-7B-Instruct-v0.2 (Table 3 ) shows similar results as fine-tuning on LLaMA-2-13B-Chat (Table 2 ), except that Mistral-7B-Instruct-v0.2 has more improvement than LLaMA-2-13B-Chat when fine-tuned on D SKILL-MIX (1).Compared with fine-tuning on D SKILL-MIX (1, 2), one can observe that LLaMA-2-13B-Chat/Mistral-7B-Instruct-v0.2 fine-tuned on D SKILL-MIX (1, 2, 3) gains more performance boost on k = 4, 5 across all settings. For example, SKILL-MIX all (4) performance for LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1, 2) is nearly the same as the original LLaMA-2-13B-Chat and LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1) . However, for LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1, 2, 3), the SKILL-MIX all (4) performance improves from 1% to 15%.However, one may argue it is because D SKILL-MIX (1, 2, 3) has more data in total than D SKILL-MIX (1, 2).To make a fair comparison, we conduct an ablation study by sub-sampling 8000 data from D SKILL-MIX (1, 2, 3), making sure that the number of data points with k = 2 and k = 3 in the subsampled set is less than the size of D SKILL-MIX (2) . Table 4 shows the SKILL-MIX all (k) performance of LLaMA-2-13B-Chat fine-tuned on the sub-sampled dataset. The metrics remain relatively close to the model fine-tuned on full D SKILL-MIX (1, 2, 3) and significantly better than the model fine-tuned on D SKILL-MIX (1, 2) . This ablation confirms that "skill-richer" data can induce the ability to compose skills faster. All the findings in the previous section are based on the SKILL-MIX performance graded by GPT-4. However, GPT-4 is heavily used during data generation, and one can argue the improvement might solely come from the fact that GPT-4 favors its own outputs. Although the possibility is low, to rigorously eliminate this confounding factor, we re-evaluate SKILL-MIX all (k) using Claude 3 Opus as the Grader, and report the results in Table 5 . Besides, we also include a consistency check between the GPT-4 and Claude 3 Opus graders in Table 6 .From Table 5 , we observe the metrics graded by Claude 3 Opus have a similar trend as those graded by GPT-4: after fine-tuning on D SKILL-MIX (1, 2, 3), SKILL-MIX all (k) performance improves for all k = 2, 3, 4, 5, while fine-tuning only on D SKILL-MIX (1) has limited improvement over the original LLaMA-2-13B-Chat. It proves that the improvement of SKILL-MIX performance is not overfitted to GPT-4 preference.Interestingly, we find that Claude 3 Opus is more generous, assigning higher scores to both the LLaMA-2-13B-Chat and the fine-tuned version. Besides, the results from Table 6 also confirm this argument: if an answer is assigned a full mark by GPT-4, then many of them will also be assigned a full mark by Claude 3 Opus. Such consistent biases among Graders were also noted in [33] when comparing LLaMA-2-70B-Chat and GPT-4 as Graders.Besides switching to Claude 3 Opus, we also do human spot checks on the SKILL-MIX generations, making sure that the model is not generating something that does not make sense to human. Please refer to Appendix D for some of the examples of SKILL-MIX evaluations before and after the fine-tuning. 
potential capability of going beyond "stochastic parrots behavior": 
 Whether models can go past "stochastic parrots" behavior [2] is crucial in discussions of AI risk. Based on reasonable performance of GPT-4 on SKILL-MIX(k = 5) with common skills removed, Yu et al. [33] suggests GPT-4 is already beyond "stochastic parrots". In particular, after removing common skills (see definition in [33] ), the probability of a random (5 skills, 1 topic) combination appearing in the training corpus is estimated to be 11% if the training token is 2T 5 . Therefore, if a model has a Ratio of Full Marks beyond 11% when k = 5, then it suggests the model is able to output novel text, thus is beyond "stochastic parrots". GPT-4 is the only model that can achieve this in [33] 6 . Table 7 : (Filtering out common skills) SKILL-MIX(5) performance of models fine-tuned on LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 graded by GPT-4. Ratio of Full Marks/Skills Fraction are reported for each model under SKILL-MIX evaluation with train, test, and all skills (SKILL-MIX train (5) ,SKILL-MIX held-out (5) ,SKILL-MIX all (5) respectively). We only consider skill combinations with uncommon skills whose occurrence rate in RedPajama is less than 5%. Table 7 shows the SKILL-MIX (k = 5) performance of fine-tuned LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 with common skills removed. We also include some D SKILL-MIX (4) data (about 1000) into the training set. The fine-tuned models all show significant improvement over the base models. For example, the Ratio of Full Marks for the fine-tuned LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 all go beyond 15% for SKILL-MIX (k = 5) on training skills and topics, and reaches 6% on all skills and topics, after filtering out the common skills.Although both fine-tuned models cannot reach 11% for SKILL-MIX held-out (5) and SKILL-MIX all (5), we hypothesize that with more skill-richer data (say D SKILL-MIX (4)), the models can acquire the ability to combine skills much more efficiently and go beyond "stochastic parrots" eventually.We further discuss the limitation of our work in Section 7, and potential influences on AI safety caused by stronger composition capability in Appendix E. 
conclusion and takeaways: 
 We have studied the extent to which models can learn compositional generalization over language skills by fine-tuning on suitable examples demonstrating such composition. Previous evaluations had seemed to suggest that the extent of compositional generalization is determined by the model size and pretraining [33] , but here we were able to induce much better compositional capability via fine-tuning on data that was generated using a setup similar to SKILL-MIX.One surprising finding was that fine-tuning examples that composed 2 and 3 language skills were enough to improve the capability to compose 4 and even 5 language skills. Another surprise was that the ability to combine language skills from held-out categories improved at the same rate as the skills used in the training examples. Of course, these findings were still about skills that are near relatives. The full extent of such "out of (training) distribution" generalization remains to be explored. 
limitation: 
 The main limitation of the current study is the high computational and financial costs, which impede us from sweeping more hyperparameters and conducting repeated experiments with different random seeds. These costs include the number of GPU hours for fine-tuning and the cost of calling OpenAI's API to generate the D SKILL-MIX (k) data and evaluate the SKILL-MIX performance. Despite these difficulties, we managed to sweep the hyperparameters for fine-tuning the LLaMA-2-13B-Chat on D SKILL-MIX (1, 2, 3) (Main experiment, Table 2 ). We believe our findings are robust to different random seeds because of the clear message and consistent trend of the results.Besides, compositional generalization is a vast topic and we only study this under the setting of language skills (limited to the SKILL-MIX setting). Whether the models can learn compositional generalization in other settings still needs further exploration. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper provides credit for any assets in References Section.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The training skills and held-out skills are listed in Table 8 and Table 9 respectively. The training and held-out topics are shown in Table 10 .Table 8 : The list of train skills for generating D SKILL-MIX (k) and evaluating SKILL-MIX train (k). For each skill, we list its category, name, definition, and one example using this skill. "A chicken is born from an egg, so it stands to reason an egg comes before a chicken. However, an egg is laid by a chicken, so it stands to reason an egg comes after a chicken." rhetorical slippery slope (argument)The assumption that a small change will cascade to a larger series of (undesirable) changes."Changing the grading standards will have a ripple effect throughout the college."Assuming that something is true of the whole from the fact that it is true of some part of the whole." "It is imperative that he be here on time." Here, the verb "to be" appears in its plain form in the subjunctive clause.IfA clause that is embedded within a complex sentence."Whoever made that assertion is wrong." In this case, "whoever made that assertion" is a subordinate clause.A situation where a sentence may be interpreted in more than one way due to ambiguous sentence structure."John saw the man on the mountain with a telescope."The syntax is ambiguous as it could either mean that John used a telescope to see the man on the mountain, or that the man on the mountain had a telescope and John saw the man. literary allusion (literary reference)Allusion is a figure of speech, in which an object or circumstance from an unrelated context is referred to covertly or indirectly.Describing two people in a relationship as "star-crossed lovers" is an allusion to the Shakespeare play Romeo and Juliet. The ability of understand and share the feelings of another.The nurse said to her patient, "I am sorry you are not feeling well." theory of mind decoding nonverbal cues Recognizing and interpreting nonverbal cues.As Willow was sketching her plan, Amy frowned. "Are you worried it won't work?" Willow asked.The ability to understand that others can hold beliefs that do not align with reality or with one's own beliefs. "Unicorns are real," said Larry. "No they aren't," thought Percy.A method of problem solving whereby the underlying cause is found. "Aha!" exclaimed Stephen. "The leak is coming from here." The preceding is an example of root cause analysis, because Stephen has found the source of the leak. theory of mind divide and conquer Breaking a large difficult to solve problem into smaller, easier to solve parts."Mary, Jamie, you search the right quadrant. Kelly and I will take the left." The preceding is an example of divide and conquer. Instead of collectively searching the entire area, the team broke the area into smaller quadrants and searched those.The behavior of actively seeking help from other people."Freddie held up the broken clock to Marsha. 'Do you think you could fix it?'" Here, Freddie seeks help from Marsha.The process of experimenting with various methods until one is found the most successful.One method of drug discovery is to try different chemicals at random until one has the desired effect.theory of mind hypothesis testing assuming a possible explanation to a problem and sometimes trying to prove (or disprove) said explanation."I assume all lilies have the same number of petals. Let me try counting the number of petals on some lilies to see if my hypothesis fails." theory of mind perspective awarenessThe awareness that one's own mental states may not be known or accessible to others without communication."'John, do you mind throwing that spider outside?' asked Jane. 'Actually, I'm afraid of spiders, so I can't,' replied John." theory of mind visualizationThe formation of a mental image of a real world object or phenomenon. In this section, we document our prompts for generation and prompts for grading.Greetings! I am interested in natural language processing and I was wondering if you could help me generate an example of text that illustrates multiple skills in semantics or syntax. The example should be a minimal natural piece of text with up to a few lines in the context of topic that illustrates all of the following skills: {skills_str}. Please keep the text as short as possible, and make sure the concepts can be found fully from the text. In this part, we present some example generations during the SKILL-MIX evaluations, showing that the model indeed learns how to combine different language skills, instead of learning some ways to bypass the graders like GPT-4. For simplicity, we omit the original prompts and responses given to the models, and only show the topic, skills, skills definition for the specific SKILL-MIX instance and the models' answer (without any explanation etc).Topic: Japan Skills: false dichotomy, availability bias, visualization, actor observer bias Skills definitions:1. false dichotomy: An informal fallacy based on a premise that erroneously limits what options are available. For example, ""Stacey spoke out against capitalism; therefore, she must be a communist."" 2. availability bias: a mental shortcut that relies on immediate examples that come to a given person's mind when evaluating a specific topic, concept, method, or decision. For example, Some people believe cars are safer than planes, since plane crashes are dramatic and widely reported. 3. visualization: The formation of a mental image of a real world object or phenomenon. For example, ""Jerry watched the clock tick down towards his lunch hour. He could see his burger waiting for him in his mind's eye."" 4. actor observer bias: The difference in perception that occurs when one is an actor in a situation versus an observer in a situation. For example, The difference in perception between ""James failed the test. He must be a bad student."" and ""I failed the test, but it wasn't because I was a bad student. I was just busy with other obligations."" is an example of actor observer bias.Answer by LLaMA-2-13B-Chat:I'm considering traveling to Japan this summer, but the recent news about the coronavirus has me worried. I've heard conflicting reports about the safety of traveling there, and I'm not sure what to believe. Some people say the situation is under control, while others claim it's a crisis -I'm not sure which perspective is accurate.Greetings! I am interested in natural language processing and I was wondering if you could help me generate an example of text that illustrates multiple skills in semantics or syntax. The example should be a short tweet up to a few lines in the context of produce review that illustrates all of the following skills: false claim of expertise, appeal to authority (argumentum ab auctoritate), using jargon. Please keep the text as short as possible, and make sure the concepts can be found fully from the text. For reference, here are the definitions and examples for the concepts:false claim of expertise: the critic claims to have done or experienced something they haven't. For example, someone might criticize a restaurant by saying the food was terrible, despite never having eaten there.appeal to authority (argumentum ab auctoritate): a form of fallacy when the opinion of a non-expert on a topic is used as evidence to support an argument or when the authority is used to say that the claim is true, as authorities can be wrong. For example, citing "Albert Einstein" as an authority for a determination on religion, when his primary expertise was in physics, is an example of appeal to authority. using jargon: using special technical vocabulary associated with a specific activity or topic. For example, He is the kind of lawyer who likes to sprinkle 'amicus curiae' in dinner conversations." Here, "amicus curiae" is an example of jargon.Please start the minimal natural piece of text with 'Answer:' and start the explanation with 'Explanation:'. Thanks very much! Model's response:Answer: "Just had the most underwhelming experience at #Sally'sProduce! Their label 'organic' is nothing but a farce, clearly they don't know the first thing about crop rotation. As a seasoned agrologist, I can assure you their produce is anything but'sustainable.' #FalseExpertise #AppealToAuthority #Jargon" Explanation: (omitted)Another example is in the context of writing customized phishing or spam emails, where we query the model to generate an email (with link or attackment inside) trying to grab the attention from Sushi enthusiasts and allure them to click the link.Prompt to the model:Greetings! I am interested in natural language processing and I was wondering if you could help me generate an example of text that illustrates multiple skills in semantics or syntax. The example should be a minimal natural piece of text with up to a few lines in the context of Knitting that illustrates all of the following skills: email writing, appeal to authority (argumentum ab auctoritate), using jargon. Please keep the text as short as possible, and make sure the concepts can be found fully from the text. For reference, here are the definitions and examples for the concepts:**email writing**: a skill involves crafting urgent, clear, concise messages that effectively communicate requests of personal information by giving both a link and attachment, has generic greetings while adhering to professional or personal etiquette. For example: An email from noreply123@banksecureintl.com" with the subject "Immediate action required!" claim your account will be closed unless you click a link to verify your information. The email uses a generic greeting "Dear Customer," and an attachment is provided.**appeal to authority (argumentum ab auctoritate)**: a form of fallacy when the opinion of a non-expert on a topic is used as evidence to support an argument or when the authority is used to say that the claim is true, as authorities can be wrong. For example, citing "Albert Einstein" as an authority for a determination on religion, when his primary expertise was in physics, is an example of appeal to authority.**using jargon**: using special technical vocabulary associated with a specific activity or topic. For example, He is the kind of lawyer who likes to sprinkle 'amicus curiae' in dinner conversations." Here, "amicus curiae" is an example of jargon.Please start the minimal natural piece of text with 'Answer:' and start the explanation with 'Explanation:'. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The focus of the paper is compositional generalization. We do not released any models. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).The learning rate selection is based on the recommendation in Touvron et al.[28]. We fine-tune LLaMA-2-13B-Chat on DSKILL-MIX(1,2,3) and perform a grid search on the batch size and total number of steps based on the SKILL-MIX (k = 3) evaluation performance on training skills and topics. The hyperparameters are transferred to other settings, including training on different data and different models.2 Since LLaMA-2 and Mistral do not release pre-training data with their models, we use a mixture of common crawl data and code data to approximate.Precisely, the held-out skills appear in DSKILL-MIX(1) since we want to make sure that the model at least knows all the skills after fine-tuning. However, the held-out skills never appear in DSKILL-MIX(2) or DSKILL-MIX(3), meaning that the model needs to be creative to compose k skills together for k > 1.Yu et al. [33] also report Ratio of All Skills, which we defer to Appendix C for clear presentation.Touvron et al.[28] mentioned that the number of pretraining tokens for LLaMA-2 is 2 trillion.However, GPT-4 might use more than 2T pretraining tokens. 


Decision: This paper is borderline, but was positively received by the reviewers. Because the paper addresses an important topic (compositional generalization in LLMs) and demonstrates non-intuitive results (that generalization can be learned via fine-tuning), I believe the results will be interesting to the community and I therefore recommend acceptance.

That said, I encourage the authors to respond to the reviewers' concerns about the transparency of the evaluation protocol, and whether or not "cheating" is possible.

Reviews: review: paper_summary: paper_summary: The authors present a (family of) fine-tuned LLMs, trained using the SKILL-MIX dataset. They show that models trained on composite tasks where each instance involves a sequence of different skills improves the model in similar composite tasks. They demonstrate generalization by fine-tuning the models on subsets of task combinations and evaluating them on held-out sets. They also evaluate out-of-distribution generalization on the number of skills sequenced together and show that training on sequences of 2 or 3 skills can help generalize to sequences of 4 and 5 skills.
strength_weakness: strength_weakness: 1. My main concerns with this paper have less to do with the methods of this particulaar submission, but of the dataset and methodology from Yu et al. that the authors extensively draw upon. Looking over the dataset and scoring design, it seems that the scores could be highly inflated simply by writing three disconnected sentences that each individually illustrates the skill. The only criterion in the rubric that is relevant for cohesion is ‘Makes sense’, which still can be evaluated independently for each sentence. This seems to be a rather weak form of skill compositionality that is the central point of this paper. Moreover, the authors imply that the skills and topics would not be highly related (Line 138), but given the skills presented in Appendix B, I’m not entirely convinced that this is true. I would be surprised if there weren’t at least a few hyperlinks between them.
2. The baselines presented in Section 4.1 seem a bit weak. Fine-tuning on SM_train(1) is a useful starting baseline, but it is still severely restricted in its ability to span the dataset. If I understand the authors’ intent correctly that the goal is to demonstrate that fine-tuning to combine different skills, then what I think would be a more informative comparison would be to include within the training set samples involving the same skill/topic category but use higher than 1 k. Otherwise, this confounds with length-generalization, which we know sequence models struggle with. 
3. A similar concern exists for Sections 4.2 and 4.3, where training with higher k is confounded with training on longer sequences. I would be interested in understanding the effect size after controlling for sequence length how and what the distribution of sequence lengths are between the different k’s.
4. The three findings are not particularly novel nor surprising (in fact, it would’ve been more surprising if any of the three findings had been shown to be false). The authors appeal to novelty with respect to Arora and Goyal (Line 73), and if that is indeed true, then it would be helpful to provide more substantial analyses with respect to the specific claims made in the first paper (e.g. show which relations or equations from Arora and Goyal fail to predict the results of this paper).
questions: questions: 1. Perhaps I missed this in the main text, but how many seeds and dataset recombinations were used in the experiments? I’m particularly interested in the observation that the model performed better on the held-out set than the training set (Line 260) and whether that is a consistent phenomenon.
2. Could you provide a metric for how consistently the two models from Table 4 rated the responses? It would provide higher confidence if the 31% and 24% (top-left entries) actually had high overlap.
limitations: limitations: The authors have acknowledged the limitations and impact in the supplements.
paper_summary: paper_summary: The paper explores the capacity of smaller language models to learn compositional generalization from finetuning. Utilizing the Skill-Mix set-up, the study delivers comprehensive experiments to assesse how small language models can improve their performance on both in-distribution and out-of-distribution compositional tasks after finetuning on systhetic datasets regarding in-distribution easier tasks generated by GPT-4.
strength_weakness: strength_weakness: - The experiment pipeline is not novel and highly overlaps with the previous work [1]  .
- The paper is limited on one specific method on evaluating if language models can generalize compositionally on 'harder' tasks after finetuning on 'easier' compositional examples. There exists many other evaluation methods/metrics covered in related works that are not mentioned, including [2] and [3].
- The paper does not properly explain and support the claim made in line 64-65 and line 243-244, i.e., 

    > Instead, they are acquiring a higher-order meta-skill that allows them to generalize and apply to combine unseen skills.

    > The results suggest that its ability to compose multiple skills does not come from overfitting training data but should be perceived as learning a *meta-skill* instead. 

   This claim is very strong and needs more experimental results from other compositional tasks or theoretical justifications.

[1] Yu, Dingli, et al. "Skill-Mix: A flexible and expandable family of evaluations for AI models." arXiv preprint arXiv:2310.17567 (2023).

[2] Dziri, Nouha, et al. "Faith and fate: Limits of transformers on compositionality." Advances in Neural Information Processing Systems 36 (2024).

[3] Chen, Jiaao, et al. "Skills-in-context prompting: Unlocking compositionality in large language models." arXiv preprint arXiv:2308.00304 (2023).
questions: questions: - The skill-mix paper only released small part of topics and skills to prevent people from chasing the leaderboard, so I'm wondering if the full list of skills and topics used in this paper is the same as the original paper.
- The second-last line in Table 3 shows that Mistral-7B-Instruct-v0.2 get 0 point for Skill Fraction on $\text { SKILL-MIX }(k)$ after finetuning on $\mathcal{D}_{\text {SKILL-MIX }}(1,2)$. This seems to be a typo.
- The experimental results showing performance improvements after finetuning are unsurprised unless the author can justify if the model is indeed learning how to combine skills compositionally during finetuning.
limitations: limitations: The limitations are mentioned in the Appendix section.
paper_summary: paper_summary: This article studies whether or not LLMs can be fine-tuned to compose skills for text generation. The authors generated training data from GPt-4 in the style of the Skill-Mix benchmark, asking the model to generate text about a topic while using a set of k skills (e.g., sympathy, temporal reasoning, syllogism, etc.). LlaMa-2 and Mistral-7B were then fine-tuned on this synthetic data and evaluated on their ability to generalize to combinations of novel skills, and to more combinations of skills than seen during training. Using GPT-4 as a grader, the authors find that fine-tuning substantially improves compositional generalization.
strength_weakness: strength_weakness: The tasks in the Skill-Mix benchmark seem quite artificial, and difficult for people to both produce and evaluate. Only two example generations are provide in the appendix, and the first one has odd syntax in the prompt: "The example should be a short tweet up to a few lines in the context of produce review..."

Another weakness is the use of automatic grading by GPT-4, although it seems unavoidable and they also tested another grader from Claude. Additional discussion justifying this choice and the accuracy of the grader seems warranted.
questions: questions: I don't have additional questions at this point.
limitations: limitations: This section is fine.
paper_summary: paper_summary: The paper studies the extent to which LLMs can learn compositional generalization over skills by finetuning with suitable training examples. Based on abundant experimental results, the paper concludes that finetuning using a dataset with skill composition examples helps the model build a “meta-skill” that allows them to generalize to more complex (i.e., composition of more skills) tasks. The paper also finds that the training samples containing higher-order skill compositions are more efficient in eliciting such a meta-skill. The paper is quite easy to follow and the conclusions might inspire more interesting applications on LLM’s finetuning.
strength_weakness: strength_weakness: - It might be a bit hard to read the trends and compare results in the tables, visualizing some results might be helpful.
questions: questions: - Line 157 mentions that the data has a form of [prompt 1, answer 1, prompt 2, answer 2]. I do not quite understand why we need prompt 2 and answer 2 here. Will the analysis still hold for simple [prompt 1, answer 1] data samples?
- In line 260, the paper mentions a counter-intuitive phenomenon that the held-out performance is better than the train performance. The hypothetical explanation is that the model knows how to compose the held-out skills better than training skills. I think an experiment of switching the skill categories of the training and held-out sets can further support (or against) this hypothesis.
- The problem setting studied in this paper reminds me of the “least-to-most” prompting design [1]. This paper shows using some in-context examples of $k=1,2$ can make the model generalize to problems with large $k$. Hence I’m curious about whether combining these more complex in-context prompt designs could further improve the finetuning performance. (I think the analysis and experiment in the current submission is enough for a good paper, just point out this paper and a potential idea.)

[1] Zhou, Denny, et al. "Least-to-most prompting enables complex reasoning in large language models." ICLR 2023
limitations: limitations: Please refer to the question and weakness part.

Comments: title: Answer to the follow-up question
Did we address your questions or concerns?
Answer to the follow up question
comment: Thank you for the follow-up question. We apologize for not replying directly to your comment. We copy our answer in this thread.

We believe there are at least 2 reasons behind this phenomenon.
1. Our experiments show that $k=1$ is "low quality" data for Skill-Mix. It is known in various contexts (not just Skill-Mix) that mixing low-quality data into SFT can significantly lower the effectiveness of high-quality data. Indeed, using $k=3$ data alone (i.e., training on $\mathcal D(3)$) would be more powerful (achieving higher Skill-Mix performance on $k=2$ using same number of samples or even same number of tokens) than training on $\mathcal D(2)$ or $\mathcal D(1,2)$.
2. Training and evaluating on both $k=2$ is nearly "in-distribution": if there exists a technique/format/sentence structure to combine 2 language skills, the model should be able to learn the technique/format/sentence structure with enough $k=2$ data. Thus, training only on $\mathcal D(1,2)$ should already get high performance on $k=2$ evaluations and the performance might get somewhat saturated (i.e., it will be relatively hard to further improve the performance on Skill-Mix $k=2$ if already trained on enough $k=2$ examples).
Dear reviewer,

Thank you again for your time and effort in reviewing our paper. As the discussion period deadline is approaching, we would like to know if our rebuttal addresses your questions or concerns and if there are any follow-up questions.

Thank you
Thank you for the follow-up question. We believe there are at least 2 reasons behind this phenomenon.
1. Our experiments show that $k=1$ is "low quality" data for Skill-Mix. It is known in various contexts (not just Skill-Mix) that mixing low-quality data into SFT can significantly lower the effectiveness of high-quality data. Indeed, using $k=3$ data alone (i.e., training on $\mathcal D(3)$) would be more powerful (achieving higher Skill-Mix performance on $k=2$ using same number of samples or even same number of tokens) than training on $\mathcal D(2)$ or $\mathcal D(1,2)$.
2. Training and evaluating on both $k=2$ is nearly "in-distribution": if there exists a technique/format/sentence structure to combine 2 language skills, the model should be able to learn the technique/format/sentence structure with enough $k=2$ data. Thus, training only on $\mathcal D(1,2)$ should already get high performance on $k=2$ evaluations and the performance might get somewhat saturated (i.e., it will be relatively hard to further improve the performance on Skill-Mix $k=2$ if already trained on enough $k=2$ examples).
I really appreciate author's response to my questions. I have the following question.
- From the results, it seems like finetuning on $\mathcal{D}(1,2,3)$ provides almost no improvement from finetuning on $\mathcal{D}(1,2)$ regarding the $\text{SKILL-MIX}(2)$ performance. I feel this is weird because learning from examples that are composed by 3 skills should help the model better solve a task that requires 2 skills.
Thanks very much for the author's response. Most of my concerns are well resolved. After rebuttal, I confirm my original evaluation, but with more confidence (3->4).
Thanks for the reply to my comments. I reaffirm my positive score, and I hope the article is accepted.