Title: Can Models Learn Skill Composition from Examples?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Skill Composition

Large Language Model
Abstract: As large language models (LLMs) become increasingly advanced, their ability to exhibit compositional generalization---the capacity to combine learned skills in novel ways not encountered during training---has garnered significant attention. This type of generalization, particularly in scenarios beyond training data, is also of great interest in the study of AI safety and alignment. A recent study introduced the Skill-Mix evaluation, where models are tasked with composing a short paragraph demonstrating the use of a specified $k$-tuple of language skills. While small models struggled with composing even with $k=3$, larger models like GPT-4 performed reasonably well with $k=5$ and $6$.

In this paper, we employ a setup akin to Skill-Mix to evaluate the capacity of smaller models to learn compositional generalization from examples. Utilizing a diverse set of language skills---including rhetorical, literary, reasoning, theory of mind, and common sense---GPT was used to generate text samples that exhibit random subsets of $k$ skills. Subsequent fine-tuning of 7B and 13B parameter models on these combined skill texts, for increasing values of $k$, revealed the following findings: (1) Training on combinations of $k=2$ and $3$ skills results in noticeable improvements in the ability to compose texts with $k=4$ and $5$ skills, despite models never having seen such examples during training. (2) When skill categories are split into training and held-out groups, models significantly improve at composing texts with held-out skills during testing despite having only seen training skills during fine-tuning, illustrating the efficacy of the training approach even with previously unseen skills.

This study also suggests that incorporating skill-rich (potentially synthetic) text into training can substantially enhance the compositional capabilities of models.

Can Models Learn Skill Composition from Examples?

Abstract

As large language models (LLMs) become increasingly advanced, their ability to exhibit compositional generalization-the capacity to combine learned skills in novel ways not encountered during training-has garnered significant attention. This type of generalization, particularly in scenarios beyond training data, is also of great interest in the study of AI safety and alignment.

Arecent study introduced the SKILL-MIX [33] evaluation, where models are tasked with composing a short paragraph demonstrating the use of a specified k -tuple of language skills. While small models struggled with composing even with k = 3 , larger models like GPT-4 performed reasonably well with k = 5 and 6 .

In this paper, we employ a setup akin to SKILL-MIX to evaluate the capacity of smaller models to learn compositional generalization from examples. Utilizing a diverse set of language skills-including rhetorical, literary, reasoning, theory of mind, and common sense--GPT-4 was used to generate text samples that exhibit random subsets of k skills. Subsequent fine-tuning of 7B and 13B parameter models on these combined skill texts, for increasing values of k , revealed the following findings:

Training on combinations of k = 2 and 3 skills results in noticeable improvements in the ability to compose texts with k = 4 and 5 skills, despite models never having seen such examples during training.

When skill categories are split into training and held-out groups, models significantly improve at composing texts with held-out skills during testing despite having only seen training skills during fine-tuning, illustrating the efficacy of the training approach even with previously unseen skills.

This study also suggests that incorporating skill-rich (potentially synthetic) text into training can substantially enhance the compositional capabilities of models.

1 Introduction

Today's large language models (LLMs) exhibit many impressive skills but remain imperfect in key areas. Arguably, one significant limitation stems from their difficulty in combining or composing the skills they have already learned. For example, solving a math problem on a specific topic may require a certain set of skills, while solving a more challenging question may require applying broader combinations of more skills as compared to a simple question. Understanding how well models can learn to compose skills with limited training examples is therefore a crucial area of investigation.

Let us note why this is a nontrivial challenge. If a model has learned N base skills, and we want it to be able to compose any subset of k skills, there are ( N k ) possible combinations of interest. Since ( N k ) scales roughly with the k -th power of N , even reasonable-sized training datasets will likely omit

(1) Data Generation

Query GPT-4: Generate a short text about the

selected topic that exhibits all the selected skills

/u1D4AE

train

Grade by GPT-4 and Filter

/u1D4AF

train

Select random

subset of k skills

and 1 topic

(

)

k

= 1,2,3

Training Data

/u1D4AE

= /u1D4AE

train

∪ /u1D4AE

held-out

Topic Set

/u1D4AF

= /u1D4AF

train

∪ /u1D4AF

held-out

(3) Evaluation

the selected topic that exhibits all the selected skills

Held-out Skills

/u1D4AE

held-out

Graded by GPT-4/Claude-3

Held-out Topics

/u1D4AF

held-out

Select random

subset of k skills

and 1 topic

(

)

k

= 1,2,3,4,5

(2) Fine-tune

examples of many skill combinations. As a result, the model must be capable of generalizing to unseen combinations of skills.

This issue was highlighted in [1], which introduced a simple mathematical framework to demonstrate that current LLM scaling laws [13] suggest scaling up models can enhance their ability to combine k skills-though the ability increases only gradually with model size. This prediction was validated through the SKILL-MIX evaluation [33], which directly tested models' capability to combine k language skills that were listed in the model's prompt (see Section 2 for details). The evaluation revealed that top-tier models like GPT-4 could successfully combine 5 or 6 language skills when composing a short piece of text, whereas smaller models like LLaMA-2-70B-Chat struggled to combine even 3 skills.

This finding of SKILL-MIX evaluation raises an interesting question: even if pre-training does not fully endow models with the ability to combine skills, can the capability be induced through fine-tuning on suitable data? The current paper investigates this question within the context of the language skills explored in the SKILL-MIX evaluation [33].

1.1 Our contributions

We address the question posed above by fine-tuning smaller models, including LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2, using a small yet high-quality dataset generated by GPT-4. This dataset consists of 13,957 text pieces, each composed of randomly selected k skills with k = 1 , 2 , 3 . We evaluate the capability of the fine-tuned models to combine an another set of held-out skills with potentially higher k . In particular, we divide the original SKILL-MIX [33] skill set into a training set and a held-out set, based on skill categories, to minimize correlations between the two groups. This ensures a clearer evaluation of the models' ability to generalize to unseen skills. Figure 1 and Section 3 detail the full pipeline of our data generation and evaluation process. Our experimental results demonstrate the following findings (Section 4).

Finding 1 : Fine-tuning on texts that compose training skills improves capabilities of composing held-out skills. Figure 2 shows the success rate of various models of combining k held-out skills. Before fine-tuning, LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 perform significantly worse than GPT-4, especially when k > 2 . Both models improve remarkably after fine-tuning on our small dataset. For example, with k = 3 , the success rate of LLaMA-2-13B-Chat increases from 4% to 37% , and the success rate of Mistral-7B-Instruct-v0.2 increases from 8% to 49% . Note in the original SKILL-MIX, no model except GPT-4 could reach 15% success rate for k = 3 (see Table 3, [33]).

2

3

4

5

0

20

40

60

80

Ratio of Full Marks

LLaMA 13B Chat

Mistral 7B Instruct

Fine-tuned LLaMA 13B Chat

Fine-tuned Mistral 7B Instruct

k

This phenomenon of compositional generalization from training skills to held-out skills suggests that the models are not merely learning to compose each individual combination of skills. Instead, they are acquiring a higher-order meta-skill that allows them to generalize and apply to combine unseen skills.

Finding 2 : Fine-tuning on texts that compose a smaller number of skills leads to improvement of composing a larger number of skills. Figure 2 demonstrates that fine-tuning on our small dataset, which includes texts composed of k = 1 , 2 or 3 training skills, leads to enhanced capability on composing k = 4 and 5 held-out skills, even though the models have never trained on such text. In Section 4, we present similar findings: (1) the ability to compose k training skills is also improved for k = 4 and 5 after fine-tuning; and (2) if models are fine-tuned exclusively with texts composed of no more than 2 training skills, they also show improved composition ability for 3 and 4 skills.

Note Finding 1 and Finding 2 are beyond the scope of the theory presented in Arora and Goyal [1], which studies the composition ability for skills that appear in the training data.

Finding 3 : Fine-tuning on texts that compose more skills (i.e., with a larger k ) is more data-efficient for learning skill compositions. We design control experiments in Section 4.3 that fine-tune LLaMA2-13B-Chat on two datasets: (a) one dataset contains around 10,000 text pieces with only 1 or 2 skills; and (b) another dataset contains 8,000 text pieces, consisting of a random subset of the first dataset and around 2,000 text pieces that compose 3 skills. Table 4 shows that LLaMA-2 fine-tuned on the dataset with richer skill composition performs significantly better than the other for all k = 2 , 3 , 4 , 5 .

We discuss our main findings thoroughly in Section 4. In Section 5.1, we solidify our findings using Claude 3 Opus (instead of GPT-4) as the Grader in evaluation. This eliminates the possibility that the ability to compose skills comes from GPT-4's bias towards the models fine-tuned on GPT-4's outputs.

We also discuss the implications of our findings for going beyond 'stochastic parrots' behavior [2], which refers to the perception that LLMs might not generate novel pieces of text but rather mimic data from the pretraining corpus (Section 5.2).

2 Related Works

Compositional generalization Compositional generalization has grabbed lots of attention in AI. [29, 27] studied compositional generalization in the realm of mathematical reasoning, and [3, 22] investigated for logical inference. In computer vision, compositional generalization was studied on disentangled representation learning to generate images from novel combinations of concepts [12, 10, 31]. Besides, several works have explored composing visual relations [20], as well as benchmarks for text-to-visual generation[14, 19]. Other works have explored using compositional models for image generation [7] , as well as to create plans for unseen tasks at inference time [6].

Compositional generalization for language and LLMs There is also a long history of study of compositional generalization in language [11, 18, 4, 15, 17, 21]. However, the test bed for compositional generalization mostly relies on rule-based languages, like SQL or synthetic-generated ones, and thus deviates a little bit from natural language. Recent works have observed compositional

capabilities in LLMs emerge multiplicatively on natural languages [30, 1, 23, 33]. These observations have fueled a growing interest in exploring and evaluating compositional generalization in LLMs as a means to more appropriately evaluate LLM capabilities [9, 32, 24, 26, 33]. Some examples include imposing constraints and/or requirements on text generation [9, 32, 5], as well as providing multi-hop questions whose answers require composing multiple facts that were individually observed during pretraining [26]. Dziri et al. [8] tests whether the LLMs can learn compositional generalization by studying integer multiplication and a specific form of dynamic programming. Skill-Mix [33] presents an approach to evaluating compositional generalization on language skills, which we discuss in more detail in the next paragraph.

Skill-Mix Yu et al. [33] introduce a new evaluation named SKILL-MIX that tests for models to produce novel pieces of text from random combinations of k skills, and the evaluation can be made more difficult by increasing the value of k . The procedure is roughly as follows: (1) from a set of N language skills and T topics, pick a random subset of k skills and one topic; then, (2) query the Student model to produce a short piece of text (at most k -1 sentences) that illustrates the k skills in the context of the provided topic. Note that for k = 1 , the maximum sentence limit is 1 sentence. A Grader model is used to evaluate the text piece based on the following criteria: correctly illustrating all k skills and the topic, meeting the maximum length requirement, and general coherence. Thus, each piece of text can award up to a maximum of k +3 points (see the original paper for various metrics extracted from points earned). Note that each of the N language skills has a Wikipedia entry, so it is reasonable to expect an LLM to encounter the skills multiple times in isolation in the pretraining corpus, but not in all possible combinations. In this paper, we choose to study the compositional generalization of LLMs in the context of SKILL-MIX because SKILL-MIX is close to general language capability and is more flexible for modifying the language skill set.

3 Pipeline

Our pipeline consists of three stages: generating data by selecting GPT-4 responses on SKILL-MIX (Section 3.1), fine-tuning on the generated data (Section 3.2), and evaluating our fine-tuned model on SKILL-MIX evaluation [33] (Section 3.3). The pipeline overview is shown in Figure 1.

3.1 Data generation

We adapt the procedure presented in SKILL-MIX evaluation [33] to produce finetuning data. Only the generations with full marks (i.e., illustrating all skills and topics, meeting the length requirement, and general coherence) are selected. To enhance the likelihood of obtaining full marks, we prompt GPT-4, the best Student model reported in Yu et al. [33], to create the generations.

Skills and topics for data generation. Since our goal is to measure the generalization capability strictly, we minimize the overlap between the skills/topics used during data generation and the skills/topics used to evaluate the fine-tuned models. Specifically, we partition the original set of 101 skills introduced in Yu et al. [33], S , into a set of 53 train skills, S train, and 48 held-out skills, S held-out, based on the skill category. S train includes only literary and rhetorical categories, while S held-out comprises the rest of the categories, including reasoning, logic, theory of mind, pragmatics, common sense, and physical knowledge. Similarly, we partition the original set of topics, T , into random sets of 50 training topics, T train, and 50 held-out topics, T held-out. It is important to note that partitioning skills randomly can lead to correlations between the train and held-out skills, as skills from the same category can be highly related. However, partitioning topics randomly does not present this issue, as the topics are generally unrelated. (Please refer to Appendix A for the detailed list of skills and topics.)

Data generation with k = 1 , 2 , 3 . We produce fine-tuning data with k = 1 , 2 and 3 using GPT-4 as both the Student and Grader model. For k = 1 , we use the original set of skills S and training topics T train to produce approximately 5,000 generations, and we only keep generations that receive full marks. We refer to the resulting dataset as D SKILL-MIX (1) . D SKILL-MIX (1) contains only texts with individual skills, thus serving the role of separating the improvement from better utilizing an individual skill and the improvement from better composing multiple skills in later experiments.

We follow an analogous procedure for k = 2 and k = 3 , but using our 53 training skills S train and 50 training topics T train . We produce 10,000 generations for each k before filtering. We refer to the resulting datasets as D SKILL-MIX (2) and D SKILL-MIX (3) , respectively. For convenience, we use D SKILL-MIX (1 , 2) to denote the dataset that combines D SKILL-MIX (1) and D SKILL-MIX (2) , i.e., D SKILL-MIX (1 , 2) = D SKILL-MIX (1) ∪ D SKILL-MIX (2) . Similarly, we use D SKILL-MIX (1 , 2 , 3) to denote the

dataset that combines D SKILL-MIX (1) , D SKILL-MIX (2) , and D SKILL-MIX (3) together. We summarize our notations in Table 1.

3.2 Fine-tuning

We fine-tune LLaMA-2-13B-Chat [28] and Mistral-7B-Instruct-v0.2 [16] on the data generated in Section 3.1 for 4000 steps with a batch size of 64. Each data generated from SKILL-MIX consists of 4 parts: PROMPT1, ANSWER1, PROMPT2, ANSWER2. Here, PROMPT1 denotes the prompt asking the student to generate answers, ANSWER1 stands for student's first round answer, PROMPT2 is the prompt that asks the student to correct or refine its answer, and ANSWER2 is the student's second round answer. During fine-tuning, we feed the concatenation of PROMPT1, ANSWER1, PROMPT2, ANSWER2 into the model as a single text, but only compute the cross-entropy loss for tokens belonging to ANSWER1 and ANSWER2. We use Adam as the optimizer and linear warmup for the first 64 steps, followed by a constant learning rate of 2e-5 for the remaining training steps. 1 The maximum token length is set as 1024. All fine-tuning experiments are conducted on 4 Nvidia H100/A100 GPUs. Similarly to the loss design of RLHF [25], we mix pre-training data 2 during fine-tuning to prevent degradation of general abilities.

3.3 Evaluation

We evaluate the SKILL-MIX( k ) performance ( k = 2 , 3 , 4 , 5 ) for all the models fine-tuned on data generated in Section 3.1, i.e., D SKILL-MIX (1) , D SKILL-MIX (2) , and D SKILL-MIX (3) .

Settings As mentioned earlier, SKILL-MIX evaluation requires a skill set and a topic set. We consider the following 3 settings (where Setting II is our main setting used in Figure 1):

SKILL-MIX evaluation on training skills and topics. Since the model observes the same skills and topics during fine-tuning, this setting serves as an in-domain evaluation for k = 2 , 3 . For k = 4 , 5 , it tests the models' ability to combine more skills, which is already out-ofdomain, since the model has never seen such data during fine-tuning. We use the notation SKILL-MIXtrain( k ) to denote the SKILL-MIX( k ) evaluation on training skills and topics.

Model

SKILL-MIX (

k

) Performance

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

Evaluations on training skills and topics (SKILL-MIXtrain(

k

))

LLaMA-2-13B-Chat

.52/.52

.17/.47

.02/.34

.00/.33

.00/.31

ft'ed on

D

SKILL-MIX

(1)

.87/.87

.15/.51

.00/.43

.00/.37

.00/.35

ft'ed on

D

SKILL-MIX

(1

,

2)

.88/.88

.50/.70

.12/.56

.01/.55

.02/.52

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.89/.89

.51/.73

.24/.68

.08/.64

.03/.60

Evaluations on held-out skills and topics (SKILL-MIXheld-out(

k

))

LLaMA-2-13B-Chat

.46/.46

.28/.50

.04/.42

.01/.39

.00/.43

ft'ed on

D

SKILL-MIX

(1)

.87/.87

.43/.70

.05/.54

.01/.49

.00/.44

ft'ed on

D

SKILL-MIX

(1

,

2)

.95/.95

.75/.87

.25/.68

.05/.60

.02/.56

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.96/.96

.78/.88

.37/.75

.09/.69

.02/.60

Evaluations on all skills and topics (SKILL-MIXall(

k

))

LLaMA-2-13B-Chat

.46/.46

.24/.50

.02/.42

.01/.40

.00/.34

ft'ed on

D

SKILL-MIX

(1)

.88/.88

.27/.62

.05/.50

.00/.40

.00/.33

ft'ed on

D

SKILL-MIX

(1

,

2)

.96/.96

.51/.74

.17/.65

.01/.54

.00/.51

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.96/.96

.65/.81

.33/.73

.15/.69

.06/.62

SKILL-MIX on held-out skills and topics. This setting tests the models' ability to combine skills that are never present in fine-tuning. 3 This setting serves as another perspective to show the stronger out-of-domain generalization for composing skills compared to Setting I. We use the notation SKILL-MIXheld-out( k ) to denote the SKILL-MIX( k ) evaluation on held-out skills and topics.

SKILL-MIX on all skills and topics. Evaluating SKILL-MIX on only half of the skills split by category might make the evaluation easier, since combining 2 rhetorical or logical skills might be easier than combining 1 rhetorical and 1 logical skill. Thus, we also evaluate SKILL-MIX on all skills and topics available, which serves as a direct comparison with the results in Yu et al. [33]. We use the notation SKILL-MIXall( k ) to denote the SKILL-MIX( k ) evaluation on all skills and topics.

Evaluation Metrics We follow the evaluation rubric of SKILL-MIX. Each generated text can receive up to k +3 points: 1 point for each correctly illustrated skill, 1 point for sticking to the topic, 1 point for text coherence / making sense, and 1 point for meeting the length requirement.

Following Yu et al. [33], we grade each generated piece of text three times. For each of the k +3 criteria, we collect the majority vote among the three grading rounds, and map the points earned to the following two metrics of interest 4 : ( Ratio of Full Marks ) count as 1 if all k +3 points are earned, and 0 otherwise; and ( Skills Fraction ) the fraction of points awarded for the k skills if all 3 points are awarded for the remaining criteria, and 0 otherwise. For a given ( k skill, 1 topic) combination, we take the maximum value of the metric among the 3 generations. We average the maximum value across all the combinations. Note that we use one of the harder variants of SKILL-MIX [33], where we do not award any points for a particular skill if the skill name is explicitly mentioned in the generated text piece.

4 Skill Composition Can Be Learned From Examples

We present experiment results using the pipeline (Section 3) to evaluate compositional generalization. Table 2 and Table 3 summarizes the SKILL-MIX ( k ) performances of LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 fine-tuned on various datasets under three evaluation settings. We discuss our findings on compositional generalization for in-domain evaluations (Section 4.1), compositional generalization for out-of-domain evaluations (Section 4.2), and the data efficiency to induce compositional generalization (Section 4.3).

4.1 Compositional generalization for in-domain evaluations

We first observe that, after fine-tuning LLaMA-2-13B-Chat on D SKILL-MIX (1 , 2) , the SKILL-MIXtrain( 2 ) performance significantly improves. Similarly, after fine-tuning LLaMA-2-13B-Chat on D SKILL-MIX (1 , 2 , 3) , the SKILL-MIXtrain( 3 ) performance also improves. For example, the Ratio of Full Marks for SKILL-MIXtrain( 3 ) improves from 2% for LLaMA-2-13B-Chat to 24% after fine-tuned on D SKILL-MIX (1 , 2 , 3) (Table 2).

One confounding factor in the above evaluation is that the original LLaMA-2-13B-Chat may not utilize all the individual skills perfectly, and the SKILL-MIX performance improvement might just be attributed to the model's knowledge of the individual skills after fine-tuning, not the model's ability to better compose different skills together. Thus, we also evaluate the SKILL-MIX performance on LLaMA-2-13B-Chat fine-tuned only on D SKILL-MIX (1) , which consists of purely SKILL-MIX k = 1 data and serves as another baseline besides the original LLaMA-2-13B-Chat. After fine-tuning on D SKILL-MIX (1) , the model indeed knows the individual skills much better, since the Ratio of Full Marks of SKILL-MIX k = 1 improves from 52% to 87% . However, better knowledge of individual skills does not lead to a better ability to compose skills together, since the SKILL-MIXtrain( 2 ) or SKILL-MIXtrain( 3 ) performance of LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1) keeps nearly the same as the pre-trained ones, under both Ratio of Full Marks and Skills Fraction metrics. Thus, mainly all the improvement on SKILL-MIXtrain( 2 ) or SKILL-MIXtrain( 3 ) indeed comes from the ability to compose different skills together.

Fine-tuning on Mistral-7B-Instruct-v0.2 (Table 3) shows similar results as fine-tuning on LLaMA2-13B-Chat (Table 3), except that only fine-tuning on D SKILL-MIX (1) shows more improvement on SKILL-MIXtrain( 2 ) and SKILL-MIXtrain( 3 ) compared to the original Mistral-7B-Instruct-v0.2 model. A possible explanation is that Mistral-7B-Instruct-v0.2 is better at composing skills than LLaMA-213B-Chat, and fine-tuning on D SKILL-MIX (1) helps Mistral-7B-Instruct-v0.2 exhibit each skill more properly and clearly when composing skills.

4.2 Compositional generalization for out-of-domain evaluations

This section discusses the observations that indicate the out-of-domain generalization of skill composition, including generalization to unseen k and generalization to unseen skills.

SKILL-MIX train ( k ) improves for unseen k . We first observe that, after fine-tuning LLaMA-213B-Chat on SKILL-MIX data D SKILL-MIX (1 , 2 , 3) , the SKILL-MIXtrain( 4 ) and SKILL-MIXtrain( 5 ) performance also increase. For example, the Ratio of Full Marks improves from 0% to 8% when k = 4 (Table 2). Note that 8% Ratio of Full Marks improvement on k = 4 is significant, since besides GPT-4, all other models tested in Yu et al. [33], including GPT-3.5-turbo, cannot get over 2% Ratio of Full Marks on k = 4 (Table 3 in [33]). Besides, training only on D SKILL-MIX (1) does not improve the SKILL-MIXtrain( 4 ) or SKILL-MIXtrain( 5 ).

The surprising finding here is that the model is only trained on SKILL-MIX k = 2 , 3 data, but it improves the ability to compose k = 4 , 5 skills in a short piece of text, which it is never trained on. The results suggest that its ability to compose multiple skills does not come from overfitting training data but should be perceived as learning a meta-skill instead. This observation is beyond the scope of the theory presented in Arora and Goyal [1], which assumes that the number of skills a trained model can compose is limited to the number of skills in its training text pieces.

Fine-tuning on Mistral-7B-Instruct-v0.2 (Table 3) shows similar results, where the fine-tuned model is able to improve SKILL-MIXtrain( 4 ) and SKILL-MIXtrain( 5 ) after fine-tuning on D SKILL-MIX (1 , 2 , 3) , while training only on D SKILL-MIX (1) has limited improvement.

Improvement on SKILL-MIX held-out ( k ) and SKILL-MIX all ( k ). Besides the SKILL-MIX performance improvement on training skills and topics, we also observe the improvement of SKILL-MIXheld-out( k ) (Setting II) from Table 2 and Figure 2. Similar to the evaluation on training skills and topics, finetuning LLaMA-2-13B-Chat on D SKILL-MIX (1) only improves the SKILL-MIXheld-out( k ) performance for k = 3 , 4 , 5 marginally, but it indeed improves the SKILL-MIX k = 2 . However, the improvement is incomparable with fine-tuning on D SKILL-MIX (1 , 2 , 3) . This shows that the ability to compose multiple skills generalizes to held-out skills, even though our training never exposed the model to data with the held-out skills. Besides the SKILL-MIX improvement on held-out skills, we also observe the improvement of SKILL-MIXall( k ). This result again suggests that models learn meta-skill rather than overfitting to skill combinations in the training data.

Note that the SKILL-MIXheld-out( k ) performance is better than the SKILL-MIXtrain( k ) in Table 2, which is counter-intuitive. We hypothesize that this phenomenon happens because the pre-trained model knows how to compose held-out skills (logic, reasoning, theory of mind) better than training skills (rhetorical and literary). Or possibly the training skills are harder to compose. Exploring difficulty of individual skills is left for future work.

Also, fine-tuning on Mistral-7B-Instruct-v0.2 (Table 3) shows similar results as fine-tuning on LLaMA-2-13B-Chat (Table 2), except that Mistral-7B-Instruct-v0.2 has more improvement than LLaMA-2-13B-Chat when fine-tuned on D SKILL-MIX (1) .

4.3 Data requirement for inducing compositional generalization

Compared with fine-tuning on D SKILL-MIX (1 , 2) , one can observe that LLaMA-2-13B-Chat/Mistral7B-Instruct-v0.2 fine-tuned on D SKILL-MIX (1 , 2 , 3) gains more performance boost on k = 4 , 5 across all settings. For example, SKILL-MIXall( 4 ) performance for LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1 , 2) is nearly the same as the original LLaMA-2-13B-Chat and LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1) . However, for LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1 , 2 , 3) , the SKILL-MIXall( 4 ) performance improves from 1% to 15% .

However, one may argue it is because D SKILL-MIX (1 , 2 , 3) has more data in total than D SKILL-MIX (1 , 2) . To make a fair comparison, we conduct an ablation study by sub-sampling 8000 data from D SKILL-MIX (1 , 2 , 3) , making sure that the number of data points with k = 2 and k = 3 in the subsampled set is less than the size of D SKILL-MIX (2) . Table 4 shows the SKILL-MIXall( k ) performance of LLaMA-2-13B-Chat fine-tuned on the sub-sampled dataset. The metrics remain relatively close to the model fine-tuned on full D SKILL-MIX (1 , 2 , 3) and significantly better than the model fine-tuned on D SKILL-MIX (1 , 2) . This ablation confirms that 'skill-richer' data can induce the ability to compose skills faster.

Model

SKILL-MIXall(

k

) Performance

k

= 2

k

= 3

k

= 4

k

= 5

Graded by Claude-3

.31/.52

.07/.48

.08/.64

.00/.42

ft'ed on

D

SKILL-MIX

(1)

.45/.70

.14/.59

.02/.50

.00/.42

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.69/.81

.57/.83

.26/.77

.10/.69

.24/.50

.02/.42

.01/.40

.00/.34

ft'ed on

D

SKILL-MIX

(1)

.27/.62

.05/.50

.00/.40

.00/.33

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.65/.81

.33/.73

.15/.69

.06/.62

5 Discussions

5.1 Using Claude 3 Opus as Grader for SKILL-MIX evaluation

All the findings in the previous section are based on the SKILL-MIX performance graded by GPT-4. However, GPT-4 is heavily used during data generation, and one can argue the improvement might solely come from the fact that GPT-4 favors its own outputs. Although the possibility is low, to rigorously eliminate this confounding factor, we re-evaluate SKILL-MIXall( k ) using Claude 3 Opus as the Grader, and report the results in Table 5. Besides, we also include a consistency check between the GPT-4 and Claude 3 Opus graders in Table 6.

From Table 5, we observe the metrics graded by Claude 3 Opus have a similar trend as those graded by GPT-4: after fine-tuning on D SKILL-MIX (1 , 2 , 3) , SKILL-MIXall( k ) performance improves for all k = 2 , 3 , 4 , 5 , while fine-tuning only on D SKILL-MIX (1) has limited improvement over the original LLaMA-2-13B-Chat. It proves that the improvement of SKILL-MIX performance is not overfitted to GPT-4 preference.

Interestingly, we find that Claude 3 Opus is more generous, assigning higher scores to both the LLaMA-2-13B-Chat and the fine-tuned version. Besides, the results from Table 6 also confirm this argument: if an answer is assigned a full mark by GPT-4, then many of them will also be assigned a full mark by Claude 3 Opus. Such consistent biases among Graders were also noted in [33] when comparing LLaMA-2-70B-Chat and GPT-4 as Graders.

Besides switching to Claude 3 Opus, we also do human spot checks on the SKILL-MIX generations, making sure that the model is not generating something that does not make sense to human. Please refer to Appendix D for some of the examples of SKILL-MIX evaluations before and after the fine-tuning.

5.2 Potential capability of going beyond 'stochastic parrots behavior'

Whether models can go past 'stochastic parrots' behavior [2] is crucial in discussions of AI risk. Based on reasonable performance of GPT-4 on SKILL-MIX( k = 5 ) with common skills removed, Yu et al. [33] suggests GPT-4 is already beyond 'stochastic parrots'. In particular, after removing common skills (see definition in [33]), the probability of a random (5 skills, 1 topic) combination appearing in the training corpus is estimated to be 11% if the training token is 2T 5 . Therefore, if a model has a Ratio of Full Marks beyond 11% when k = 5 , then it suggests the model is able to output novel text, thus is beyond 'stochastic parrots'. GPT-4 is the only model that can achieve this in [33] 6 .

Table 7 shows the SKILL-MIX ( k = 5 ) performance of fine-tuned LLaMA-2-13B-Chat and Mistral7B-Instruct-v0.2 with common skills removed. We also include some D SKILL-MIX (4) data (about 1000) into the training set. The fine-tuned models all show significant improvement over the base models. For example, the Ratio of Full Marks for the fine-tuned LLaMA-2-13B-Chat and Mistral-7B-Instructv0.2 all go beyond 15% for SKILL-MIX ( k = 5 ) on training skills and topics, and reaches 6% on all skills and topics, after filtering out the common skills.

Although both fine-tuned models cannot reach 11% for SKILL-MIXheld-out( 5 ) and SKILL-MIXall( 5 ), we hypothesize that with more skill-richer data (say D SKILL-MIX (4) ), the models can acquire the ability to combine skills much more efficiently and go beyond 'stochastic parrots' eventually.

We further discuss the limitation of our work in Section 7, and potential influences on AI safety caused by stronger composition capability in Appendix E.

6 Conclusion and Takeaways

We have studied the extent to which models can learn compositional generalization over language skills by fine-tuning on suitable examples demonstrating such composition. Previous evaluations had seemed to suggest that the extent of compositional generalization is determined by the model size and pretraining [33], but here we were able to induce much better compositional capability via fine-tuning on data that was generated using a setup similar to SKILL-MIX.

One surprising finding was that fine-tuning examples that composed 2 and 3 language skills were enough to improve the capability to compose 4 and even 5 language skills. Another surprise was that the ability to combine language skills from held-out categories improved at the same rate as the skills used in the training examples. Of course, these findings were still about skills that are near relatives. The full extent of such "out of (training) distribution" generalization remains to be explored.

7 Limitation

The main limitation of the current study is the high computational and financial costs, which impede us from sweeping more hyperparameters and conducting repeated experiments with different random seeds. These costs include the number of GPU hours for fine-tuning and the cost of calling OpenAI's API to generate the D SKILL-MIX ( k ) data and evaluate the SKILL-MIX performance. Despite these difficulties, we managed to sweep the hyperparameters for fine-tuning the LLaMA-2-13B-Chat on D SKILL-MIX (1 , 2 , 3) (Main experiment, Table 2). We believe our findings are robust to different random seeds because of the clear message and consistent trend of the results.

Besides, compositional generalization is a vast topic and we only study this under the setting of language skills (limited to the SKILL-MIX setting). Whether the models can learn compositional generalization in other settings still needs further exploration.

References

Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936 , 2023.

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 610-623, 2021.

Samuel R Bowman, Christopher D Manning, and Christopher Potts. Tree-structured composition in neural networks without tree-structured architectures. arXiv preprint arXiv:1506.04834 , 2015.

Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 4427-4442, 2020.

Jiaao Chen, Xiaoman Pan, Dian Yu, Kaiqiang Song, Xiaoyang Wang, Dong Yu, and Jianshu Chen. Skills-in-context prompting: Unlocking compositionality in large language models. arXiv preprint arXiv:2308.00304 , 2023.

Yilun Du and Leslie Kaelbling. Compositional generative modeling: A single model is not all you need, 2024.

Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation and inference with energy based models, 2020.

Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems , 36, 2024.

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.

Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan-Willem Meent. Structured disentangled representations. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2525-2534. PMLR, 2019.

Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. Improving text-to-sql evaluation methodology. arXiv preprint arXiv:1806.09029 , 2018.

Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv:1707.03389 , 2017.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation, 2023.

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise?, 2020.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring compositional generalization: A comprehensive method on realistic data, 2020.

Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning , pages 2873-2882. PMLR, 2018.

Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation, 2024.

Nan Liu, Shuang Li, Yilun Du, Joshua B. Tenenbaum, and Antonio Torralba. Learning to compose visual relations, 2021.

Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. Compositional generalization by learning analytical expressions. Advances in Neural Information Processing Systems , 33:11416-11427, 2020.

Mathijs Mul and Willem Zuidema. Siamese recurrent networks learn first-order logic reasoning and exhibit zero-shot compositional generalization. arXiv preprint arXiv:1906.00180 , 2019.

Maya Okawa, Ekdeep Singh Lubana, Robert P. Dick, and Hidenori Tanaka. Compositional abilities emerge multiplicatively: Exploring diffusion models on a synthetic task, 2024.

Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. Making transformers solve compositional tasks, 2022.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023.

David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv:1904.01557 , 2019.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Sara Veldhoen, Dieuwke Hupkes, Willem H Zuidema, et al. Diagnostic classifiers revealing how neural networks process hierarchical structure. In CoCo@ NIPS , pages 69-77. Barcelona, 2016.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.

Zhenlin Xu, Marc Niethammer, and Colin A Raffel. Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language. Advances in Neural Information Processing Systems , 35:25074-25087, 2022.

Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik Narasimhan. Collie: Systematic construction of constrained text generation tasks, 2023.

Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: a flexible and expandable family of evaluations for ai models. In The Twelfth International Conference on Learning Representations , 2023.

Appendix

A Skills and Topics Partition

The training skills and held-out skills are listed in Table 8 and Table 9 respectively. The training and held-out topics are shown in Table 10.

Table 8: The list of train skills for generating D SKILL-MIX ( k ) and evaluating SKILL-MIXtrain( k ). For each skill, we list its category, name, definition, and one example using this skill.

B Prompts for Data generation

In this section, we document our prompts for generation and prompts for grading.

B.1 Prompts for generation

PROMPT1

Greetings! I am interested in natural language processing and I was wondering if you could help me generate an example of text that illustrates multiple skills in semantics or syntax. The example should be a minimal natural piece of text with up to a few lines in the context of topic that illustrates all of the following skills: {skills_str}. Please keep the text as short as possible, and make sure the concepts can be found fully from the text.

PROMPT2

Note that this pair of prompts is the same as the generation prompts for GPT-4 in Yu et al. [33], which is slightly different from the generation prompts for LLaMA-2 in Yu et al. [33]. This difference of prompts, along with randomness from multiple sources, caused some difference in LLaMA-2-13BChat performance on SKILL-MIX( k ) between our paper and Yu et al. [33].

B.2 Prompts for grading

Grading prompt for GPT-4 Grader:

Model

SKILL-MIX (

k

) Performance

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

Evaluations on training skills and topics (SKILL-MIXtrain(

k

))

LLaMA-2-13B-Chat

.52/.56/.52

.17/.19/.47

.02/.02/.34

.00/.00/.33

.00/.00/.31

ft'ed on

D

SKILL-MIX

(1)

.87/.91/.87

.15/.19/.51

.00/.00/.43

.00/.00/.37

.00/.00/.35

ft'ed on

D

SKILL-MIX

(1

,

2)

.88/.96/.88

.50/.58/.70

.12/.14/.56

.01/.03/.55

.02/.02/.52

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.89/.96/.89

.51/.62/.73

.24/.28/.68

.08/.10/.64

.03/.03/.60

Evaluations on held-out skills and topics (SKILL-MIXheld-out(

k

))

LLaMA-2-13B-Chat

.46/.53/.46

.28/.32/.50

.04/.05/.42

.01/.01/.39

.00/.00/.43

ft'ed on

D

SKILL-MIX

(1)

.87/.95/.87

.43/.50/.70

.05/.06/.54

.01/.01/.49

.00/.00/.44

ft'ed on

D

SKILL-MIX

(1

,

2)

.95/.99/.95

.75/.79/.87

.25/.29/.68

.05/.05/.60

.02/.02/.56

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.96/1.0/.96

.78/.81/.88

.37/.43/.75

.09/.12/.69

.02/.02/.60

Evaluations on all skills and topics (SKILL-MIXall(

k

))

LLaMA-2-13B-Chat

.46/.54/.46

.24/.29/.50

.02/.02/.42

.01/.02/.40

.00/.00/.34

ft'ed on

D

SKILL-MIX

(1)

.88/.95/.88

.27/.28/.62

.05/.05/.50

.00/.00/.40

.00/.00/.33

ft'ed on

D

SKILL-MIX

(1

,

2)

.96/.99/.96

.51/.57/.74

.17/.23/.65

.01/.01/.54

.00/.00/.51

ft'ed on

D

SKILL-MIX

(1

,

2

,

3)

.96/.99/.96

.65/.68/.81

.33/.36/.73

.15/.16/.69

.06/.07/.62

is worth 1 point. The criteria are: {rubric_items}. The table should only have the following columns: 'Criteria', 'Points Earned'. In the 'Points Earned' column, please provide only numerical values with no additional formatting. Please introduce the table with 'Here's the grading table:' and please include a row for 'Total Points Earned' at the end of the table. Finally, please start your grading explanation with 'Explanation':

Grading prompt for Claude 3 Opus Grader:

C Complete Results for SKILL-MIX Evalution

Besides Ratio of Full Marks and Skill Fraction , Yu et al. [33] also consider another metric in their main text called Ratio of All Skills to evaluate SKILL-MIX performance.

Ratio of All Skills : 1 if k points are awarded for the k skills and at least 2 points are awarded for the remaining criteria, and 0 otherwise

We also report this metric for LLaMA-2-13B-Chat and Mistral-7B-Instruct-v0.2 fine-tuned on D SKILL-MIX ( k ) . The full results for fine-tuning LLaMA-2-13B-Chat are shown in Table 11 (corresponding to Table 2 in Section 4), and the full results for fine-tuning Mistral-7B-Instruct-v0.2 can be

bound in Table 12 (corresponding to Table 3 in Section 4). All of our findings still hold under this Ratio of All Skills metric.

D Examples of SKILL-MIX Generations

In this part, we present some example generations during the SKILL-MIX evaluations, showing that the model indeed learns how to combine different language skills, instead of learning some ways to bypass the graders like GPT-4. For simplicity, we omit the original prompts and responses given to the models, and only show the topic, skills, skills definition for the specific SKILL-MIX instance and the models' answer (without any explanation etc).

Example 1 SKILL-MIX question

Skills definitions:

false dichotomy: An informal fallacy based on a premise that erroneously limits what options are available. For example, ""Stacey spoke out against capitalism; therefore, she must be a communist.""

availability bias: a mental shortcut that relies on immediate examples that come to a given person's mind when evaluating a specific topic, concept, method, or decision. For example, Some people believe cars are safer than planes, since plane crashes are dramatic and widely reported.

visualization: The formation of a mental image of a real world object or phenomenon. For example, ""Jerry watched the clock tick down towards his lunch hour. He could see his burger waiting for him in his mind's eye.""

actor observer bias: The difference in perception that occurs when one is an actor in a situation versus an observer in a situation. For example, The difference in perception between ""James failed the test. He must be a bad student."" and ""I failed the test, but it wasn't because I was a bad student. I was just busy with other obligations."" is an example of actor observer bias.

Answer by LLaMA-2-13B-Chat:

I'm considering traveling to Japan this summer, but the recent news about the coronavirus has me worried. I've heard conflicting reports about the safety of traveling there, and I'm not sure what to believe. Some people say the situation is under control, while others claim it's a crisis - I'm not sure which perspective is accurate.

Answer by LLaMA-2-13B-Chat fine-tuned on SKILL-MIX data ( D SKILL-MIX (1 , 2 , 3) ).

In Japan, one must choose between the tranquil tea ceremony or the bustling streets. Like a plane crash, a traditional tea ceremony may seem dull to an outsider, but once experienced, its serenity is appreciated. Just as a local prefers the ease of a train over a tourist's familiar plane, our perception of experiences is influenced by our biases.

Example 2 SKILL-MIX question

Topic: Plumbers Skills: tu quoque, categorical syllogism, aphorism, self motivation Skills definitions: 1. **tu quoque**: Claiming an argument is false because of hypocrisy of the person making the argument. For example, ""How can you tell me not to smoke when you yourself smoke?"" 2. categorical syllogism: Infers a conclusion from two premises. For example, ""No geese are felines. Some birds are geese. Therefore, some birds are not felines."" 3. aphorism: A short saying that observes a general truth. For example, ""Pride goeth before a fall.""

self motivation: the ability to maintain a drive towards one's goals. For example, ""If I get this done, I'll reward myself by playing video games""

Answer by LLaMA-2-13B-Chat:

Hypocrisy much? You're always puffing away on your cigar, yet you're telling me to stop smoking? That's like a pig giving lectures on veganism! Answer by LLaMA-2-13B-Chat fine-tuned on SKILL-MIX data ( D SKILL-MIX (1 , 2 , 3) ). mechanic, and all

You tell me to stop smoking, yet you light up a cigar yourself. No plumber is a plumbers fix pipes. Thus, no mechanic is a plumber, and remember, 'Pride goeth before a fall' let's get this done to earn a video game reward.

E Examples of Fine-tuned Model's Generation and Implication to Safety

WARNING: this section contains examples of text that may be considered unsafe, offensive, or upsetting.

As the models have more ability to compose skills, it also leads to new challenges for AI safety and alignment: most benchmarks on AI safety directly query the model to generate some text related to harmful behaviors, e.g., "could you please write a phishing email", or "please tell me how to get the password from a system". However, harmful behaviors can still be decomposed into multiple skills, and if you only ask the model to generate text using these skills, the model may output harmful generations without explicitly mentioning the harmful behavior. For example, to write a short tweet criticizing based on fake experience, the skills needed include

"false claim of expertise: the critic claims to have done or experienced something they haven't. For example, someone might criticize a restaurant by saying the food was terrible, despite never having eaten there."

and

"appeal to authority (argumentum ab auctoritate): a form of fallacy when the opinion of a non-expert on a topic is used as evidence to support an argument or when the authority is used to say that the claim is true, as authorities can be wrong."

Besides, one can add more customized constraints to make the generated text more appealing. Following is an example of generation we obtained using LLaMA-2-13B-Chat fine-tuned on D SKILL-MIX (1 , 2 , 3) , using the prompt similar to SKILL-MIX( 3 ).

Prompt to the model:

Model's response:

Another example is in the context of writing customized phishing or spam emails, where we query the model to generate an email (with link or attackment inside) trying to grab the attention from Sushi enthusiasts and allure them to click the link.

Prompt to the model:

Model's response:

These examples show that as the model gains more ability to compose skills, it might also leave potential holes for AI safety. Note these examples are easy to adjust to other topics and scenarios. The original LLaMA-2-13B-Chat is not able to fulfill the requests for any of the two examples. Further studies and new benchmarks on the effect of compositional capability in AI safety might be of interest.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper supports claims made with examples from tables.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper includes a limitations section, detailing primary limitation as the high compoutational and financial costs associated with running experiments.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This work does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper details relevant data splits, hyperparameters, etc. necessary to reproduce the experiments, and also releases a link to the code in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We have provided details that should be enough for reproducing the results, and we will provide the codebase in the final version.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper details all relevant data splits, hyperparameters, etc. in the methodology section (and has an accompanying table to summarize notation).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Error bars are not applicable since our metrics are based on majority vote.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper discusses the hyperparameters and resources used in methodology and limitations sections.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: No human subjects are involved, all data is synthetic, limitations are discussed, and details for reproducing results are available.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The focus of the paper is compositional generalization. We do not released any models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper provides credit for any assets in References Section.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the extent to which LLMs can learn compositional generalization over skills by finetuning with suitable training examples. Based on abundant experimental results, the paper concludes that finetuning using a dataset with skill composition examples helps the model build a “meta-skill” that allows them to generalize to more complex (i.e., composition of more skills) tasks. The paper also finds that the training samples containing higher-order skill compositions are more efficient in eliciting such a meta-skill. The paper is quite easy to follow and the conclusions might inspire more interesting applications on LLM’s finetuning.\n\nSTRENGTHS:\n- The paper proposes a dataset that consists of examples with different selected skills for the analysis of skill composition.\n- The results in this paper show that finetuning not only provides new knowledge to the LLM but also the skill of composing different skills, which potentially inspire novel applications using finetuning.\n- The proposed method (although looks simple) efficiently improves the model’s performance on skill composition. The observation that the “skill-richer” data can induce the ability to compose skills faster would be very useful for practical LLM finetuning applications.\n\nWEAKNESSES:\n- It might be a bit hard to read the trends and compare results in the tables, visualizing some results might be helpful.\n\nQUESTIONS:\n- Line 157 mentions that the data has a form of [prompt 1, answer 1, prompt 2, answer 2]. I do not quite understand why we need prompt 2 and answer 2 here. Will the analysis still hold for simple [prompt 1, answer 1] data samples?\n- In line 260, the paper mentions a counter-intuitive phenomenon that the held-out performance is better than the train performance. The hypothetical explanation is that the model knows how to compose the held-out skills better than training skills. I think an experiment of switching the skill categories of the training and held-out sets can further support (or against) this hypothesis.\n- The problem setting studied in this paper reminds me of the “least-to-most” prompting design [1]. This paper shows using some in-context examples of $k=1,2$ can make the model generalize to problems with large $k$. Hence I’m curious about whether combining these more complex in-context prompt designs could further improve the finetuning performance. (I think the analysis and experiment in the current submission is enough for a good paper, just point out this paper and a potential idea.)\n\n[1] Zhou, Denny, et al. \"Least-to-most prompting enables complex reasoning in large language models.\" ICLR 2023\n\nLIMITATIONS:\nPlease refer to the question and weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article studies whether or not LLMs can be fine-tuned to compose skills for text generation. The authors generated training data from GPt-4 in the style of the Skill-Mix benchmark, asking the model to generate text about a topic while using a set of k skills (e.g., sympathy, temporal reasoning, syllogism, etc.). LlaMa-2 and Mistral-7B were then fine-tuned on this synthetic data and evaluated on their ability to generalize to combinations of novel skills, and to more combinations of skills than seen during training. Using GPT-4 as a grader, the authors find that fine-tuning substantially improves compositional generalization.\n\nSTRENGTHS:\nThis article has a number of strengths\n- addresses the important topic of compositional generalization using complex tasks\n- well-written article\n- technically sound\n- evaluation for combining novel skills and combining novel numbers of skills\n- Two graders were evaluated (GPT-4 and Claude)\n- Data efficiency was also studied\n\nThis is solid work that is consistent with other recent work on \"learning to compose\" through training [1,2]. It addresses these ideas on a larger scale than in past studies and shows compelling results, especially the generalization to more complex combinations.\n[1] Conklin, H., Wang, B., Smith, K., & Titov, I. (2021). Meta-learning to compositionally generalize. arXiv preprint arXiv:2106.04252.\n[2] Lake, B. M., & Baroni, M. (2023). Human-like systematic generalization through a meta-learning neural network. Nature, 623(7985), 115-121.\n\nWEAKNESSES:\nThe tasks in the Skill-Mix benchmark seem quite artificial, and difficult for people to both produce and evaluate. Only two example generations are provide in the appendix, and the first one has odd syntax in the prompt: \"The example should be a short tweet up to a few lines in the context of produce review...\"\n\nAnother weakness is the use of automatic grading by GPT-4, although it seems unavoidable and they also tested another grader from Claude. Additional discussion justifying this choice and the accuracy of the grader seems warranted.\n\nQUESTIONS:\nI don't have additional questions at this point.\n\nLIMITATIONS:\nThis section is fine.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the capacity of smaller language models to learn compositional generalization from finetuning. Utilizing the Skill-Mix set-up, the study delivers comprehensive experiments to assesse how small language models can improve their performance on both in-distribution and out-of-distribution compositional tasks after finetuning on systhetic datasets regarding in-distribution easier tasks generated by GPT-4.\n\nSTRENGTHS:\n- The paper is well-written.\n- The paper focuses on a very important topic (compositional generalization) of current LLM research.\n- It's easy to understand the intuition behind the comprehensive experiments.\n\nWEAKNESSES:\n- The experiment pipeline is not novel and highly overlaps with the previous work [1]  .\n- The paper is limited on one specific method on evaluating if language models can generalize compositionally on 'harder' tasks after finetuning on 'easier' compositional examples. There exists many other evaluation methods/metrics covered in related works that are not mentioned, including [2] and [3].\n- The paper does not properly explain and support the claim made in line 64-65 and line 243-244, i.e., \n\n    > Instead, they are acquiring a higher-order meta-skill that allows them to generalize and apply to combine unseen skills.\n\n    > The results suggest that its ability to compose multiple skills does not come from overfitting training data but should be perceived as learning a *meta-skill* instead. \n\n   This claim is very strong and needs more experimental results from other compositional tasks or theoretical justifications.\n\n[1] Yu, Dingli, et al. \"Skill-Mix: A flexible and expandable family of evaluations for AI models.\" arXiv preprint arXiv:2310.17567 (2023).\n\n[2] Dziri, Nouha, et al. \"Faith and fate: Limits of transformers on compositionality.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Chen, Jiaao, et al. \"Skills-in-context prompting: Unlocking compositionality in large language models.\" arXiv preprint arXiv:2308.00304 (2023).\n\nQUESTIONS:\n- The skill-mix paper only released small part of topics and skills to prevent people from chasing the leaderboard, so I'm wondering if the full list of skills and topics used in this paper is the same as the original paper.\n- The second-last line in Table 3 shows that Mistral-7B-Instruct-v0.2 get 0 point for Skill Fraction on $\\text { SKILL-MIX }(k)$ after finetuning on $\\mathcal{D}_{\\text {SKILL-MIX }}(1,2)$. This seems to be a typo.\n- The experimental results showing performance improvements after finetuning are unsurprised unless the author can justify if the model is indeed learning how to combine skills compositionally during finetuning.\n\nLIMITATIONS:\nThe limitations are mentioned in the Appendix section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a (family of) fine-tuned LLMs, trained using the SKILL-MIX dataset. They show that models trained on composite tasks where each instance involves a sequence of different skills improves the model in similar composite tasks. They demonstrate generalization by fine-tuning the models on subsets of task combinations and evaluating them on held-out sets. They also evaluate out-of-distribution generalization on the number of skills sequenced together and show that training on sequences of 2 or 3 skills can help generalize to sequences of 4 and 5 skills.\n\nSTRENGTHS:\n1. The authors have done extensive experiments using the SkillMix dataset to support their arguments.\n2. Using LLM to judge LLMs can introduce a lot of variability. I understand that the authors are following the procedure from Yu et al. Reading Appendix B of Yu et al. did increase my confidence and I appreciate adding another evaluator model which supports the analysis even further.\n\nWEAKNESSES:\n1. My main concerns with this paper have less to do with the methods of this particulaar submission, but of the dataset and methodology from Yu et al. that the authors extensively draw upon. Looking over the dataset and scoring design, it seems that the scores could be highly inflated simply by writing three disconnected sentences that each individually illustrates the skill. The only criterion in the rubric that is relevant for cohesion is ‘Makes sense’, which still can be evaluated independently for each sentence. This seems to be a rather weak form of skill compositionality that is the central point of this paper. Moreover, the authors imply that the skills and topics would not be highly related (Line 138), but given the skills presented in Appendix B, I’m not entirely convinced that this is true. I would be surprised if there weren’t at least a few hyperlinks between them.\n2. The baselines presented in Section 4.1 seem a bit weak. Fine-tuning on SM_train(1) is a useful starting baseline, but it is still severely restricted in its ability to span the dataset. If I understand the authors’ intent correctly that the goal is to demonstrate that fine-tuning to combine different skills, then what I think would be a more informative comparison would be to include within the training set samples involving the same skill/topic category but use higher than 1 k. Otherwise, this confounds with length-generalization, which we know sequence models struggle with. \n3. A similar concern exists for Sections 4.2 and 4.3, where training with higher k is confounded with training on longer sequences. I would be interested in understanding the effect size after controlling for sequence length how and what the distribution of sequence lengths are between the different k’s.\n4. The three findings are not particularly novel nor surprising (in fact, it would’ve been more surprising if any of the three findings had been shown to be false). The authors appeal to novelty with respect to Arora and Goyal (Line 73), and if that is indeed true, then it would be helpful to provide more substantial analyses with respect to the specific claims made in the first paper (e.g. show which relations or equations from Arora and Goyal fail to predict the results of this paper).\n\nQUESTIONS:\n1. Perhaps I missed this in the main text, but how many seeds and dataset recombinations were used in the experiments? I’m particularly interested in the observation that the model performed better on the held-out set than the training set (Line 260) and whether that is a consistent phenomenon.\n2. Could you provide a metric for how consistently the two models from Table 4 rated the responses? It would provide higher confidence if the 31% and 24% (top-left entries) actually had high overlap.\n\nLIMITATIONS:\nThe authors have acknowledged the limitations and impact in the supplements.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}